We help IT Professionals succeed at work.

TLS handshake failure on web browser connection to ESXi host

Ben Conner
Ben Conner asked
on
Hi,

Over the last day or so I tried to connect to my ESXi hosts directly in a web browser.  The browser is connecting but failing to complete the TLS handshake negotiation, eventually timing out.

Anyone see this before?  I can get to them through the VCSA but sometimes I want to check on something directly.

6.7.0.

Thanks!

--Ben
Comment
Watch Question

CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
Check approved encryption the browser can use. It is possible that the available options on the system does not match the options on the host.

I.e. Browser only supports sslv3 tls 1.0 while esxi has tls 1.1, 1.2
Either way the browser used does not have a matching secure protocol to that available ...
Ben ConnerCTO, SAS developer

Author

Commented:
Hi Arnold,

Thanks for the suggestion.  I checked the tls minimum security level in Firefox and it is set to 1.  Since this happens in Chrome, same result.
Also tried it in IE 11 which came out in 2015.  Same thing.  

Just tried it on a Windows 7 VM that had an older version of FF on it.  From there I got the following:

The certificate is not trusted because the issuer certificate is unknown.
The server might not be sending the appropriate intermediate certificates.
An additional root certificate may need to be imported.
Error code: SEC_ERROR_UNKNOWN_ISSUER

--Ben
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
This is a warning not an error deals with certificate being issued by a non public entity, ignore the warning, and proceed.
It is up to you whether to exempt.

The option if you gave an internal CA is to generate a CSR on the host and gave it signed by the internal CA then load the signed vert..

Whenever you get issuer type of errors, expiry, it is up to you to make a se isin on whether the certificate issue us a primary concern that would prevent you from transacting ..


I.e. If you go to your banks URL and get this error, it would be understandable that you would not proceed.

When you go to your internal resources, esx hosts, etc and get cert errors about issuer, it is safe enough to proceed,
Up to you whether to add the cert as trusted permanently (add exception).
Ben ConnerCTO, SAS developer

Author

Commented:
Hi Arnold,

In this case I'm not getting an option on whether to proceed or not.  It is a timeout error and several browsers get the same non-response (FF, Chrome, and Edge).  Is there a crypto package available that could aid in diagnosing this?  I'm not even sure which side is dropping the ball at this point.
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
Connection failure could mean that you do not have a path to ir the port on which it is setup is different than what you expect.

Check the vcsa potentially it may have been setup such that the access to the individual host is limited to a set of iOS, the vcsa and other specified systems.
Ben ConnerCTO, SAS developer

Author

Commented:
Where do I even check that in the VCSA?  Never had that restricted before.  Could have fat-fingered something I guess in the last week.  Who knows.
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
Ben ConnerCTO, SAS developer

Author

Commented:
I have the Lockdown mode disabled.  Host encryption mode is also disabled.  ?

--Ben
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
Are your esxi hosts behind their own firewall?
You have to untangle where the block is.

If you have a Linux system or OpenSSL

openssl s_client -connect ipaddress_of_vmeare:443 and see what you get
the point is whether access is being blocked and where the block is.
Ben ConnerCTO, SAS developer

Author

Commented:
I use a 10.x.x.x block on my lan.  That host is at 10.0.0.52 and I'm at 10.0.1.100.    Don't have OpenSSL installed; will do that and run the test.
Ben ConnerCTO, SAS developer

Author

Commented:
I get back:
CONNECTED(00000164)
write:errno=10053
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 308 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1579577124
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
Use the vcsa to check the services on the host, it seems thT the service HTTPS or sonething is listening on the port but there is no certificate, even self sign, self generated
Without a certificate, there is no way to exchange information.
Ben ConnerCTO, SAS developer

Author

Commented:
If I look at the certificate screen, I see what appears to be a valid cert.  

Certificate

Subject
    emailAddress=vmca@vmware.com,CN=10.0.0.50,OU=VMware Engineering,O=VMware,L=Palo Alto,ST=California,C=US
Issuer
    OU=VMware Engineering,O=vcsa.wwi.webworldinc.com,ST=California,C=US,DC=local,DC=vsphere,CN=CA
Valid from
    Dec 26, 2019 6:47 PM
Valid to
    Dec 25, 2024 6:47 PM
Status
    Good
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
I am at a loss about what certificate you are seeing. From the connection you have posted:
"write:errno=10053
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 308 bytes
---"
use openssl example and target the vcsa or any other secure site including this where you will see the expected response.
Part of the TLS/SSl setup after the connection (confirms the port is open) the server is supposed to present a certificate which will do several things:
1) validate the server
2) provide the public key for the client to use to send encrypted data
3) The client uses that to send a request to the server, with its own public key which the server will use to encrypt the response.
this is the process of the setup where TLS fails. The client is not getting a certificate, no public key available to communicate with the server.
Ben ConnerCTO, SAS developer

Author

Commented:
The screenshot I had was under System...Certificate in the VCSA UI for that host.

If I go into System/Firewall and look at the Service list, I see the services listed and allowed IP addresses for the different services.

Is there some other place I should be looking?
fw1.JPG
fw2.JPG
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
The issue is not solely the certificate, the host has to have the private key as well.
The certificate is a multi-year cert signed by vmare.

what happens if you browse to http://10.0.0.50?
Do you get redirected, or offered an option to download the vsphere client?

Something is not working, but I am unable to place my finger on what that might be.
Ben ConnerCTO, SAS developer

Author

Commented:
It redirects to https: and then times out.  That's what's so weird.
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
Try enabling the ssh service on the esxi host and see if you can get in that way.

Trying to avoid going the regenerate the certificate route that may result in locking the vcsa out at the attempt.
See if you open ssh whether it will get you through after making an update to the firewall to allow access via ssh and see where that gets you.
Ben ConnerCTO, SAS developer

Author

Commented:
I have ssh enabled already.  Was able to ssh to it via PuTTY.    Unfortunately I had already regenerated the cert earlier to see if that was the issue.  It wasn't.

Am also seeing replications failing now with Veeam (time out to target but not on all of them.  Nor the same ones each time).  Wasn't on the most current point release so I'm upgrading that now.
Ben ConnerCTO, SAS developer

Author

Commented:
That didn't help, but I didn't think it would, so that's fine.
Ben ConnerCTO, SAS developer

Author

Commented:
Hi Arnold,

This issue is much bigger than trying to access the hosts in a web browser.  I can't bring up a directory after sftp'ing in to either host using WinSCP.  Not even sure where to start, but I'll start a new question on this one.

Thanks!

--Ben
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
Could you check how many ips does the host have?
Ifconfig -a
Ben ConnerCTO, SAS developer

Author

Commented:
Hm.  Doesn't recognize that command or variants.  ?
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
Ifconfig deals with network adapter listing the -a enumerates all
Ben ConnerCTO, SAS developer

Author

Commented:
[root@localhost:~] Ifconfig
-sh: Ifconfig: not found
[root@localhost:~] ifconfig
-sh: ifconfig: not found
[root@localhost:~]
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
VCSA access is still there?
If you go through VEEAM you can list the Vms on the host?

Seems this issue was raised https://communities.vmware.com/thread/598844

not qute sure why the certificate is not being presented in the TLS session.
Ben ConnerCTO, SAS developer

Author

Commented:
Yes, VCSA access appears to be unaffected.  I see all the hosts and VMs under them.
Ben ConnerCTO, SAS developer

Author

Commented:
Well this is interesting.  I went to a different workstation that is running Windows 7 (don't ask) and tried bringing up the hosts on that machine.  They came up in a web browser with no issues.  So I tried it on a Win 10 laptop as well just now and have no issues getting to the hosts directly.

That still doesn't explain the Veeam inability to replicate, but that is a good data point regarding my Win 10 workstation.  My assumption is that something is tanked in the cert storage area on my computer.   I don't have much working knowledge in this area so I'm not sure where to begin.  Different question?

--Ben
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
On the system where it failed. Check to make sure the browser supports tls1.2 in the security section. Internet security, advanced.

The strange thing openssl was it being run from a windows or linux system?
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
Veeam check a channel, crypto to make sure it supports tls 1.2. the other option is enable tls 1.1 or 1.0 which are off by default on 6.7
Ben ConnerCTO, SAS developer

Author

Commented:
In FF, the minimum setting for TLS is 1.0.
I installed Openssl on my Win 10 workstation.

Where is that setting in 6.7?
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
See https://kb.vmware.com/s/article/2147469
You might need to have a subscription, account to access.
CTO, SAS developer
Commented:
I had to call in a VMWare expert to track this down.  Apparently the issue was related to a switch configuration error on my part I did about a week earlier that I didn't catch.  What is odd is it didn't really manifest itself until days later, as far as I recall.

Specifically what I botched was failing to assign some NICs to the vmotion network on one host.  

I still don't understand how that would cause both hosts to become inaccessible, but I can't argue with the result.  They simplified my switch configuration and everything works flawlessly now.

Sometimes it just pays...to pay for an extra set of experienced eyes.

Thanks Arnold for trying to figure this out with insufficient information.

--Ben