Cant add any NEW ESXi host to VCenter!

Environment = Vcenter Version 6.7.0.20000 Build 10244845
ESXi host version = 6.7 U1 11675023


We are trying to add a Host to a new cluster that was just created.
We are going through the process and the Host is on the network, it tests successfully in the ESXi console.  WE can ping the IP and the gateway and dns.


Here is the Issue:
When we go through the wizard to add a new host to this cluster we get the certificate warning, and select "Yes" and go through and select a license and finish.  The task gets to 100% and then fails with the following error:

Cant add Host to Cluster - A general system error occurred: Unable to get signed certificate forhost name 'ESXi1.mydomain.com' ip 'xx.xx.xx': Error: Failed to connect to the remote host, reason = rpc_s_no_memory (0x16c9a012). (382312466)

Never seen this before, any ideas?
JB BlancoSr Systems EngineerAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Do you get the same error if you just add the host to the inventory

And later drag and drop host into Cluster
JB BlancoSr Systems EngineerAuthor Commented:
Do you get the same error if you just add the host to the inventory

And later drag and drop host into Cluster

Yes same error
JB BlancoSr Systems EngineerAuthor Commented:
I get it when adding the host at the datacenter level - before even moving it to the cluster.  Just adding the host in general is giving that error

We also tried another host on another subnet that's fully reachable and ready to be added to vcenter and we also get the same error.

Its like Vcenter all of the sudden is no longer allowing us to add anymore hosts!
Your Guide to Achieving IT Business Success

The IT Service Excellence Tool Kit has best practices to keep your clients happy and business booming. Inside, you’ll find everything you need to increase client satisfaction and retention, become more competitive, and increase your overall success.

JB BlancoSr Systems EngineerAuthor Commented:
we have another host we added to inventory a new host last week with the same 6.7 U1 11675023 and it gave us no problems.

Now for some odd reason, we are getting this error constantly.

still researching here.

Thanks for your help Hancock.  Let me know if you have any further ideas! thanks!
JB BlancoSr Systems EngineerAuthor Commented:
It appears like vcenter is unable to replace the Host's Self signed cert with the Vcenter one.  Even though we click "Yes" to the Cert warning, and are able to go through the rest of the wizard and add the host, it seems like there is an issue right at 100%.  We see the host popping up in Vcenter, then it immediately disappears with that error above.  

We just now tried to remove a previously added host and re-add it to the inventory and it gave us no problem.  I think this is because, the self signed cert was successfully replaced with VCenter's since this host was added already successfully in the past.  

We also tried rebooting our VCSA with no luck, still get the same error.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Your time on hosts and vCenter are all in sync?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Are these upgraded hosts ?

Upgraded VCSA?
Paul SolovyovskySenior IT AdvisorCommented:
How many hosts and what type of vCenter/Host Licensing?
JB BlancoSr Systems EngineerAuthor Commented:
Your time on hosts and vCenter are all in sync?
yes.  But this is a new host we are adding.  But yes i browsed directly to the host and set the NTP settings and started the service just to see if that was the case.



Are these upgraded hosts ?

Upgraded VCSA?

These hosts are brand new install of 6.7 U1 11675023
The VCSA is upgraded

I read that google search earlier and if you look at the error we are getting, its different than what that google search result is pointing to .
JB BlancoSr Systems EngineerAuthor Commented:
How many hosts and what type of vCenter/Host Licensing?

157 total hosts

Vsphere 6 advanced ROBO
vsphere 6 with vSOM 6 Enterprise Plus
VSAN Witnesses

But we get the same error even when i select evaluation license on the host
JB BlancoSr Systems EngineerAuthor Commented:
Not Sure if this would have somthing to do with it.  I am new to the VCSA.  
hmmm
Not sure how to check from within the VCSA console more details on the disk storage space
hmm2
JB BlancoSr Systems EngineerAuthor Commented:
I found more details,
hmm3that hardisk 13 looks almost full, not sure if this would have somthing to do with the error we are getting.
Murali SripadaVMware Engineer | vExpert 18/19Commented:
you can ignore the /storage/archive partition full as it is known issue and does not impact vcsa workflow.

both vcsa and host are in same subnet or different?
 take ssh on vcsa and run the below command and check if you are getting output

openssl s_client -connect <host name/ip of esxi>:443

if you are receiving an output with certificate then it is good else there is an issue with network and needs further troubleshooting however we can isolate if the issue with network or not
JB BlancoSr Systems EngineerAuthor Commented:
both vcsa and host are in same subnet or different?

Different subnets

take ssh on vcsa and run the below command and check if you are getting output

openssl s_client -connect <host name/ip of esxi>:443

if you are receiving an output with certificate then it is good else there is an issue with network and needs further troubleshooting however we can isolate if the issue with network or not

Here is what i get:

meNotice the last message at the very bottom "Verify return code: 21 (unable to verify the first certificate)
JB BlancoSr Systems EngineerAuthor Commented:
both vcsa and host are in same subnet or different?

I should also point out that this happens with any NEW ESXi host we try to add to vcenter.   We tested removing and re-adding ESXi hosts we already have in vcenter with the same ESXi Version and build and the issue does not occur.  Only with NEW Hosts we try to add.
Murali SripadaVMware Engineer | vExpert 18/19Commented:
so 443 https connection is good between host and vcenter. interesting that it is happening with any new host.. try to change the cert mode in centre we client to thumbprint and try adding Esxi host and let us know if it makes any difference https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.security.doc/GUID-122A4236-9696-4E1F-B9E8-738855946A93.html
JB BlancoSr Systems EngineerAuthor Commented:
so 443 https connection is good between host and vcenter. interesting that it is happening with any new host.. try to change the cert mode in centre we client to thumbprint and try adding Esxi host and let us know if it makes any difference https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.security.doc/GUID-122A4236-9696-4E1F-B9E8-738855946A93.html

We are thinking it is network related issue.  We have vMware support on the line helping us troubleshoot.  If we get stuck, I will suggest giving that a try.
JB BlancoSr Systems EngineerAuthor Commented:
Any other suggestions are welcom guys!

Thanks for all the responses!!

Any way we can confirm this is the network?  Cause thats what we are leaning towards since we changed nothing on the Vcenter or host end.  

But we are having a hard time being able to isolate if its a network issue.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
The logs should help, VMware (probably certificate issue and bug with Platform Service Controller)
Murali SripadaVMware Engineer | vExpert 18/19Commented:
please rebootvcentre and psc as final resort
Murali SripadaVMware Engineer | vExpert 18/19Commented:
rebooting will fix the issue for sure I just fixed it for someone

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
JB BlancoSr Systems EngineerAuthor Commented:
rebooting will fix the issue for sure I just fixed it for someone

Yes! that's what we missed!  we only rebooted the Vcenter appliance but not the PSC!.

Thank you !

we were able to add the host successfully!   VMware is looking into why this happened.  

But they have no answer yet.
Murali SripadaVMware Engineer | vExpert 18/19Commented:
vmca service might be out of memory based on error
Murali SripadaVMware Engineer | vExpert 18/19Commented:
rpc_s_no_memory  this is why I suspect it
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
BUG!!
Murali SripadaVMware Engineer | vExpert 18/19Commented:
possible but cannot say all the time unless 3
we know what's going on in the environment.. if this is considered bug,  we have many bugs is most of the softwares in the world as many of them don't have reasons for errors
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Networking

From novice to tech pro — start learning today.