Cannot reconnect ESXi host to vCenter after changing its management IP

We have ESXi 6.0 hosts with vCenter appliance version 6.7. Needed to move vCenter and ESXi hosts to a new IP subnet. Changed IP address of vCenter a few weeks ago without a problem.

Yesterday, after putting an ESXi host in Maintenance mode, I changed its management IP and after hours of troubleshooting, the ESXi started responding on new IP, with old name. (There is redundancy in links.) However we cannot reconnect this ESXi host back to vCenter now.

This host and vCenter are on the same subnet. (They can ping each other.)  I have restarted management service of host a few times. Have also rebooted host and vCenter. Still get the same error: "Cannot contact the <> host (10.0.xx.xx). The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding."
Please advise. Thanks.
LVL 3
AkulshAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Can you connect directly to ESXi 6.0 host using vSphere Client ?

Have you removed the older vCenter Server agent ?

what builds of ESXi 6.0 and vCenter Server Appliance do you have ?

also check the logs...
AkulshAuthor Commented:
Hi Andrews,

1. Yes, no problem connecting directly to ESXi 6.0 host using web interface. In fact, I keep that interface open all the time to check on Management Network and status of Service of this disconnected host.

2. Not sure what you mean by "Have you removed the older vCenter Server agent?" Versions of vCenter and Hosts have not changed at all. Only IP of 1st host has changed so far.

3. As mentioned earlier, we have ESXi 6.0 hosts with vCenter appliance version 6.7. (The hosts could not be upgraded.)

4. Which log should I focus on, and how/where to find it? Thanks.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
2. Remove the vCenter Server agent from the host.

3. I'm not asking to upgrade, but ensure that you are using the latest builds of ESXi 6.0 and 6.7.

4. Do the above first.
Your Guide to Achieving IT Business Success

The IT Service Excellence Tool Kit has best practices to keep your clients happy and business booming. Inside, you’ll find everything you need to increase client satisfaction and retention, become more competitive, and increase your overall success.

AkulshAuthor Commented:
Removing vCenter Server Agent from this ESXi host is proving rather difficult. My research has shown that it entails removing vpxuser and removing two services.

A. I tried to delete vpxuser account but "userdel vpxuser" command does not work. Get message 'no such command'. Also in web interface of host, only user listed is root.

B. In the uninstaller directory (/opt/vmware/uninstallers), I found VMware-fdm-uninstall.sh file but not VMware-vpxa-uninstall.sh file. I therefore have uninstalled the HA (or fdm) agent and its service is gone. However, 'VMware vCenter Agent' service is still there since I find no way to uninstall it. Even other hosts don't have VMware-vpxa-uninstall.sh file on them. I have stopped this service but host still cannot connect to the cluster.

Please advise. Thanks.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
You do it by disassociating the host from vCenter Server - via the client!
AkulshAuthor Commented:
On connecting directly to the ESXi host using the vSphere Client, there is no Summary page anymore. I cannot find "Disassociate Host from vCenter Server" link anywhere. (The host is 6.0 and vCenter is 6.7.)

BTW, the host has been taken out of cluster and also from inventory of vCenter. It still has remnants of vCenter which need to go.

Please advise. Thanks.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Was the host just originally disconnected or removed ?

You suggest reconnecting in the OP, so I assume it was just disconnected.

otherwise it's time to start looking at logs, or if there are no VMs on this host, re-install ESXi from scratch.
AkulshAuthor Commented:
The host was removed from the vCenter when it could not be connected back after IP change.
Which logs would be most pertinent and where to locate them?
(I am little hesitant to reinstall since it has many networks configured.) Thanks.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
See here for logs locations

https://kb.vmware.com/s/article/2032892

Also you could backup the configuration and restore the configuration for your server to a fresh ESXi install.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
AkulshAuthor Commented:
I had to open a case with VMware, but 2 engineers did not solve my problem. The last one did point out that vCenter cannot reach ESXi server on HTTPS port.

The cause of my problem was the local firewall settings of ESXi host, configured by previous admin, which were preventing vCenter from adding that host. None of two engineers thought of looking at the firewall of ESXi host.

The reason the problem was so hard to spot was the fact that other two ESXi hosts, with identical firewall settings, were already connected to vCenter server. (The IP of vCenter server was changed a few weeks ago, and it kept working fine with 3 ESXi hosts even though all 3 firewalls should have blocked https connection from new IP subnet of vCenter. Somehow 'Denied' setting of firewall only went into effect when IP of ESXi server was changed.) Quite odd behavior of vSphere.  Thanks.
AkulshAuthor Commented:
Andrew, as always, was most helpful, but because of the very odd scenario, I ended up finding the solution myself, as described in my previous post. (In this instance vCenter logs were not informative.)
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
vSphere

From novice to tech pro — start learning today.