Cannot reconnect ESXi host to vCenter after changing its management IP

Akulsh
Akulsh used Ask the Experts™
on
We have ESXi 6.0 hosts with vCenter appliance version 6.7. Needed to move vCenter and ESXi hosts to a new IP subnet. Changed IP address of vCenter a few weeks ago without a problem.

Yesterday, after putting an ESXi host in Maintenance mode, I changed its management IP and after hours of troubleshooting, the ESXi started responding on new IP, with old name. (There is redundancy in links.) However we cannot reconnect this ESXi host back to vCenter now.

This host and vCenter are on the same subnet. (They can ping each other.)  I have restarted management service of host a few times. Have also rebooted host and vCenter. Still get the same error: "Cannot contact the <> host (10.0.xx.xx). The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding."
Please advise. Thanks.
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
Can you connect directly to ESXi 6.0 host using vSphere Client ?

Have you removed the older vCenter Server agent ?

what builds of ESXi 6.0 and vCenter Server Appliance do you have ?

also check the logs...

Author

Commented:
Hi Andrews,

1. Yes, no problem connecting directly to ESXi 6.0 host using web interface. In fact, I keep that interface open all the time to check on Management Network and status of Service of this disconnected host.

2. Not sure what you mean by "Have you removed the older vCenter Server agent?" Versions of vCenter and Hosts have not changed at all. Only IP of 1st host has changed so far.

3. As mentioned earlier, we have ESXi 6.0 hosts with vCenter appliance version 6.7. (The hosts could not be upgraded.)

4. Which log should I focus on, and how/where to find it? Thanks.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
2. Remove the vCenter Server agent from the host.

3. I'm not asking to upgrade, but ensure that you are using the latest builds of ESXi 6.0 and 6.7.

4. Do the above first.
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Author

Commented:
Removing vCenter Server Agent from this ESXi host is proving rather difficult. My research has shown that it entails removing vpxuser and removing two services.

A. I tried to delete vpxuser account but "userdel vpxuser" command does not work. Get message 'no such command'. Also in web interface of host, only user listed is root.

B. In the uninstaller directory (/opt/vmware/uninstallers), I found VMware-fdm-uninstall.sh file but not VMware-vpxa-uninstall.sh file. I therefore have uninstalled the HA (or fdm) agent and its service is gone. However, 'VMware vCenter Agent' service is still there since I find no way to uninstall it. Even other hosts don't have VMware-vpxa-uninstall.sh file on them. I have stopped this service but host still cannot connect to the cluster.

Please advise. Thanks.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
You do it by disassociating the host from vCenter Server - via the client!

Author

Commented:
On connecting directly to the ESXi host using the vSphere Client, there is no Summary page anymore. I cannot find "Disassociate Host from vCenter Server" link anywhere. (The host is 6.0 and vCenter is 6.7.)

BTW, the host has been taken out of cluster and also from inventory of vCenter. It still has remnants of vCenter which need to go.

Please advise. Thanks.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
Was the host just originally disconnected or removed ?

You suggest reconnecting in the OP, so I assume it was just disconnected.

otherwise it's time to start looking at logs, or if there are no VMs on this host, re-install ESXi from scratch.

Author

Commented:
The host was removed from the vCenter when it could not be connected back after IP change.
Which logs would be most pertinent and where to locate them?
(I am little hesitant to reinstall since it has many networks configured.) Thanks.
VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017
Commented:
See here for logs locations

https://kb.vmware.com/s/article/2032892

Also you could backup the configuration and restore the configuration for your server to a fresh ESXi install.

Author

Commented:
I had to open a case with VMware, but 2 engineers did not solve my problem. The last one did point out that vCenter cannot reach ESXi server on HTTPS port.

The cause of my problem was the local firewall settings of ESXi host, configured by previous admin, which were preventing vCenter from adding that host. None of two engineers thought of looking at the firewall of ESXi host.

The reason the problem was so hard to spot was the fact that other two ESXi hosts, with identical firewall settings, were already connected to vCenter server. (The IP of vCenter server was changed a few weeks ago, and it kept working fine with 3 ESXi hosts even though all 3 firewalls should have blocked https connection from new IP subnet of vCenter. Somehow 'Denied' setting of firewall only went into effect when IP of ESXi server was changed.) Quite odd behavior of vSphere.  Thanks.

Author

Commented:
Andrew, as always, was most helpful, but because of the very odd scenario, I ended up finding the solution myself, as described in my previous post. (In this instance vCenter logs were not informative.)

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial