Regaining admin connectivity to vSphere host

Ben Conner
Ben Conner used Ask the Experts™
on
I've lost admin connectivity to a production host, although the VMs on it appear to be fine.  Can't log in through the VCSA nor directly to the host.

Any suggestions on getting this under control?  I can log in through PuTTY.

vSphere 6.7 environment.

--Ben
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
Reset Network configuration to defaults at the console?

Reset Management Agents at the console ?

can you ping the host ?

can you telnet to 80 and 443 ?
Ben ConnerCTO, SAS developer

Author

Commented:
I can ping the host and telnet to both 80 & 443.

How do I reset Management agents after logging in?

--Ben
Ben ConnerCTO, SAS developer

Author

Commented:
Looks like the local management console isn't available at the moment.  Should be after a reboot.

Is there a way to initiate a reboot from the console where all the VMs are shut down gracefully?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
no console ? to login with F2 ?

if you are connected with SSH, type reboot.
Ben ConnerCTO, SAS developer

Author

Commented:
reboot shuts down the VMs gracefully?  Just want to make sure
Yeah, I switched KVM cables to it and apparently confused the USB ports
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
Oh the VMs are not shutdown yet....

vim-cmd vmsvc/getallvms

Open in new window


will list all registered VMs with their "vmid" in the first column

 vim-cmd vmsvc/power.getstate vmid

Open in new window


get the power state of a VM

 vim-cmd vmsvc/power.shutdown vmid

Open in new window


shut down a VM through VMware Tools

and then reboot.

you may want to check again all VMs are off.
Ben ConnerCTO, SAS developer

Author

Commented:
Hm.  the vim-cmd vmsvc/getallvms    isn't responding.  I can log in to each vm and shut it down.  Will do that instead, then try to reboot
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
you'll need to shutdown VMs manually from within the OS then....

your host is stuffed!
Ben ConnerCTO, SAS developer

Author

Commented:
does a reboot come back with anything other than the command prompt?

Yeah, I'm guessing.  Fortunately I have a DR box standing by.
BTW, it wasn't the ESXi host I had swapped the KVM cable out on.  Got confused.  Not sure why the console wasn't responding.
Ben ConnerCTO, SAS developer

Author

Commented:
The host is coming up but hanging after the nfs41client loaded.  I'm thinking this has to do with the iscsi target.  Does it time out?  I shut the box down I had defined the target on.

Ok, it finally finished coming up
VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017
Commented:
It will eventually time out....

but if you have not removed the LUNs, it will be trying to find the SAN and LUN.

Remove the iSCSI IP Address from Dynamic Discovery.
Ben ConnerCTO, SAS developer

Author

Commented:
Well that was exciting.  All appears to be normal now, with the ISCSI datastore gone from the configuration.  Not at all unhappy with that.  Will do this a lot more cautiously with a host I can afford to tank.  Good thing it is a Sunday.

All the leftover pieces are now removed.  Will go through this later when I have a better clue.

Thanks Andrew!

--Ben
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017
Commented:
It happens today even with a proper SAN, which is one of the reasons why some of our clients gave up with iSCSI and migrated to NFS because then said configuration was difficult or required, and one client stated "iSCSI is a ticking time bomb! who uses it really"

migrated to NFS!

We still use iSCSI but tread carefully with it....
Ben ConnerCTO, SAS developer

Author

Commented:
Yes, I can see that.  Now am trying to configure NFS to be as fast as possible.  Poking at different things, teaming 2 10g nic cards together, increasing the MTU values on the virtual switch for vmotion, etc.

May try a Lin*x distro as well for NFS, just haven't gotten there yet.

Thanks, Andrew!

--Ben

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial