ESXi 6.7 Remote Login issue
I have an odd issue with a vSphere 6.7 environment. 2 Hosts that have been running fine for about 4 months are now exhibiting connection errors. I disconnected and removed them from inventory in vCenter and they were removed without error, but I cannot get them back in. I keep getting incorrect login. I cannot event log into the local WebUI of the hosts. I get an error saying username or password are incorrect, I cannot SSH to them either. However, connecting via DCUI through F1 or F2 logins work without issue....
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Did you enable lockdown mode?
ASKER
No Lockdown is not enabled...
ASKER
Now that I look again it is actually Greyed out in the DCUI
ASKER
Saw that Article and attempted the command line method at the bottom, but getting an authentication login error there too
It looks like Lockout Mode is enabled ?
SSH access ?
SSH access ?
ASKER
SSH is enabled, but get connection refused, telnet to port 22 also fails i am looking into firewall
ASKER
Cannot run that command either esxcli network firewall get returns General System Error: Internal error on both Hosts...
Why did you remove them from vCenter Server ?
ASKER
It all started because Veeam backup was failing do to NFC storage connection errors and I found a VMware knowledge base that suggested disconnecting and reconnecting the Hosts in vCenter. I disconnected but could not reconnect do to login failures so I removed them from inventory and tried to add them back in which put me where I am at right now.
I'm almost ready to reboot the host but my only access to shut down the VMs is through the Guest OS and if I still cannot get into WebUI after the reboot I'll have no way of powering them back up. Best I can do is hard boot and hope they come back on line which I am very reluctant to do.
I'm almost ready to reboot the host but my only access to shut down the VMs is through the Guest OS and if I still cannot get into WebUI after the reboot I'll have no way of powering them back up. Best I can do is hard boot and hope they come back on line which I am very reluctant to do.
ASKER
could a ramdisk full situation be the cause?
yes, or scratch full, temp
ASKER
So I can be reasonably co fident that after a reboot I'll be able to log into the WebUI ?
There are no guarantees.
ASKER
when I run vdf -h I see that "var" and "tmp" are 100% Use on Host 1 and Host 2 just "var" is 100%. I opened a ticket with VMware and will try to have them on the line when I reboot.
okay, running out of space.....
look at logs and delete them, same with /tmp
look at logs and delete them, same with /tmp
ASKER
I only have shell access should I just delete the file or is there another way to clear contents but leave file?
ASKER
Also a file size of 12288.. this is KB correct?
you need to look at the contents of
/var/log and /tmp
and either copy the files off using Winscp, and check if that clears up the space if 100% full.
/var/log and /tmp
and either copy the files off using Winscp, and check if that clears up the space if 100% full.
ASKER
Can't do that because SSH and SCP connections are refused even though both are SSH is enabled on the hosts
ASKER
I think I have no choice but to reboot at this point...
oh, I see - this can happen when it fills up.....
ASKER
OK... so I finally got VMware to remote in and they found an HPE VIB was part of the issue and need to apply this update "esxi6.7uX-mgmt-bundle-3.4
In the var there was a file /var/log/mili/mili2d.txt which is related to a QLogic driver. They deleted that, but it requires a reboot to reflect recovered space. There is one more step to modify a config, but that will be done tomorrow so I will update that process when complete.
In the var there was a file /var/log/mili/mili2d.txt which is related to a QLogic driver. They deleted that, but it requires a reboot to reflect recovered space. There is one more step to modify a config, but that will be done tomorrow so I will update that process when complete.
seen the same problem before with iSCSI filling up the log space!
this can also affect FDM agent deployment and failure because no space!
this can also affect FDM agent deployment and failure because no space!
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.