VMware - Cannot reconnect ESXi Host in Vcenter

This morning I came in to find that one of my ESXi hosts is disconnected in vCenter.  Attempting to reconnect gives the following error:

Cannot contact the specified host.  The host may not be available on the network, a network configuration problem may exist or the management services on this host may not be responding.

Following the VMware KB procedures in " Restarting the Management agents on an ESXi or ESX host (1003490)" i tried restarting the management agents via the DCUI.  I pressed F2 and received a login prompt.  I enter the root credentials and pressed Enter.  The screen went back to the default splash screen and now hitting F2 does nothing (nor does hitting F12).

Continuing to follow the same KB article, I tried restart the management agents via SSH and received errors about "no space left on device"  (See attached Image).  There is 1.5TB free on the VMFS-5 volume.

Can anyone suggest how to proceed with regaining connectivity of the host to vCenter or the viClient?
ESXi-Host-Failure.jpg
CipherUserAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
The lack of space, is probably the space on the ESXi installation partitions.

try removing the logs.
0
CipherUserAuthor Commented:
Andrew,

That's a great idea! Could you please point me to the location of the logs that can be safely deleted?  

Thanks!
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
/var/logs

and you can remove all the tar balls or copy them off via SCP.

you may need to reboot, if the services have hung1
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

CipherUserAuthor Commented:
Andrew,

There doesn't seem to be any space to recover in the /var/log directory (See 1st image).

Also, I found out that I'm out of inodes using VMware KB article: ESXi/ESX error: No free space left on device (1007638).  I ran the command they suggested to find large files to delete, but none of them look like good candidates (See 2nd Image).

You are dead on that I need to delete files, but deleting anything in the /var/log directory doesn't look like it will free up more than a few KBs.

Do you have any other suggestions on how to free up some space after seeing the images I posted?

Thanks in advance for your help!!
var-log-Directory.jpg
28634766OutOfinodes.jpg
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
do you have any third party VIBS or management installed ?
0
CipherUserAuthor Commented:
No third party VIBs or managament installed.  I do use Veeam Backup and Recovery, but it's installed on a separate physical server.
0
CipherUserAuthor Commented:
I found the root cause of the problem here: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2037798

After following the directions in the KB article, I was able to reconnect the ESXi host to vCenter.

Thanks for your help Andrew!
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
CipherUserAuthor Commented:
Andrew was correct in determining that the ESX partition was full thus the awarding him all the points.  My last comment with the KB link was the actual fix that allowed the ESX server to be reconnected.  Selected it as the Best Solution so that others would benefit from it as well as I did.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.