ESXi 4 host not responding, sort of hung, VM down

I had a ESXi 4 host server that had problems today. It only had one VM on it and the call started with the VM going offline. I could connect to the ESXi host through the vSphere client but I could not reset or power off the VM or look at the console, nothing showed up in the ESX event log and the server never rebooted or anything.

I tried putting the host in maintenance mode and I got a error that it timed out, I tried updating it through the host update tool and it timed out, I connected to the ILO and tried to look at the console and it started to let me connect, I hit F2, it asked me to change the admin password and I was able to put the old and both new in and then it just sort of hung and I hit enter but it never did anything and just sat at the change password screen.

I ended up just rebooting the server through the ILO and now the host and VM are up.

I tried to look at the logs and they only go back about 15 minutes ago and have no previous events.

This is a weird problem and I was hoping you guys could give me advice on how to handle this if it were on a major host server, if this thing had 50 production hosts rebooting would not be an option unless all of the host just went down. At any rate what could I have checked before rebooting the host and does anyone have any ideas what would cause this to happen?

The one VM is a SMS server. The host is up to date other than 2 patches that I am applying right now.now.

EDIT: Also need to add the ESXi host is running the basic free version and it is not connected to a vCenter server, yaay cheapo's in management.
REIUSAAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Did the virtual actually stop responding you can always confirm by ping or rdp.

Sometimes hosts can become unresponsive, its very rare. Can you ping, connect to host via ssh remotely, use the console. This will confirm if host is responsive.

if the issue is you cannot manage the virtual machine, or connect with vsphere client, or you get a message stating a task is in progress, this can sometimes happen, and the agent on the host server needs restarting, select restart network management agent on host, this should resolve the issue for you.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
free or licensed ESXI it does not matter.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
what host server make and model are you using?

Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

REIUSAAuthor Commented:
The VM was totally down, no ping no RDP and no connection through vShpere.

I believe the server is a HP 380 G5.

Can you still SSH into a ESXi host? I didn't try that.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay, well if it happens agin, try the following:-

1. ping ESX host.
2. connect to ESX host via ssh.
3. connect to ESX host via vSphere Client
4. ping VMs
5. RDP to VMs

Articles on how to enable ssh on ESXi host

http://kb.vmware.com/kb/1017910 
http://kb.vmware.com/kb/8375637 
http://kb.vmware.com/kb/1033013

Also, you may want to use a Syslog server to monitor ESXi host events, because after you reboot they are gone.

How to enable Syslog

http://kb.vmware.com/kb/1016621 

A simple Syslog server

www.kiwisyslog.com/kiwi-syslog-server-overview/ 

http://www.splunk.com/

(i prefer Splunk these days, because you can also sent all Syslogs from every service/server to it, and it gives a good timeline of events).

and finally, if the server is new I would check

1. Disk Heath from Insight Manager, or Health Status within vSphere Client/Host/Configuration

2. Network Health as above.

3. Memory Check (use http://www.memtest.org/)

Hopefully it was a one off, but that's what I would setup in place and check next time.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Also your server is on the HCL for ESXi 4

http://www.vmware.com/resources/compatibility/search.php?action=search&deviceCategory=server&productId=1&advancedORbasic=advanced&maxDisplayRows=1000000&key=380&release[]=-1&datePosted=-1&partnerId[]=41&formFactorId[]=-1&filterByEVC=0&filterByFT=0&min_sockets=&min_cores=&min_memory=&rorre=0

Also check you are running the latest firmwares from HP, at least P56 bios for server from Smartstart Firmware CD.

Are you running the latest VMware vSphere 4.1 U1?
Paul SolovyovskySenior IT AdvisorCommented:
If the server is ESXi a lot of the logs are memory resident so when you shut it down they go away. Configure vMA to get syslogs

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1024122
REIUSAAuthor Commented:
Thanks for the info. I will see if we can set them up to retain the logs.

I was also using the latest version vSphere.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.