ESXi 6.0 host locked up

Came in this morning and had a Host locked up. Tried to restart management services and hung up, after submitting command. Only thing I knew to do was hard reset. VM's were inaccessible.

Wanting to know where I can get information on what happened to this Host, so I can make sure I'm not looking at hardware failure or potentially.
LVL 1
HaroldNetwork EngineerAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
All the logs will be in /var/log

the problem is, depending upon how you have installed ESXi, is whether those logs have been reset, when you restarted the host, and it's recommended

1. to specify a remote location to store the logs.
2. specify and configure a syslog server to send all the logs to.

the host is not oversubscribed, this is to mean all the host memory has not been allocated to VMs...

were none of the VMs responsive or running ?

Is the host patched to 6.0.0B?

Is this a new host?

Have you tested the hardware ?
0
HaroldNetwork EngineerAuthor Commented:
I've redirected the logs to a LUN, but didn't know if there was a specific one or quick way to view.

Host has been up and running for about a year.  ESXi 6.0.0, 2809209

Not even half memory is used.

Now getting this "vmrc console has disconnected attempting to reconnect" on vCenter VM, which was one VM I couldn't access.
0
Luis MouraCommented:
How many hosts do you have?
Did you try to move this VM to other host, see if its works?
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
ESXi 6.0, was only released in March 2015!

You are on 6.0.0b, released in July 2015.

you need to check and inspect all the logs, but the main log is \var\vmkernel.log

do you have shared storage ?
0
HaroldNetwork EngineerAuthor Commented:
Luis: my vCenter VM is the one I can't get into.  :-(

Andrew: there is a dedicated LUN for the logs.

I can't even power off the VM, DANG!!   vim-cmd vmsvc/power.off 11
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
check the /var/logs and vmkernel.log

and check there are no loss of paths, if iSCSI, which could cause the hang!
0
HaroldNetwork EngineerAuthor Commented:
wow, this is not good. I can't even force reboot the host.

Which log?

Can't view anything in configuration. hangs
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
check the /var/logs and vmkernel.log

configure a syslog server.
0
HaroldNetwork EngineerAuthor Commented:
how do I view them?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
vi or download them and use notepad++
0
HaroldNetwork EngineerAuthor Commented:
Never viewed this, can you make any of this out?
vmkernel.log
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
there is an error

with

"naa.6000402004e0141f7feec5d400000000" on path "vmhba35:C0:T2:L2" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL

can you check this iSCSI LUN maybe ?

is vmhba35 the software iSCSI initiator?

if it's polling this lun, it could cause the issue
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
HaroldNetwork EngineerAuthor Commented:
That's my vCenter LUN. I see it but don't see errors on SAN or ESX side.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
something is not happy with that LUN.

do you have multi paths...
0
HaroldNetwork EngineerAuthor Commented:
"do you have multi paths?"  meaning to the SAN?  Yes, and they are all up.

That scared me! Finally got back into vCenter VM. Not sure what's going on but have to investigate the SAN for potential problems.

But as far as the Host failing, is there something I need to look for along with this other problem and how do I locate particulars in the logs?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
you have access to all the logs.

I would not concern yourself with vCenter Server, it's just a management server, and that's all it does, whether it's on or off, will not affect the running VMs.

vmkernel.log is the log, along with dmesg, syslog.log.

you have access to the logs.

I would check everything is in order on the SAN, and network, the log is reporting a failure access a LUN.
0
HaroldNetwork EngineerAuthor Commented:
What is fip? The SAN is fiber channel.
fip: host14: FIP VLAN ID unavail. Retry VLAN discovery.
2015-09-10T18:27:17.275Z cpu15:33252)<6>host14: fip: fcoe_ctlr_vlan_request() is done
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Fibre Channel over Ether discovery.
0
HaroldNetwork EngineerAuthor Commented:
This acronym would have made more since..fcoe. Anyway, sorry for the delay, it was the FC controller that failed.

Thanks again
0
HaroldNetwork EngineerAuthor Commented:
Regards
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.