Mysterious case of random Individual ESXi hosts will become disconnected from VCenter ?

People,

Can anyone please assist and help me with the issue of ESXi random disconnection from VCenter as below:

[VMware vCenter - Alarm alarm.HostConnectionStateAlarm] alarm.HostConnectionStateAlarm changed status from Green to Red

But the VMs are fine.  This has happened when we were on 4.1 and also now on 5.1.  I did some investigation but could not find anything definitive.
LVL 11
Senior IT System EngineerSenior Systems EngineerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

tshearonCommented:
Can you tell us what troubleshooting you have done to try to resolve this issue? That will save yes some time. Also, can you tell us what you do to resolve it temporarily as I assume they don't stay disconnected?

In an effort to save you time you can take a look at this VMware KB that addresses ESXi hosts reasons for disconnecting.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003409
Senior IT System EngineerSenior Systems EngineerAuthor Commented:
What I did was to right click and connect them back to VCenter.
Zacharia KurianAdministrator- Data Center & NetworkCommented:
I suspect possible duplicated IP/IPs on the network.

or please refer to the below KB from VMware.
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2020620
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay, lets check, what build versions on your on?

have you looked at the logs?

have you checked your network?

is vCenter physical or virtual ?
Senior IT System EngineerSenior Systems EngineerAuthor Commented:
VCenter is virtual
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
What build versions are you using ?

Can you upload your vCenter logs ?

Do all your hosts and vCenter Server have STATIC IPs ?

Do you use DNS or IP Address when adding to vCenter Server?
Senior IT System EngineerSenior Systems EngineerAuthor Commented:
VMware is 5.1 Update 1 - VCenter and - 5.1.0 1064983
ESXi hosts 5.1.0 1065491

Do all your hosts and vCenter Server have STATIC IPs ?
yes they do have static IPs

Do you use DNS or IP Address when adding to vCenter Server?
from the vSphere console it is using DNS name or FQDN not using IP address.

VCenter Logs will be uploaded again when the problem occurring, it affects random ESXi hosts not just one single ESXi host.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
do they disconnect and re-connect?
Senior IT System EngineerSenior Systems EngineerAuthor Commented:
They just went disconnected and I got the VCenter alarm, not reconnecting back so I have to manually reconnect them.
Senior IT System EngineerSenior Systems EngineerAuthor Commented:
it is possibly due to this advisories: http://h20566.www2.hp.com/hpsc/doc/public/display?sp4ts.oid=5215333&docId=emr_na-c04218016&docLocale=en_US

because the description and the symptomps is the same that I experience.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Senior IT System EngineerSenior Systems EngineerAuthor Commented:
as advised by HP in the other instance of hardware issue.
Senior IT System EngineerSenior Systems EngineerAuthor Commented:
OK, another update for this case:

All of my ESXi are on 5.1.0 build-1065491
This issue occurs because the vCenter Server agent on the host (vpxa) fails to send heartbeats to the vCenter Server.
This is a known issue affecting ESXi/ESX 4.x and ESXi 5.x, the resolution is to update the ESXi host to Build 1157734 ( ESXi 5.1 Patch 2 )
 
Reference - http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2054052

Patch details - http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2052143

Hope this helps.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.