VMWare HA Failed - All VMs are in unknown state. This includes VCenter. How to fix?

We have two ESXi Hosts and a Netapp filer.  Our VCenter server is on one of the hosts.  The network went down for one host and HA was initiated.  However, the network went down for the second host. We tried rebooting the hosts and the storage, but this did not fix the issue.

All VMs are now in unknown state.  How can I get VCenter working? Can I move it to a desktop and run it on VMWare Server? Would this be the fix?

What would be the best solution?

Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Login to the hosts individually and check to make sure that your storage adapters are connected. I frequently have to reset connections in my SAN and then rescan from vsphere to reestablish links for the guests.
anthonyphamAuthor Commented:
I have tried resetting the filer connection and connecting to the hosts, but when I add them to the inventory, it says unknown (invalid) as the VM.  This includes the vSphere.  
Paul SolovyovskySenior IT AdvisorCommented:
Had a Netapp panic this morning as well...fun stuff.

1.  First make sure that you can browse the datastores or nfs mounts.
2.  Login to the host and do the same (using VI client)

If you can than that's good, all you need to do is remove from inventory and browse the nfs mount or datastore and add it back in.

Powerful Yet Easy-to-Use Network Monitoring

Identify excessive bandwidth utilization or unexpected application traffic with SolarWinds Bandwidth Analyzer Pack.

anthonyphamAuthor Commented:
1.  We can browse our datastores and NFS mounts
2.  We can log into the host using VI client.
We tried removing it from inventory and browsing the nfs amount to add it.  However, when it's added, it says Unknown (invalid).  We have older VMs on direct storage and it doesn't say unknown on those.  This seems to only occur on VMs on the filer.  These VMs were setup with Vsphere 4.  The direct storage VMs weren't associated with Vsphere before.

Paul SolovyovskySenior IT AdvisorCommented:
see if you can unmount the nfs mount and re-add back in.  Since it's NFS you don't have any re-signature issues.

If you have another ESX/ESXi host sharing the same mount you should remove from inventory and re-register to the second host.  Try to run, if it works then you can just vmotion over to he first host
anthonyphamAuthor Commented:
I'm still waiting for VMWare support.  But is it possible to recreate the vxm file?  Would this fix the issue?
Paul SolovyovskySenior IT AdvisorCommented:
It is.   What I normally do, just in case

1.  Create a VM with no hard disks or a dummy hard disk
2.  Copy a vmdk file and vmdk-flat file to the new folder.
3.  Add exisiting hard drive ot the VM

Paul SolovyovskySenior IT AdvisorCommented:
Not sure if it will resolve issue but give a small VM a try.  

Have you tried to remove and add the VM from inventory localy on one of the hosts or on vCenter only?
Hi anthonypha,

We had a similiar problem. The only solution was to remove the VM from the VC inventory, browse the datastore and re-register it, then migrate the VM to a new host. If we tried to do anything to the VM after re-registering it but before the migration, the VM got marked as 'invalid' again.

If this doesn't work you can try a variation of what paulsolov has suggested, remove the vCenter from inventory and try creating a new VM. when creating a hard disk, select use existing disk. Then give the path of the vCenter vmdk here. This should work and your VM should come online.

And if you are in a mood for some sleuthing then check out this link http://itsupportjournal.com/2008/12/09/fix-invalid-guest-on-virtual-center/


Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
anthonyphamAuthor Commented:
We had two ESXi hosts and vCenter was running on one of them.  We tried adding and removing VMs locally, but it was still invalid...  This was a good lesson to not have vCenter or the vCenter DB reside with all your other VMs!

Though we wanted to have the latest data, we ended up just restoring our database and vCenter by copying it from the .snapshot in Netapp.  It was from 5PM from the earlier day.  After I moved it, I reconnected our VMs.

Thanks for the help everyone!
Paul SolovyovskySenior IT AdvisorCommented:
A good suggestion is to have the DB on a physical SQL Server since you can install vCenter on pretty much any server in under 30 min.  We have also been placing a non authorotative dns server to replicate dns to vcenter so if you have domain controllers go down you'll have the entries dynamically (vs creating host files entries).

Good to hear that you were able to use snapmanager for VI (my assumption) to resovle the issue.

BTW, make sure you use the netapp plug-in into virtual center, gives you great insight.

You can browse the storage, excxellent, try a rescan a few times more, it often needs this.. But see the following as well:

You stated "However, when it's added, it says Unknown (invalid).  We have older VMs on direct storage and it doesn't say unknown on those."

What client are you using?  If you attempt to access vSPhere VMs with an older client you will recieve this message while trying to add them to the inventory, verify you are using the corrent client, connect the ESX web page http://ESXHOSTNAME and install it.  
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft Server OS

From novice to tech pro — start learning today.