VMWare HA Failed - All VMs are in unknown state. This includes VCenter. How to fix?

We have two ESXi Hosts and a Netapp filer.  Our VCenter server is on one of the hosts.  The network went down for one host and HA was initiated.  However, the network went down for the second host. We tried rebooting the hosts and the storage, but this did not fix the issue.

All VMs are now in unknown state.  How can I get VCenter working? Can I move it to a desktop and run it on VMWare Server? Would this be the fix?

What would be the best solution?

Who is Participating?
SunshineVKConnect With a Mentor Commented:
Hi anthonypha,

We had a similiar problem. The only solution was to remove the VM from the VC inventory, browse the datastore and re-register it, then migrate the VM to a new host. If we tried to do anything to the VM after re-registering it but before the migration, the VM got marked as 'invalid' again.

If this doesn't work you can try a variation of what paulsolov has suggested, remove the vCenter from inventory and try creating a new VM. when creating a hard disk, select use existing disk. Then give the path of the vCenter vmdk here. This should work and your VM should come online.

And if you are in a mood for some sleuthing then check out this link http://itsupportjournal.com/2008/12/09/fix-invalid-guest-on-virtual-center/

Login to the hosts individually and check to make sure that your storage adapters are connected. I frequently have to reset connections in my SAN and then rescan from vsphere to reestablish links for the guests.
anthonyphamAuthor Commented:
I have tried resetting the filer connection and connecting to the hosts, but when I add them to the inventory, it says unknown (invalid) as the VM.  This includes the vSphere.  
Get your problem seen by more experts

Be seen. Boost your question’s priority for more expert views and faster solutions

Paul SolovyovskySenior IT AdvisorCommented:
Had a Netapp panic this morning as well...fun stuff.

1.  First make sure that you can browse the datastores or nfs mounts.
2.  Login to the host and do the same (using VI client)

If you can than that's good, all you need to do is remove from inventory and browse the nfs mount or datastore and add it back in.

anthonyphamAuthor Commented:
1.  We can browse our datastores and NFS mounts
2.  We can log into the host using VI client.
We tried removing it from inventory and browsing the nfs amount to add it.  However, when it's added, it says Unknown (invalid).  We have older VMs on direct storage and it doesn't say unknown on those.  This seems to only occur on VMs on the filer.  These VMs were setup with Vsphere 4.  The direct storage VMs weren't associated with Vsphere before.

Paul SolovyovskySenior IT AdvisorCommented:
see if you can unmount the nfs mount and re-add back in.  Since it's NFS you don't have any re-signature issues.

If you have another ESX/ESXi host sharing the same mount you should remove from inventory and re-register to the second host.  Try to run, if it works then you can just vmotion over to he first host
anthonyphamAuthor Commented:
I'm still waiting for VMWare support.  But is it possible to recreate the vxm file?  Would this fix the issue?
Paul SolovyovskySenior IT AdvisorCommented:
It is.   What I normally do, just in case

1.  Create a VM with no hard disks or a dummy hard disk
2.  Copy a vmdk file and vmdk-flat file to the new folder.
3.  Add exisiting hard drive ot the VM

Paul SolovyovskySenior IT AdvisorCommented:
Not sure if it will resolve issue but give a small VM a try.  

Have you tried to remove and add the VM from inventory localy on one of the hosts or on vCenter only?
anthonyphamAuthor Commented:
We had two ESXi hosts and vCenter was running on one of them.  We tried adding and removing VMs locally, but it was still invalid...  This was a good lesson to not have vCenter or the vCenter DB reside with all your other VMs!

Though we wanted to have the latest data, we ended up just restoring our database and vCenter by copying it from the .snapshot in Netapp.  It was from 5PM from the earlier day.  After I moved it, I reconnected our VMs.

Thanks for the help everyone!
Paul SolovyovskyConnect With a Mentor Senior IT AdvisorCommented:
A good suggestion is to have the DB on a physical SQL Server since you can install vCenter on pretty much any server in under 30 min.  We have also been placing a non authorotative dns server to replicate dns to vcenter so if you have domain controllers go down you'll have the entries dynamically (vs creating host files entries).

Good to hear that you were able to use snapmanager for VI (my assumption) to resovle the issue.

BTW, make sure you use the netapp plug-in into virtual center, gives you great insight.

You can browse the storage, excxellent, try a rescan a few times more, it often needs this.. But see the following as well:

You stated "However, when it's added, it says Unknown (invalid).  We have older VMs on direct storage and it doesn't say unknown on those."

What client are you using?  If you attempt to access vSPhere VMs with an older client you will recieve this message while trying to add them to the inventory, verify you are using the corrent client, connect the ESX web page http://ESXHOSTNAME and install it.  
All Courses

From novice to tech pro — start learning today.