My VMs aren't booting up from one of my ESX/i servers


I was doing some updating and had to shut down and reboot two VMs on one of my ESX/i servers. When I tried to start my VMs by hitting the green power on button on top of my VCenter server, a pop-up came up saying, Insufficient resources to satisfy configured failover level for HA.

I'm not sure what happened, so I was wondering if any EE can help me.


nimdatx VMs Error
Jaime CamposAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Danny McDanielClinical Systems AnalystCommented:
guessing that those red x's mean that HA is unconfigured on two of your hosts.  check their summary page to confirm, then right-click on each and select reconfigure for HA
Paul SolovyovskySenior IT AdvisorCommented:
how many hosts do you have?  If you are sure you have available resource you can disable admission control in HA under cluster properties
I have had that happen and the easiest way I have found to get around it is to

1. Select the cluster, edit settings, uncheck HA
2. Wait - it will take a short while to disable the agent on all of the hosts
3. Select the cluster, edit settings, check HA to re-enable
4. Wait - it will take a short while to enable the agent on all of the hosts

If you still get the message we will look into your HA settings. Why is the error showing on two hosts? Do you have anything showing up in alerts?

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Making Bulk Changes to Active Directory

Watch this video to see how easy it is to make mass changes to Active Directory from an external text file without using complicated scripts.

Jaime CamposAuthor Commented:
I went to summary page for ESX1 and ESX3 and selected reconfigure for HA. ESX1 went oriange and then back to red with error on top of page. ESX3 turned oriange, however I still cannot power on VMs on ESX3.

See picture.
 ESX errors
Did you try disabling on the entire cluster, then re-enabling?
Jaime CamposAuthor Commented:
See picture
 ESX Alarms
Jaime CamposAuthor Commented:
I'll try, one second.
Jaime CamposAuthor Commented:
It worked. VMs are up.I still get oriange alarm and I'm not sure why. VMs
Danny McDanielClinical Systems AnalystCommented:
the yellow ! marks are for the datastore usage alarms
Jaime CamposAuthor Commented:
My VMs datastore are primary a SAN with enough space, except my VCenter server I left on physical datastore/HD on ESX server 1. So what exactly does this mean? How do I fix this alarm?


Jaime CamposAuthor Commented:
Also can someone explain why I got errors/alarms in the first place?
I have often seen the yellow if no redundancy is configured on management network. Check the summary tab for your hosts and see if there is a message in yellow on that page.
Also how much total space and free space do you show on datastore1 and datastore2?
Danny McDanielClinical Systems AnalystCommented:
it's the local datastores on the hosts that are triggering the alarms.
You get the warning yelloe alarm if more than (default) 75% percentage of the space is used, and a red Alert (again by defualt) more than 85% - VMware likes to reserve some space for snapshots and such if you create them.

You can adjust those thresholds if you want. This particular alarm is defined in the virtual center object so applies to every host and cluster managed by the vCenter Server. Select the vCenter server, alarm tabs, definitions, Right click and edit settings, triggers tab and change to what you want.
And yes, danm66 is correct about it looking like it is local datastores. You should really open another question on alarms as it is kind of out of scope for "My VMs not booting up..." resolved to be a problem with getting HA to configure.
Listen up - just because VMware's configured performance algorithms say you can't power up due to insufficient resources, doesn't mean you can't most of the time.  Within the properties of HA there is a setting to disable the function to prevent machines from powering on if they violate resource availability, its called adminssion control..  its right in front of you when looking ay HA properties.  When you disable adminssion control, all of the options in the panel below that allow you to get granular in regards to which failover, like how many hosts and what % and whether or not you want to designate a specific host for failover..  Once this is disabled HA will attempt to power on all VMs regardless of the resources available in the cluster.  If are totally overloaded then you might want to keep the setting enabled, then set restart priorities for VMs in your HA properties., set it up so that your least important VMs power on last (which could mean not at all).  

Getting real:   if HA actually had a failover event, you wouldn't want to trust that all of your applications were running properly without verifying them would you?  NO.   So why not just let the VMs power up and then as you verify your applications you can then also determine whether or not each application is getting enough resources to funciton.. Slow is better than not at all right?   HA is great for powering the crashed VMs back on rapidly, so if your at lunch or its after work hours and there is a failover event you can be assured the VMs will power on, but regardless you still need to answer you email alert that gets generated from the crash and validate that your apps are up.  I personally perfer to make the decision as to whether I see the ESX servers have enough resources to handle the workloads of the failed VMs.  

Best practice, always be sure your hosts have enough resources to handle a failover event.  if you have 2 hosts and both hosts are running at 80% CPU and memory usage , then you won;'t have enough capacity, its a no brainer.. make sure if you are running 2 hosts that each host stays below 50% utilization, at least on CPU, memory can be over committed but your performance will be slower if ESX starts having to swap memory to satisfy the mem needs of VMs.  If you have 3 hosts, then make sure you don't have more than 66% usage on each host... makes sense?    Take control of your environment, don't trust VMware with everything.  

DATASTORES - don't run VMs on local disk, HA won't work for them, if ESX crashes the VMs stored on local disk are dead and you won't get them back till you either fix your issue or restore them from backups.  Keep your use of local datastores to a minimum.  ALSO, keep in mind each time a VM powers on it creates a file the same as as its allocated memory.  If the VM has 4GB of memory a file 4GB will get created each time the VM is powered on unless you create a memory reservation for the VM.  IF you create a memory reservation, the size of the VM swap file that gets created will be memory size - reservation size.  ALSO - avoid thin provisioning, it wouldn't surprise me if this is what you have done and as a result lost track of your actual disk usage..  plus, keeping your disks eager zeroed thick will provide a slight perf boost in regards to writes to the disk.    
Jaime CamposAuthor Commented:
Great Explaination VMwareGuy!!!!!

Two questions:

1. Where would I move data from one physical store to another physical datastore when I'm running low on a space???? I have my vCenter server on a physical datastore. I understand the risk.
2. What do I look at within VCenter to ensure I'm under 50% utlization on my HOST? Honestly, I see performance tab, however I don't clearly understand what it all means. Maybe you can explain for me?


Look at the summary tab for the cluster and you will find the CPU and Memory available for all hosts in the cluster. Next look at the summary tab for each host under resources and you will see a number and a graph for cpu and memory actually being used. Sum those numbers up and you will get the total utilization of resources on teh cluster. By default Admission Control (first mentioned by paulsolov) will prevent if the current values of the amounts for each host would exceed 75% of the capacity of the cluster. With admission control disabled no such check is made.

If you look at the performance tab on the cluster under overview you will see two graphs, select 1 day and it will get the average over the last 24 hours. On each graph there will be two lines, the top line is the total capacity and the bottom is the actual utilization over the time period. It is easy to tell at a glance whether you are using 50% or 66% of the resources. If the bottom line gets to 75% than if you haven't turned admission control off, or adjusted the default threshold vmware will prevent powering up additional virtual machines on the cluster.

In this thread your issue was that HA needed to be reinitialized on the entire cluster before it would allow you to power up everything.

Use storage vmotion (easiest if you are licensed for it) to migrate vms amongst datastores. Otherwise the free VMware converter or Veeam FastSCP can be used to copy them. (So can the datastore browser, but that is dog slow)
Jaime CamposAuthor Commented:
GREAT!!!! I'm getting it. The light bulb came on....dim, but it's coming on. ;-)

You know I started this project two weeks ago and I couldn't have done it without everyones help from EE.
I use the host's performance tab \ advanced button and then modify chart options to see what is going on, there is an expanation for each counter just like in MS Perfmon.  
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.