ESX shutdown when battery low

I'm assuming that EVERYONE running ESX is using a battery backup.  I'm assuming that nearly everyone is using an APC battery.  I then made the assumption that there would be some sort of support within ESX for safely shutting the system down if running on battery and it's low, but I'm not seeing it.  All I see in searching around are the typical twisted unix command line install instructions for installing various APC utilities directly on my ESX box, thus making me risk bringing the whole thing down.

Am I missing something?  Is there some other way for us people who try to avoid unix to achieve the same results?

gary
garystarkAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

coolsport00Commented:
I don't believe there is a native way to do this without using APC's software, but don't quote me on that. :)

Here is an EE article on this issue (somewhat) Gary:
http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Server/Hypervisor/Q_25103869.html?sfQueryTermInfo=1+apc+esx+power

Make sure to also read the 'last' post that is an "Open Discussion" post. Good material. Not sure if this is what you're asking or not...please let me know if I'm understanding your question.

Regards,
~coolsport00
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
garystarkAuthor Commented:
coolsport00, I actually did read that post before doing my own post.  Clearly I'm not the only one struggling with this presumably common problem.  I don't really see my answer in that post.  I actuallly don't mind shelling out the money for the APC network card, plus ~$100 for the agent install if that's what it is, but my goal is to find a solution that does NOT involve an installation on my ESX4 box.  I'm not a unix guy, too old to learn unix, to wise to do my learning on our production ESX box.

From my perspect this is obviously a hole in the system.  I'd happily pay some 3rd party for a utility that runs (anywhere but on my ESX box) do do the monitoring and communication with ESX to shut it down.  Question is...does such a thing exist?

gary
0
coolsport00Commented:
Gary...I totally agree. And, part of it is 1. USB functionality and 2. $$$$$$$$$$$ (of course) Personally, I think it's downright dumb (I'd use a stronger word there, but for...umm..."professionalism's" sake, I'll refrain) :)

You can certainly await other potential posts/comments, but I don't think you can do this without something installed on the host, unfortunately. :(  It would be nice though, wouldn't it? :)

~coolsport00
0
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

mastooCommented:
This has been on my list to figure out, but I haven't gotten to it yet.  Thanks for the preview.  I have a few random thoughts in case they help: 1) Can't you use the serial port as passthru in ESXi to make it available within the VM?  2) How about usb from the ups to the vm?  3) If a vm (or for that matter any windows box at all) can see the ups, and you use the powerchute software, doesn't it let you run a script when events happen?  ESX has the command-line stuff that can be client installed and I presume one of those commands can initiate a shutdown.
0
garystarkAuthor Commented:
mastoo, I'm specifically looking for a solution that does NOT involve every single VM.  I see the goal as shutting down the ESX box, which in TURN shuts down all the VMs.  We already know that ESX just that just fine.  Besides, if I go direct to the VMs and ignore ESX itself, that still leaves it vulnerable even if the VMs are safe.

btw, I'm using ESX4, not ESXi.

gary
0
mastooCommented:
Yes, that's what I was suggesting.  I'm only familiar with ESXi but I thought any single VM or physical computer could use a command from a script to tell the host to shutdown, and as part of the host's normal shutdown it tells all the VM's to shutdown first.  Like I said though, I'm not very familiar with this so I might be speaking gibberish.
0
IanThCommented:
have you configured the vm's to shutdown when the host shuts down ?
0
garystarkAuthor Commented:
IanTh, the answer is yes.  That part is easy.  The issue is getting the host to shut down on low battery, not the associated VMs.

gary
0
Paul SolovyovskySenior IT AdvisorCommented:
This is what I recieved from a collegue of mine, I don't think he minds if I share.  This is for ESX4 and he ap network card.  He's using it with ESX.  Create file witht the text for each VM, esxshutdown.sh
-------------
#####################################################################
#!/bin/sh
#
#  UPS shutdown script for VMware ESX 3.0.x
#
#
#####################################################################
# set the paths that the vmware tools need
PATH="/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin"


#####################################################################
# try do a nice shutdown of VM if there is power
#
count_vm_on=0
for vm in `vmware-cmd -l` ; do
  #echo "VM: " $vm
  for VMstate in `vmware-cmd "$vm" getstate` ; do
  #echo $VMstate

  # If the VM is power ON
  if [ $VMstate = "on" ] ; then
     echo " "
     echo "VM: " $vm
     echo "State: is on and will now tell it to shut down"
     echo "Shutting down: " $vm
     vmware-cmd "$vm" stop trysoft
     vmwarecmd_exitcode=$(expr $?)
     if [ $vmwarecmd_exitcode -ne 0 ] ; then
         echo "exitcode: $vmwarecmd_exitcode so will now turn it off hard"
         vmware-cmd "$vm" stop hard
     fi
     count_vm_on=$count_vm_on+1
    sleep 2
  # if the VM is power OFF
  elif [ $VMstate = "off" ] ; then
     echo " "
     echo "VM: " $vm
     echo "State: is off, so i skip it"
  # if the VM is power suspended
  elif [ $VMstate = "suspended" ] ; then
     echo " "
     echo "VM: " $vm
     echo "State: is suspended, so i skip it"
  # if state is getstate or =
  else
     printf ""
    #echo "unknown state: " $VMstate
  fi

  done
done


########################################################################
# wait for up to 5 min for the VM to shutdown
#
if [ $count_vm_on = 0 ] ; then
  echo " "
  echo "All VM is off or suspended"
else
  echo " "
  vm_time_out=300
  count_vm_on=0
  echo "Waiting for VMware virtual machines."
  for (( second=0; second<$vm_time_out; second=second+5 )); do
     sleep 5
     printf "."
     count_vm_on=0
     for vm in `vmware-cmd -l` ; do
        for VMstate in `vmware-cmd "$vm" getstate` ; do
        if [ $VMstate = "on" ] ; then
           count_vm_on=$(expr $count_vm_on + 1)
        fi
     done
  done
  if [ $count_vm_on = 0 ] ; then
     #echo "exit for"
     break
  fi
  done
  #echo $VMstate
fi
#echo $count_vm_on


#####################################################################
# checking if all the VM are off and if not then turn them off
for vm in `vmware-cmd -l` ; do
  #echo "VM: " $vm
  for VMstate in `vmware-cmd "$vm" getstate` ; do
    # If the VM is power ON
    if [ $VMstate = "on" ] ; then
      echo " "
      echo "Found this VM: " $vm
      echo "it is still on but now i will turn it off"
      vmware-cmd "$vm" stop hard
      sleep 2
    fi
  done
done


#####################################################################
# Will now shutdown the ESX server as all VM are now off
echo " "
echo "All VMs have been powered down.  Powering down ESX host now."
echo " "
shutdown -h now

-------------------------------------

I just tested the APC agent with ESX 4.0 and it worked great.  After the install take the esxshutdown.txt file and upload it to /usr/local/bin/PowerChute/bin and rename it to a .sh file.  Do a ‘chmod u+x’ on it to make it executable.  On a server you are afford to shutdown, test it with ‘./esxshutdown.sh’.  It should enumerate running VMs and go through a series of checks to try to softly shut them down before forcing them to quit.  If you run into issues where it won’t run I’ve gotten around that by editing the file in nano, putting in an extra carriage return at the top of the file and saving it…that seems to fit whatever the issue is that it complains about (likely the file conversion from a DOS formatted file to a native shell script happens when anything in it is changed and saved).

 

As long as you get that far you can go to the esx server (http://<serverip>:3052 ) and login and configure your chosen events to “Run a Script” (rather than “Shutdown”).  In the script path just put “/usr/local/bin/PowerChute/esxshutdown.sh” and you’re set!  I usually do this when “On Battery” and “Communication Failure While On Battery” events happen.

 

Just make sure that your DRS cluster is not set to power on machines, or it will detect a host going down as a failure and try to VMotion the VMs to another host (which, in the event of a power failure means that all hosts will be headed south).

 

This may be a solution for customers like me who can only afford smaller single UPSs, so each esx server can monitor its own UPS as well as the UPS that powers a SAN or shared storage device



Hope this helps
0
garystarkAuthor Commented:
paulsolov, why on earth would I need a script for shutting down each individual VM?  If you shut down the ESX system itself, it does a fine job of shutting down all the VMs.  Secondly, this involves editing unix scripts...exactly what I'm trying to avoid.

gary
0
Paul SolovyovskySenior IT AdvisorCommented:
Think about this way..

The VMs are running, the Network Shutdown only knows about ESX, it will shutdown the ESX host without shutting down the VMs first.  The script ensures that the VMs are shutdown first in a organized fashion

Another solution is to put he network shutdown in the VMs themselved. If UPS starts to show low power, the VMs will shutdown and the ESX host will die once the battery is low which is usually not that bad even with a re-install would take about 30-40min.  This allowts you keep it simple but adds some risk.
0
garystarkAuthor Commented:
paulsolov, this could easily be my misunderstanding, but bear with me.  Do you agree that ESX is designed to shut down all of it's VMs prior to shutting down itself?  If you agree, why would you need another mechanism to do so, especially when it involves the extra work of configuring each VM separately?

You are referring to "network shutdown".  I'm wondering if we're taking two different things.  I'm referring to a power outage situation where the ESX server is running off the APC battery.  At some point the battery gets too low and, to be safe, we want to shut down the entire system (ESX & VMs) in a clean fashion rather than effectively just pulling the plug.  And since ESX is designed to shut it's VMs down before shutting itself down, I only need to find a way of telling ESX to shut down.  And finding a way to tell ESX to shut down when the battery is low is the problem I'm trying to solve.

gary
0
Paul SolovyovskySenior IT AdvisorCommented:
When the APC Powerchute shuts down the ESX server it does not know anything about ESX servers, the tool is primarily a Linux shutdown application and only shuts down the ESX host without any considerations for the VMs running.  In this case it would shut down the VMs as if it was pulling the plug on them.  

This would be the case if you're using a direct connection to the ESX host or network card.  In this case you would need to script the VMs to shutdown the VMs before the host.

APC has a vmware add on but I haven't been able to gauge if it actually works, we're an APC partner and I have been trying to get their tech team to tell me how I can shutdown the VMs prior to the host but they haven't been able to give me a straight answer.

Hope this helps.
0
garystarkAuthor Commented:
Paulsolov, do you agree that if I manually give ESX a command to shut down, it automatically shuts down its VMs first?

gary
0
mastooCommented:
Trick question - the vm's need vmtools installed and need to be set to shut down :-)
0
garystarkAuthor Commented:
mastoo, that's certainly true...you need to have vmtools installed on each VM for ESX to do a clean shutdown, but I'm not trying to make a trick question out of this.  I suppose I just assumed that most people using ESX did the same, not just for shutdown, but for the other advantages the tools provide.  So allow me to revise the question...please assume that I do in fact have vmtools installed on each VM.

gary
0
Paul SolovyovskySenior IT AdvisorCommented:
If you shutdown the ESX host it does not shutdown the VMs.  If you have a cluster it will ask you if you want to shutdown the VMs and/or migrate.  This action does not take place if the APC Powershute initiates shutdown.

To validate this go into the service console and use the shutdown command.
0
Paul SolovyovskySenior IT AdvisorCommented:
Let me clarify.  If you shutdown via VI Client it will ask you.  If you do it via the service console shutdown command which is essentially what the APC is doing it will not, this is why you have to script it which is what the VI client is doing in the backgroung.
0
garystarkAuthor Commented:
paulsolov, I wasn't aware that shutting down an ESX box via the service console worked differently, specifically that it would just kill the VMs.  I wouldn't have expected that.  But the fact remains...it IS possible to manually shut down an ESX box such that the VMs are shut down properly, like through the VI Client as you mention.  I just want to achieve this same thing in an automated way.

Geez people...is it REALLY so much to ask?  I just want the damn ESX box to do a "proper" shutdown when the battery is low.  It shouldn't be that complicated.  EVERYBODY needs to do this.  Am I to understand that the vmware engineers never thought of this need?  Given how easy it is for me, a non-unix guy, to get an ESX box up and into production, this seems like a glaring oversight.  And the APC engineers really seem asleep at the wheel on this one as well.  I want an appliance, not a hobby.

This question obvously isn't going will.  I'm beginning to believe that this is another of those "you can't get there from here" IT nightmares.  No wonder they pay us the "big bucks".

gary
0
Paul SolovyovskySenior IT AdvisorCommented:
From my understanding what you want to do IS possible with the script provided above, otherwise not.  Approx 50% of customers we deal with have a generator that comes online so it it's not an issue.  With other customers we just shutdown the VMs and let the ESX shutdown hard.
0
garystarkAuthor Commented:
My initial question stipulated that I wanted a solution that does not involve scripting in unix, so by that criteria my conclusion so far is that my goal is not possible.  I was hoping someone either had a trick (maybe using a VM monitoring the battery & sending a powershell command to shutdown ESX) or some third party product recommendation.  I'll wait a week, then close this question, splitting the points between those of you who were kind enough to reply.

gary
0
garystarkAuthor Commented:
Bottom line is that I did not get what I want, but I'm awarding points for the help I was offered.  Much thanks.

gary
0
coolsport00Commented:
Thanks Gary; this is a pickle to be sure. I don't think APC/VMware will allow this to be a seamless/incorporated technology anytime soon since $$$ is involved :(

Regards,
~coolsport00
0
garystarkAuthor Commented:
coolsport00, how so?  I don't see that vmware and APC are competitors...

gary
0
coolsport00Commented:
Well, you'd think the technology to incorporate APC would be within ESX natively, but it isn't. You have to purchase separate licensing, etc (based off the link I provided in my 1st post). That's what I was referring to. Not that they're competitors, but that VMware won't incorporate APC within it's hypervisor natively. And that is more than likely because APC is a partner of Vmware and Vmware is doing APC a financial favor. That's my own personal assumption though. :)

~coolsport00
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Components

From novice to tech pro — start learning today.