VMware VMs high CPU usage and Ballooning

We have an ESXi 6.0 host with vCenter and about 230 VMs.  
About 120 are active with a mix of Windows and Linux
I don't have the exact specs, but I believe it is an intel quad core with 512 GB of RAM.

The other day i was looking at the perf monitor and noticed ballooning occurring, which we all know is an indicator of the host running low on RAM.
I also noticed 4-5 VMs showing over 4000 MHz on the host VM tab.

If i remember, each VM has a balloon driver called vmmemctl.sys which is available as part of VMware Tools.
When ESXi runs low on physical RAM, It uses the ballooning driver to determine the how much memory from virtual machines can be get back to prevent the hypervisor swapping.

My question is, what happens if the VMs don't have VMware tools installed?
How is ballooning then handled?

The reason i ask, i see a several VMs on the system without VMware tools running.
Could this be one issue causing high RAM and CPU usage?
Stan NovinskyVirtualization Engineer Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

David FavorLinux/LXD/WordPress/Hosting SavantCommented:
If you really mean the machine running your VMs only has "512 GB of RAM" total for the entire machine...

This simply won't work.

You'll end up with massive disk thrash related to swapping, which will likely cause what you're calling ballooning.

Start with a physical machine with a minimum of 32G RAM, which will be sufficient to run your machine OS + maybe a few VMs, depending on exactly what's running inside your VMs.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
My question is, what happens if the VMs don't have VMware tools installed?
How is ballooning then handled?

It cannot activate the balloon driver for that VM,
Stan NovinskyVirtualization Engineer Author Commented:
512 GB is the ESXi Server total ....we have used less RAM on other ESXi Servers for a lower number of VMs and did not have issues...

If it cannot activate the driver, then i assume swapping occurs?

Is it better to have ballooning (with VMware tools installed)  or no VMware tools and swapping ?

Sounds like both are not good,,, but which would be the least of two evils..
Your Guide to Achieving IT Business Success

The IT Service Excellence Tool Kit has best practices to keep your clients happy and business booming. Inside, you’ll find everything you need to increase client satisfaction and retention, become more competitive, and increase your overall success.

Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
VMware Tools is not optional and should be installed on all VMs.

However the ballon driver is generally installed on all VMs, unless there is good reason to remove it, because of high memory usage virtual machines e.g. Remote Desktop Servers, Citrix Application Servers, that generally are 100% Memory Bound for good reason due to concurrent user usage.

The balloon driver is only used when the host is running out of memory, and to stop the Host swapping to disk because it's running short of memory, so inflates the balloon driver to reclaim memory from the VM, and thus prevents the Host swapping to disk.

and thus this memory reclaimed from VMs is given to other VMs.

I recommend it's better to have the balloon driver installed so memory can be reclaimed from other VMs, to prevent host swapping, when this starts to happen, CPU will then start getting high, due to swap activity.
Stan NovinskyVirtualization Engineer Author Commented:
interesting..... your quote
 ".. to prevent host swapping, when this starts to happen, CPU will then start getting high, due to swap activity.."

this then may why we see the ESXi CPU hitting 98%?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Yes, quite possibly swapping can bring a host to it's knees, as it's slow.... and it will have a detrimental effect on everything, and in bad cases it can prevent even management, e.g. not responding, SSH sessions, Web Client sessions, vCenter Server not responding sessions, or disconnected, loss of storage, pings etc...

and to regain control shutting down VMs via RDP/SSH can be enough to give the Host OS enough resource to regain control.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
David FavorLinux/LXD/WordPress/Hosting SavantCommented:
Tip: The only real bottle neck of any system is disk i/o, which can dramatically increase CPU usage as CPUs attempt to handle all disk i/o required.

Check your swapping + if swapping is occurring, then increase VM RAM for each instance till swapping stops.
Stan NovinskyVirtualization Engineer Author Commented:
thanks,,,,,,we have noticed that  CPU usage is at or near 95% and sometimes 100%.  

If VMs are not using VMware tools, then the balloon driver is not being used and swapping may be taking place and driving the CPU up.

I will need to check this,,,Disk I/O was not an issue to my knowledge as I recall we are using SSD drives
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.