VMware VMs high CPU usage and Ballooning

Stan Novinsky
Stan Novinsky used Ask the Experts™
on
We have an ESXi 6.0 host with vCenter and about 230 VMs.  
About 120 are active with a mix of Windows and Linux
I don't have the exact specs, but I believe it is an intel quad core with 512 GB of RAM.

The other day i was looking at the perf monitor and noticed ballooning occurring, which we all know is an indicator of the host running low on RAM.
I also noticed 4-5 VMs showing over 4000 MHz on the host VM tab.

If i remember, each VM has a balloon driver called vmmemctl.sys which is available as part of VMware Tools.
When ESXi runs low on physical RAM, It uses the ballooning driver to determine the how much memory from virtual machines can be get back to prevent the hypervisor swapping.

My question is, what happens if the VMs don't have VMware tools installed?
How is ballooning then handled?

The reason i ask, i see a several VMs on the system without VMware tools running.
Could this be one issue causing high RAM and CPU usage?
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
David FavorFractional CTO
Distinguished Expert 2018

Commented:
If you really mean the machine running your VMs only has "512 GB of RAM" total for the entire machine...

This simply won't work.

You'll end up with massive disk thrash related to swapping, which will likely cause what you're calling ballooning.

Start with a physical machine with a minimum of 32G RAM, which will be sufficient to run your machine OS + maybe a few VMs, depending on exactly what's running inside your VMs.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
My question is, what happens if the VMs don't have VMware tools installed?
How is ballooning then handled?

It cannot activate the balloon driver for that VM,
Stan NovinskyVirtualization Engineer

Author

Commented:
512 GB is the ESXi Server total ....we have used less RAM on other ESXi Servers for a lower number of VMs and did not have issues...


If it cannot activate the driver, then i assume swapping occurs?

Is it better to have ballooning (with VMware tools installed)  or no VMware tools and swapping ?

Sounds like both are not good,,, but which would be the least of two evils..
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
VMware Tools is not optional and should be installed on all VMs.

However the ballon driver is generally installed on all VMs, unless there is good reason to remove it, because of high memory usage virtual machines e.g. Remote Desktop Servers, Citrix Application Servers, that generally are 100% Memory Bound for good reason due to concurrent user usage.

The balloon driver is only used when the host is running out of memory, and to stop the Host swapping to disk because it's running short of memory, so inflates the balloon driver to reclaim memory from the VM, and thus prevents the Host swapping to disk.

and thus this memory reclaimed from VMs is given to other VMs.

I recommend it's better to have the balloon driver installed so memory can be reclaimed from other VMs, to prevent host swapping, when this starts to happen, CPU will then start getting high, due to swap activity.
Stan NovinskyVirtualization Engineer

Author

Commented:
interesting..... your quote
 ".. to prevent host swapping, when this starts to happen, CPU will then start getting high, due to swap activity.."

this then may why we see the ESXi CPU hitting 98%?
VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017
Commented:
Yes, quite possibly swapping can bring a host to it's knees, as it's slow.... and it will have a detrimental effect on everything, and in bad cases it can prevent even management, e.g. not responding, SSH sessions, Web Client sessions, vCenter Server not responding sessions, or disconnected, loss of storage, pings etc...

and to regain control shutting down VMs via RDP/SSH can be enough to give the Host OS enough resource to regain control.
David FavorFractional CTO
Distinguished Expert 2018

Commented:
Tip: The only real bottle neck of any system is disk i/o, which can dramatically increase CPU usage as CPUs attempt to handle all disk i/o required.

Check your swapping + if swapping is occurring, then increase VM RAM for each instance till swapping stops.
Stan NovinskyVirtualization Engineer

Author

Commented:
thanks,,,,,,we have noticed that  CPU usage is at or near 95% and sometimes 100%.  

If VMs are not using VMware tools, then the balloon driver is not being used and swapping may be taking place and driving the CPU up.

I will need to check this,,,Disk I/O was not an issue to my knowledge as I recall we are using SSD drives

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial