Extreme sluggishness on VMware ESXi 5.1.0 server running four virtual servers

Hello all:

I am taking over management of a VMware ESXi host (the hypervisor runs off an internally-connected USB drive) that is running four virtual servers.

The physical machine is a Dell PowerEdge R720 server with 2 Xeon E5-2630 CPUs @2.30 GHz, 32 GB of RAM, and one RAID-5 2 TB array.

All four virtual servers are sharing the RAID-5 array which is made up of five 600 GB SAS drives plus another 600 GB hot spare.

The four virtual servers are the following:
   1 - Server 2012 domain controller/file server
   2 - Server 2012 Exchange 2010 server
   3 - Server 2003 R2 MS SQL server
   4 - Server 2003 application server

The performance of all four virtual servers either through the vSphere Client console or direct remote connection is extremely sluggish.

The Exchange 2010 server in particular is experiencing severe performance problems with slow and intermittent Outlook Web Access from both inside and outside the company network, slow Outlook access, etc.

Has any Experts Exchange members come across a situation like this before?

How can I pinpoint which components of the system is causing the sluggishness (bottleneck)?
KYiin-ComputerMDAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
What is the total memory allocated to the VMs.

What is the allocated of vCPUs to the VMs.

Have you looked at the performance, using vCenter Server, or ESXi, see my EE Article

HOW TO:  Performance Monitor vSphere 4.x or 5.0
KYiin-ComputerMDAuthor Commented:
Hello Andrew:

Thank you for responding!

I will provide the answers using your article as soon as possible.
KYiin-ComputerMDAuthor Commented:
Hello Andrew:

I apologize for the delay in response.

The physical host has 32 GB (32722.48 MB) of RAM and 2 Intel Xeon E5-2630 CPUs (2 processor sockets, 6 cores per socket, 24 total logical processors).

As of 3:35 PM today (04/20/2015), the host CPU usage is 492 MHz (almost nonexistent on the usage bar) and the memory usage is 30921.00 MB (almost completely filling up the usage bar).

The VMs have the following RAM and vCPU allocations:
   Server 2012 - Roles: file server/domain controller
     6 vCPUs, 8 GB (8,192 MB) RAM
   Server 2012 - Roles: Exchange 2010
     6 vCPUs, 16 GB (16,384 MB) RAM
   Server 2003 R2 - Roles: SQL server
     4 vCPUs, 4 GB (4,096 MB) RAM
   Server 2003 - Roles: application server for SQL database
     4 vCPUs, 4 GB (4,096 MB) RAM
SolarWinds® VoIP and Network Quality Manager(VNQM)

WAN and VoIP monitoring tools that can help with troubleshooting via an intuitive web interface. Review quality of service data, including jitter, latency, packet loss, and MOS. Troubleshoot call performance and correlate call issues with WAN performance for Cisco and Avaya calls

KYiin-ComputerMDAuthor Commented:
Here is an esxtop screenshot showing the average CPU load:
esxtop vCPU screenshot
KYiin-ComputerMDAuthor Commented:
An esxtop screenshot showing the average RAM load:
RAM-usage.png
KYiin-ComputerMDAuthor Commented:
Storage ("u") -
Storage-usage---u.png
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay, can you increase the memory in your Host Server, or reduce the memory in the VMs ?

You are very close, to the server swapping put to disk, which will cause performance issues.

You have a lot of vCPUs, allocated all over the place here.....

Do you really need all these vCPUs in VMs ?

6 and 4 vCPUs per VM ?

SQL servers, look low on memory, Domain Controller, seems to have a lot of memory, and too many vCPUs.

I would give exchange more memory.
KYiin-ComputerMDAuthor Commented:
Storage usage ("v") -
Storage-usage---v.png
KYiin-ComputerMDAuthor Commented:
Okay, how about if I reconfigure the VMs as per the following?

Domain controller -
     4 vCPUs, 4 GB (4,096 MB) RAM
Exchange 2010 -
     6 vCPUs, 20 GB (20,480 MB) RAM
SQL (32-bit, so can't see more than 4 GB RAM) -
     4 vCPUs, 4 GB (4,096 MB) RAM
Application (32-bit, so can't see more than 4 GB RAM) -
     2 vCPUs, 4 GB (4,096 MB) RAM
KYiin-ComputerMDAuthor Commented:
Network usage -
Network-usage.png
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Check you need the vCPUs, in ALL of the VMs.

Check the vCPUs in Domain Controller, we only have 1 vCPU, and 4GB RAM.

Check vCPUs in SQL, and also check memory.

With those VMs and Memory, you are still out of memory!

Add more to host. 32GB is nothing!
KYiin-ComputerMDAuthor Commented:
Hmm ... after tuning it for several days:

 Domain controller* -
      4 vCPUs, 4 GB (4,096 MB) RAM
 Exchange 2010 -
      6 vCPUs, 20 GB (20,480 MB) RAM
 SQL (32-bit, so can't see more than 4 GB RAM) -
      2 vCPUs, 4 GB (4,096 MB) RAM
 Application (32-bit, so can't see more than 4 GB RAM) -
      2 vCPUs, 2 GB (2,048 MB) RAM

*It is not just DC, it is also running at least one CPU-heavy program on a near-daily basis so one or two vCPUs would not be enough (tested and confirmed).
 
This configuration seems to result in significantly less latency and sluggishness, but I will see if we can get more RAM added (probably not for awhile).
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay, so the physical host has 32GB, we will reserve 2-4GB for the Host Hypervisor, as safeguards.

so this leaves 28GB for all VMs.

You have got:-

4GB
20GB
4GB
2GB

30GB Allocated to VMs.

Your are OVER SUBSCRIBED!

I would think this is the reason for your issues, you are on the memory limit.

1. Increase the memory in the host.

2. reduce the number of VMs.

3. Purchase a new host.

4. Turn off VMs.

5. Reduce VM memory,

Memory is always the bottleneck! before CPU and Storage.

You are low.

Come on, you've allocated 20GB to a VM, and your host only has 32GB!

My local desktop workstation has more memory than your server!
jmcgOwnerCommented:
Can you see what the vSphere client says is the active % of memory in use on the Exchange Server VM? I'd suggest cutting the Exchange VM back to 16GB of memory and 4 vCPUs as an experiment. The should reduce the memory overcommit and let the VM be scheduled more easily.
KYiin-ComputerMDAuthor Commented:
Hello:

I have made the following adjustments to the VMs:

 Domain controller -
      4 vCPUs, 4 GB (4,096 MB) RAM
 Exchange 2010 -
      6 vCPUs, 20 GB (20,480 MB) RAM
 SQL (32-bit, so can't see more than 4 GB RAM) -
      2 vCPUs, 4 GB (4,096 MB) RAM
 Application (32-bit, so can't see more than 4 GB RAM) -
      1 vCPUs, 1 GB (1,024 MB) RAM

I have attached a vSphere Client screenshot of the ESXi host Summary page.
KYiin-ComputerMDAuthor Commented:
Umm ... the vSphere Client screenshot.

(It didn't upload with my previous post for some reason.)

vSphere-Client.png
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Under Memory Usage if that is the current metric!

WAY TO LOW!!!! e.g. very high, and not much free memory left for the Host OS.

The Host ESXi OS will be close to swapping to disk, and all the VMs will be close to swapping to disk, and when that happens, the entire host will crawl.

You need more memory, or suffer the performance issues.

Spend some money, on an upgrade.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
jmcgOwnerCommented:
On that screenshot, it looks like the VMs are doing next to nothing as far as CPU usage.

You could click on the virtual machines tab to see how much use each VM is making of its allocation. That would be informative for where you should go next.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.