KYiin-ComputerMD
asked on
Extreme sluggishness on VMware ESXi 5.1.0 server running four virtual servers
Hello all:
I am taking over management of a VMware ESXi host (the hypervisor runs off an internally-connected USB drive) that is running four virtual servers.
The physical machine is a Dell PowerEdge R720 server with 2 Xeon E5-2630 CPUs @2.30 GHz, 32 GB of RAM, and one RAID-5 2 TB array.
All four virtual servers are sharing the RAID-5 array which is made up of five 600 GB SAS drives plus another 600 GB hot spare.
The four virtual servers are the following:
1 - Server 2012 domain controller/file server
2 - Server 2012 Exchange 2010 server
3 - Server 2003 R2 MS SQL server
4 - Server 2003 application server
The performance of all four virtual servers either through the vSphere Client console or direct remote connection is extremely sluggish.
The Exchange 2010 server in particular is experiencing severe performance problems with slow and intermittent Outlook Web Access from both inside and outside the company network, slow Outlook access, etc.
Has any Experts Exchange members come across a situation like this before?
How can I pinpoint which components of the system is causing the sluggishness (bottleneck)?
I am taking over management of a VMware ESXi host (the hypervisor runs off an internally-connected USB drive) that is running four virtual servers.
The physical machine is a Dell PowerEdge R720 server with 2 Xeon E5-2630 CPUs @2.30 GHz, 32 GB of RAM, and one RAID-5 2 TB array.
All four virtual servers are sharing the RAID-5 array which is made up of five 600 GB SAS drives plus another 600 GB hot spare.
The four virtual servers are the following:
1 - Server 2012 domain controller/file server
2 - Server 2012 Exchange 2010 server
3 - Server 2003 R2 MS SQL server
4 - Server 2003 application server
The performance of all four virtual servers either through the vSphere Client console or direct remote connection is extremely sluggish.
The Exchange 2010 server in particular is experiencing severe performance problems with slow and intermittent Outlook Web Access from both inside and outside the company network, slow Outlook access, etc.
Has any Experts Exchange members come across a situation like this before?
How can I pinpoint which components of the system is causing the sluggishness (bottleneck)?
ASKER
Hello Andrew:
Thank you for responding!
I will provide the answers using your article as soon as possible.
Thank you for responding!
I will provide the answers using your article as soon as possible.
ASKER
Hello Andrew:
I apologize for the delay in response.
The physical host has 32 GB (32722.48 MB) of RAM and 2 Intel Xeon E5-2630 CPUs (2 processor sockets, 6 cores per socket, 24 total logical processors).
As of 3:35 PM today (04/20/2015), the host CPU usage is 492 MHz (almost nonexistent on the usage bar) and the memory usage is 30921.00 MB (almost completely filling up the usage bar).
The VMs have the following RAM and vCPU allocations:
Server 2012 - Roles: file server/domain controller
6 vCPUs, 8 GB (8,192 MB) RAM
Server 2012 - Roles: Exchange 2010
6 vCPUs, 16 GB (16,384 MB) RAM
Server 2003 R2 - Roles: SQL server
4 vCPUs, 4 GB (4,096 MB) RAM
Server 2003 - Roles: application server for SQL database
4 vCPUs, 4 GB (4,096 MB) RAM
I apologize for the delay in response.
The physical host has 32 GB (32722.48 MB) of RAM and 2 Intel Xeon E5-2630 CPUs (2 processor sockets, 6 cores per socket, 24 total logical processors).
As of 3:35 PM today (04/20/2015), the host CPU usage is 492 MHz (almost nonexistent on the usage bar) and the memory usage is 30921.00 MB (almost completely filling up the usage bar).
The VMs have the following RAM and vCPU allocations:
Server 2012 - Roles: file server/domain controller
6 vCPUs, 8 GB (8,192 MB) RAM
Server 2012 - Roles: Exchange 2010
6 vCPUs, 16 GB (16,384 MB) RAM
Server 2003 R2 - Roles: SQL server
4 vCPUs, 4 GB (4,096 MB) RAM
Server 2003 - Roles: application server for SQL database
4 vCPUs, 4 GB (4,096 MB) RAM
ASKER
Here is an esxtop screenshot showing the average CPU load:
ASKER
An esxtop screenshot showing the average RAM load:
ASKER
Storage ("u") -
Okay, can you increase the memory in your Host Server, or reduce the memory in the VMs ?
You are very close, to the server swapping put to disk, which will cause performance issues.
You have a lot of vCPUs, allocated all over the place here.....
Do you really need all these vCPUs in VMs ?
6 and 4 vCPUs per VM ?
SQL servers, look low on memory, Domain Controller, seems to have a lot of memory, and too many vCPUs.
I would give exchange more memory.
You are very close, to the server swapping put to disk, which will cause performance issues.
You have a lot of vCPUs, allocated all over the place here.....
Do you really need all these vCPUs in VMs ?
6 and 4 vCPUs per VM ?
SQL servers, look low on memory, Domain Controller, seems to have a lot of memory, and too many vCPUs.
I would give exchange more memory.
ASKER
Storage usage ("v") -
ASKER
Okay, how about if I reconfigure the VMs as per the following?
Domain controller -
4 vCPUs, 4 GB (4,096 MB) RAM
Exchange 2010 -
6 vCPUs, 20 GB (20,480 MB) RAM
SQL (32-bit, so can't see more than 4 GB RAM) -
4 vCPUs, 4 GB (4,096 MB) RAM
Application (32-bit, so can't see more than 4 GB RAM) -
2 vCPUs, 4 GB (4,096 MB) RAM
Domain controller -
4 vCPUs, 4 GB (4,096 MB) RAM
Exchange 2010 -
6 vCPUs, 20 GB (20,480 MB) RAM
SQL (32-bit, so can't see more than 4 GB RAM) -
4 vCPUs, 4 GB (4,096 MB) RAM
Application (32-bit, so can't see more than 4 GB RAM) -
2 vCPUs, 4 GB (4,096 MB) RAM
ASKER
Network usage -
Check you need the vCPUs, in ALL of the VMs.
Check the vCPUs in Domain Controller, we only have 1 vCPU, and 4GB RAM.
Check vCPUs in SQL, and also check memory.
With those VMs and Memory, you are still out of memory!
Add more to host. 32GB is nothing!
Check the vCPUs in Domain Controller, we only have 1 vCPU, and 4GB RAM.
Check vCPUs in SQL, and also check memory.
With those VMs and Memory, you are still out of memory!
Add more to host. 32GB is nothing!
ASKER
Hmm ... after tuning it for several days:
Domain controller* -
4 vCPUs, 4 GB (4,096 MB) RAM
Exchange 2010 -
6 vCPUs, 20 GB (20,480 MB) RAM
SQL (32-bit, so can't see more than 4 GB RAM) -
2 vCPUs, 4 GB (4,096 MB) RAM
Application (32-bit, so can't see more than 4 GB RAM) -
2 vCPUs, 2 GB (2,048 MB) RAM
*It is not just DC, it is also running at least one CPU-heavy program on a near-daily basis so one or two vCPUs would not be enough (tested and confirmed).
This configuration seems to result in significantly less latency and sluggishness, but I will see if we can get more RAM added (probably not for awhile).
Domain controller* -
4 vCPUs, 4 GB (4,096 MB) RAM
Exchange 2010 -
6 vCPUs, 20 GB (20,480 MB) RAM
SQL (32-bit, so can't see more than 4 GB RAM) -
2 vCPUs, 4 GB (4,096 MB) RAM
Application (32-bit, so can't see more than 4 GB RAM) -
2 vCPUs, 2 GB (2,048 MB) RAM
*It is not just DC, it is also running at least one CPU-heavy program on a near-daily basis so one or two vCPUs would not be enough (tested and confirmed).
This configuration seems to result in significantly less latency and sluggishness, but I will see if we can get more RAM added (probably not for awhile).
Okay, so the physical host has 32GB, we will reserve 2-4GB for the Host Hypervisor, as safeguards.
so this leaves 28GB for all VMs.
You have got:-
4GB
20GB
4GB
2GB
30GB Allocated to VMs.
Your are OVER SUBSCRIBED!
I would think this is the reason for your issues, you are on the memory limit.
1. Increase the memory in the host.
2. reduce the number of VMs.
3. Purchase a new host.
4. Turn off VMs.
5. Reduce VM memory,
Memory is always the bottleneck! before CPU and Storage.
You are low.
Come on, you've allocated 20GB to a VM, and your host only has 32GB!
My local desktop workstation has more memory than your server!
so this leaves 28GB for all VMs.
You have got:-
4GB
20GB
4GB
2GB
30GB Allocated to VMs.
Your are OVER SUBSCRIBED!
I would think this is the reason for your issues, you are on the memory limit.
1. Increase the memory in the host.
2. reduce the number of VMs.
3. Purchase a new host.
4. Turn off VMs.
5. Reduce VM memory,
Memory is always the bottleneck! before CPU and Storage.
You are low.
Come on, you've allocated 20GB to a VM, and your host only has 32GB!
My local desktop workstation has more memory than your server!
Can you see what the vSphere client says is the active % of memory in use on the Exchange Server VM? I'd suggest cutting the Exchange VM back to 16GB of memory and 4 vCPUs as an experiment. The should reduce the memory overcommit and let the VM be scheduled more easily.
ASKER
Hello:
I have made the following adjustments to the VMs:
Domain controller -
4 vCPUs, 4 GB (4,096 MB) RAM
Exchange 2010 -
6 vCPUs, 20 GB (20,480 MB) RAM
SQL (32-bit, so can't see more than 4 GB RAM) -
2 vCPUs, 4 GB (4,096 MB) RAM
Application (32-bit, so can't see more than 4 GB RAM) -
1 vCPUs, 1 GB (1,024 MB) RAM
I have attached a vSphere Client screenshot of the ESXi host Summary page.
I have made the following adjustments to the VMs:
Domain controller -
4 vCPUs, 4 GB (4,096 MB) RAM
Exchange 2010 -
6 vCPUs, 20 GB (20,480 MB) RAM
SQL (32-bit, so can't see more than 4 GB RAM) -
2 vCPUs, 4 GB (4,096 MB) RAM
Application (32-bit, so can't see more than 4 GB RAM) -
1 vCPUs, 1 GB (1,024 MB) RAM
I have attached a vSphere Client screenshot of the ESXi host Summary page.
ASKER
Umm ... the vSphere Client screenshot.
(It didn't upload with my previous post for some reason.)
(It didn't upload with my previous post for some reason.)
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
What is the allocated of vCPUs to the VMs.
Have you looked at the performance, using vCenter Server, or ESXi, see my EE Article
HOW TO: Performance Monitor vSphere 4.x or 5.0