VMWare performance Question (esxstop)

I have a lab env with a Dell server that has 2 CPUs and 10Gb of RAM and ESX 3.5 install. I have some users connecting and doing some testing, nothig big just some stuff with Microsoft Office and running some other apps but all simple stuff. I have about  between 8 to 10 VMs running all the time and most of the time they are idle and perhaps sometimes I can get between 2 or 4 computers with connected users (so 4 VMs being used an 4 or more up to 6 running but idle).

The performance of those machines sometimes is pretty bad and even the management of the ESX server with VC is also really slow.

I was wondering if the cpu over utilization might be the cause of the issue.

In the current situation we are running 10 virtual machine on a host which has only 2 physical processors.

Because of this only 2 virtual machines can execute their processes on the physical processor.

Rest of the 8 virtual machine has to wait till the processor is free.

Does that make sense?

I have ran "esxtop" I have attached the results on a screenshoot. I think the performance looks pretty sluggish specially I am concerned about the %wait time (miliseconds) for accessing the CPU.

The name of the VMs have been removed basically each line represents a VM that is turned ON.
 
Other information about the server:

What type of CPU' do you have (single, dual or quad core, what speed) - Single CPU - Processor, 80556K, Xeon Woodcrest, 5130, LGA771, Burn 2

http://ark.intel.com/Product.aspx?id=27216&processor=5130&spec-codes=SLAGC 

The server is a Dell PowerEdge 1950 / Service tag: 2H7QQC1

What type of disks do you have (SAS, SATA) -> The VMs are in local storage and the disk is a Serial Attached SCSI, 3, 10K, 3.5

Do you have a RAID configuration (RAID1, 1+0, 5) - Yes

Does the RAID controller have battery backed write cache ? No

How many vCPU's did you assign to the VMs - 1 vCPU per machine

esxtop.gif
llaravaAsked:
Who is Participating?
 
VMwareGuyCommented:
2ND LINE ABOVE YOUR CURSOR.  :)  PCPU stands for physical CPU and the percentages that follow are your overall utilization per core.  IF there are more cores they aren't showing up in the command output, please run the ESXTOP command and post all the output starting from the prompt so I can see if there is a listing of logical CPUs.  If there are, I want to see what the utilization is on them.  I didn't bother to check the server model, but if there are 8 cores and all are properly functioning they will show up in the ESXTOP output.  If they aren't there but should be, immediately log into ESX with the VI client and check your hardware health status.  Make sure all of your CPU cores are working.    

The sources of poor performance that you should be examining include:  

1  - if your storage is all local, did you format your VMFS volumes during the installation or afterward with your VI client?  If you formatted during your installation your VMFS volumes aren't properly aligned.  The ESX installer has limitations that prevent a VMFS volume from being properly formatted.  It won't be much of a contributor to poor performance, I've measured it, but it does create a few additional CPU cycles to handle I/O and will slow sequential I/O operations down by 20%.  .

2. - Your RAID array, what type of disk and how fast, and how many spindles to you have to support your I/O?  This won't be much of an isssue with a small test environment, but will compound itself if there is any growth to your environment and or if you expect production level performance.  In order to handle I\O from mulitple VMs you need lots of disk to handle it.  But since you are using local disk I realize you are limited unless you configure some external storage.          

3.  Since you only have 2 CPUs, think about suspending VMs not in use, minimizing memory and CPU allocations in each VM, keeping all IDE devices disconnected (CDROM, Floppy) from ISOs

4.)  Your environment is in contention for CPU, perfect time to leverage CPU shares and set your priority VMs with higher level of shares so they perform better, great feature that does work.  

5)  Stop unnecessary services in each of your guest OSs, any service that can be disabled or set to manual should be.  Make a standard build that is as light weight as possible and convert it to a template.  This helps.  

0
 
VMwareGuyCommented:
Hi there - your PCPUs, as shown in your ESXTOP output, are taxed at 95% and 81%, expect poor performance.  The CPU ready time signifies how long it needs to wait to get free CPU threads from the host to execute requests to the CPU.  This # is obviously high.   A good rule of thumb is 4 vCPUs per each physical core, you currently are 10 VMs running on just 2 cores.  Doesn't look like you are really asking anything you hvaen't already answered and perhaps are just looking for confirmation, well, here it is.  
0
 
IanThCommented:
you say your server has 2 cpu's but they are quad core so your esx has 8 cores which sould be just about enough for 10vm's are you running on a fast 'gig' ethernet?
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

 
VMwareGuyCommented:
funny, I don't see more than 2 PCPUs in the ESXTOP output... I don't believe this server has more than the 2 cores shown, logical CPUs would show up in the output along with the physical cores if there were any.  
0
 
nithinprabhakarCommented:
Let me understand your question first,

My understanding is that you have performance issue on Guest OS,

My questions:

Whats Guest OS are you using? Windows/Unix? and what type os Apps are running?

One thing I would like to explain you here that no matter how many Physical Processor you have, all VMs would be able to run multiple application at a time unlike Physical Servers.

I guess that answers your below question,
"In the current situation we are running 10 virtual machine on a host which has only 2 physical processors.

Because of this only 2 virtual machines can execute their processes on the physical processor.

Rest of the 8 virtual machine has to wait till the processor is free.

Does that make sense?"

All 10 VMs should be able to perform equally and if you have created identical Virtual HW settings.

Let me know the ESX version that your running. if possible get me vm-support logs from ESX server.

Note : if its less than 50 MB attach the same as a file here.
0
 
llaravaAuthor Commented:
VMwareGuy: That is exactly what I have been told by VMWare support but I was just looking for other point of view.

The server has only 2 CPUs however from the  ESXTOP output where are you seen this?

IanTh: Here are the spec. of the server if you want to check Dell PowerEdge 1950 / Service tag: 2H7QQC1

nithinprabhakar: We are running WXP computers they just run office and the line of business applications which they are not really heavy stuff (thik client install running against a server)

We run ESX 3.5

Finally are you sure of "One thing I would like to explain you here that no matter how many Physical Processor you have, all VMs would be able to run multiple application at a time unlike Physical Servers" can you please point me out to an official VMWare KB article or resource that mentions what you have said. That is not what I have heard from VMWare support.

Thanks.
0
 
IanThCommented:
the dell 1950 might have 2 cpu's but they are quad core so that means 8 cores in your 2 cpu's

are you running on a gigabit lan ?
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.