VMWare performance Question (esxstop)

I have a lab env with a Dell server that has 2 CPUs and 10Gb of RAM and ESX 3.5 install. I have some users connecting and doing some testing, nothig big just some stuff with Microsoft Office and running some other apps but all simple stuff. I have about  between 8 to 10 VMs running all the time and most of the time they are idle and perhaps sometimes I can get between 2 or 4 computers with connected users (so 4 VMs being used an 4 or more up to 6 running but idle).

The performance of those machines sometimes is pretty bad and even the management of the ESX server with VC is also really slow.

I was wondering if the cpu over utilization might be the cause of the issue.

In the current situation we are running 10 virtual machine on a host which has only 2 physical processors.

Because of this only 2 virtual machines can execute their processes on the physical processor.

Rest of the 8 virtual machine has to wait till the processor is free.

Does that make sense?

I have ran "esxtop" I have attached the results on a screenshoot. I think the performance looks pretty sluggish specially I am concerned about the %wait time (miliseconds) for accessing the CPU.

The name of the VMs have been removed basically each line represents a VM that is turned ON.
 
Other information about the server:

What type of CPU' do you have (single, dual or quad core, what speed) - Single CPU - Processor, 80556K, Xeon Woodcrest, 5130, LGA771, Burn 2

http://ark.intel.com/Product.aspx?id=27216&processor=5130&spec-codes=SLAGC 

The server is a Dell PowerEdge 1950 / Service tag: 2H7QQC1

What type of disks do you have (SAS, SATA) -> The VMs are in local storage and the disk is a Serial Attached SCSI, 3, 10K, 3.5

Do you have a RAID configuration (RAID1, 1+0, 5) - Yes

Does the RAID controller have battery backed write cache ? No

How many vCPU's did you assign to the VMs - 1 vCPU per machine

esxtop.gif
llaravaAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

VMwareGuyCommented:
Hi there - your PCPUs, as shown in your ESXTOP output, are taxed at 95% and 81%, expect poor performance.  The CPU ready time signifies how long it needs to wait to get free CPU threads from the host to execute requests to the CPU.  This # is obviously high.   A good rule of thumb is 4 vCPUs per each physical core, you currently are 10 VMs running on just 2 cores.  Doesn't look like you are really asking anything you hvaen't already answered and perhaps are just looking for confirmation, well, here it is.  
0
IanThCommented:
you say your server has 2 cpu's but they are quad core so your esx has 8 cores which sould be just about enough for 10vm's are you running on a fast 'gig' ethernet?
0
VMwareGuyCommented:
funny, I don't see more than 2 PCPUs in the ESXTOP output... I don't believe this server has more than the 2 cores shown, logical CPUs would show up in the output along with the physical cores if there were any.  
0
Acronis Data Cloud 7.8 Enhances Cyber Protection

A closer look at five essential enhancements that benefit end-users and help MSPs take their cloud data protection business further.

nithinprabhakarCommented:
Let me understand your question first,

My understanding is that you have performance issue on Guest OS,

My questions:

Whats Guest OS are you using? Windows/Unix? and what type os Apps are running?

One thing I would like to explain you here that no matter how many Physical Processor you have, all VMs would be able to run multiple application at a time unlike Physical Servers.

I guess that answers your below question,
"In the current situation we are running 10 virtual machine on a host which has only 2 physical processors.

Because of this only 2 virtual machines can execute their processes on the physical processor.

Rest of the 8 virtual machine has to wait till the processor is free.

Does that make sense?"

All 10 VMs should be able to perform equally and if you have created identical Virtual HW settings.

Let me know the ESX version that your running. if possible get me vm-support logs from ESX server.

Note : if its less than 50 MB attach the same as a file here.
0
llaravaAuthor Commented:
VMwareGuy: That is exactly what I have been told by VMWare support but I was just looking for other point of view.

The server has only 2 CPUs however from the  ESXTOP output where are you seen this?

IanTh: Here are the spec. of the server if you want to check Dell PowerEdge 1950 / Service tag: 2H7QQC1

nithinprabhakar: We are running WXP computers they just run office and the line of business applications which they are not really heavy stuff (thik client install running against a server)

We run ESX 3.5

Finally are you sure of "One thing I would like to explain you here that no matter how many Physical Processor you have, all VMs would be able to run multiple application at a time unlike Physical Servers" can you please point me out to an official VMWare KB article or resource that mentions what you have said. That is not what I have heard from VMWare support.

Thanks.
0
IanThCommented:
the dell 1950 might have 2 cpu's but they are quad core so that means 8 cores in your 2 cpu's

are you running on a gigabit lan ?
0
VMwareGuyCommented:
2ND LINE ABOVE YOUR CURSOR.  :)  PCPU stands for physical CPU and the percentages that follow are your overall utilization per core.  IF there are more cores they aren't showing up in the command output, please run the ESXTOP command and post all the output starting from the prompt so I can see if there is a listing of logical CPUs.  If there are, I want to see what the utilization is on them.  I didn't bother to check the server model, but if there are 8 cores and all are properly functioning they will show up in the ESXTOP output.  If they aren't there but should be, immediately log into ESX with the VI client and check your hardware health status.  Make sure all of your CPU cores are working.    

The sources of poor performance that you should be examining include:  

1  - if your storage is all local, did you format your VMFS volumes during the installation or afterward with your VI client?  If you formatted during your installation your VMFS volumes aren't properly aligned.  The ESX installer has limitations that prevent a VMFS volume from being properly formatted.  It won't be much of a contributor to poor performance, I've measured it, but it does create a few additional CPU cycles to handle I/O and will slow sequential I/O operations down by 20%.  .

2. - Your RAID array, what type of disk and how fast, and how many spindles to you have to support your I/O?  This won't be much of an isssue with a small test environment, but will compound itself if there is any growth to your environment and or if you expect production level performance.  In order to handle I\O from mulitple VMs you need lots of disk to handle it.  But since you are using local disk I realize you are limited unless you configure some external storage.          

3.  Since you only have 2 CPUs, think about suspending VMs not in use, minimizing memory and CPU allocations in each VM, keeping all IDE devices disconnected (CDROM, Floppy) from ISOs

4.)  Your environment is in contention for CPU, perfect time to leverage CPU shares and set your priority VMs with higher level of shares so they perform better, great feature that does work.  

5)  Stop unnecessary services in each of your guest OSs, any service that can be disabled or set to manual should be.  Make a standard build that is as light weight as possible and convert it to a template.  This helps.  

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.