Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Vmware CPU LATENCY Counter ON ESXI 5.0.

Posted on 2013-05-10
17
Medium Priority
?
2,400 Views
Last Modified: 2013-05-21
CPU LATENCY Counter ON ESXI 5.0.

I am having trouble understanding the results from the latency counter on the vcentre for our guest VM.

My understanding of the latency counter is that it will increase when there is contention for physical CPU resources and that it is a measurement of percentage of time that a guest has to wait to get access to the CPU resource.  However, all our hosts have more CPU physical cores than the total number of VCPU’s of our guests.  Therefore, if my understanding is correct, there should be minimal latency and there should be no contention for physical CPU resources.  We are seeing a number of guest VM wares with a latency of above 10% which I have read shows significant degradation of performance.

However, our CPU Ready counter are well below 500 milliseconds.  Can anyone shed any light as to why this might be the case?
0
Comment
Question by:James-Sillett
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 7
  • 6
  • 3
  • +1
17 Comments
 
LVL 123
ID: 39155152
use the esxtop command to determine if the ESX/ESXi server is being overloaded.

Examine the %READY field for the percentage of time that the virtual machine was ready but could not be scheduled to run on a physical CPU.

Under normal operating conditions, this value should remain under 5%

see my EE Article

HOW TO:  Performance Monitor vSphere 4.x or 5.0

also see Page 19

http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf
0
 
LVL 19

Expert Comment

by:vmwarun - Arun
ID: 39157426
Are you sure if the VMs are not over-sized with vCPUs? I had earlier experienced this issue when a Windows Server 2008 R2 VM was provisioned with 4 vCPUs and the folks complained about latency. Reducing the vCPUs to 2 finally fixed the issue when we found that it was a P2V conversion and the application within the VM didn't require more than 2 vCPUs.
0
 

Author Comment

by:James-Sillett
ID: 39161281
Hi, Well its an exchange server with 2 vcpus these are both being used to about 80% So I did an esxTop the cpu Rdy % is 0.18 But in Vcentre the Latency CPU is show 20%. I dont why these number are so different
Latency.JPG
exitop.JPG
0
Simple, centralized multimedia control

Watch and learn to see how ATEN provided an easy and effective way for three jointly-owned pubs to control the 60 televisions located across their three venues utilizing the ATEN Control System, Modular Matrix Switch and HDBaseT extenders.

 
LVL 123
ID: 39161351
Seems acceptable to me, unless you are experiencing Exchange Server performance issues.
0
 

Author Comment

by:James-Sillett
ID: 39161431
The exitop is fine however the latency I have been told if it is between 5 and 10 then you have problems and ours is showing  25% so the question I am asking is why we are seeing such different results between to counters ?
0
 
LVL 123
ID: 39161485
I've not heard of latency between 5 and 10 causing any issues, I'm afraid, I would look at performance long term.
0
 
LVL 20

Expert Comment

by:compdigit44
ID: 39182311
How is the performance over all on the host your Exchange VM is running on?

I.E: OverAlll CPU, Memory and disk usage..

DO you have access to Vmware Operations Manager?
0
 

Author Comment

by:James-Sillett
ID: 39183625
Everything is working fine, the we have updated our VMWARE Monitoring software Nworks for Operations Manager and it was this that has picked up a the high latency. They say that his mean there is contention but how can there be contention if we have more cores the VCPU. VMware have said don’t worry about it, but still no one seems to know what it actually means
0
 
LVL 123
ID: 39183664
Both we and VMware would agree with the findings, check performance long term.
0
 
LVL 20

Expert Comment

by:compdigit44
ID: 39184114
I agree with hanccoka, you need to monitor your system and gauge it's long term performance. I have never used NWorks and not sure if it has any performance forecasting abilities.

I am interested to find out more specifics on your ESXi host.

1) Make and Model
2) Type and number of CPU's
3) Amount of memory?
4) Are you assigning vCPU on Guest VM's per socket or per core?
0
 

Author Comment

by:James-Sillett
ID: 39184133
Hi, This is what Nworks say about the counter

After consulting with PM it turns out that this threshold of 10% is gathered from theirs research, and discussions with other experts in VMware monitoring. CPU Latency is the best metric to monitor overall issues with vSphere CPU scheduler, it can be used instead of the older metrics CPU Ready and Wait. CPU latency includes many factors that impact VM CPU performance; and 10% of latency indicates that some investigation should be performed, using the other available counters at VM and also Host level, as detailed in our Knowledge base.


Then

CPU ready is the time a virtual CPU is ready to run but is not being scheduled on a physical CPU. It means that when the OS wants to process something on the processor the hypervisor is saying, ok hang on until it is your turn.
 
A VM in READY state is dispatched by the vmkernel CPU scheduler and enters RUN state. At RUN state, the VM is being served by the hypervisor, and can do what it is expected to do. CPU Ready & CPU Latency are similar
Latency: % of time the VM is unable to run because it is contending for access to the physical CPU.
Ready: % of time that the VM was ready, but could not get scheduled to run on the physical CPU.

So it seems like Ready is after Latency. Latency will go up first, as the preferred NUMA core might not be available, then Ready will follow if there is no core available at all.
0
 

Author Comment

by:James-Sillett
ID: 39184146
3x Dell R720

16 Cores (2 x 8 core) 2.599 Xeron Each

98Gig Memory Each

90% of Guest are 1VCPU but one the servers that have 2 VCPU that virtual Sockets, I read this does not make a difference.

Average of 14 VCPU per Host (less than 16 Available) We do have hyper threading on, not sure this make a difference apart from showing 32cores
0
 
LVL 123
ID: 39184157
how many VMs per server?

Total host CPU in use?
0
 

Author Comment

by:James-Sillett
ID: 39184240
9-10 Guest Per Host


2 host adverages @ 15% usage
1 host adverages@ 7% usage

HOST Latency

12% , 15% and 7%
0
 
LVL 123

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE^2) earned 1000 total points
ID: 39184262
I would not be concerned with those figures, and concur with VMware.
0
 
LVL 20

Assisted Solution

by:compdigit44
compdigit44 earned 1000 total points
ID: 39184387
I concur with Hanccooka with the config and stats you provide on your servers, your ESXi host seems to be running fine...
0
 

Author Closing Comment

by:James-Sillett
ID: 39184412
Thanks for your feedback
0

Featured Post

Create the perfect environment for any meeting

You might have a modern environment with all sorts of high-tech equipment, but what makes it worthwhile is how you seamlessly bring together the presentation with audio, video and lighting. The ATEN Control System provides integrated control and system automation.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Ransomware is a malware that is again in the list of security  concerns. Not only for companies, but also for Government security and  even at personal use. IT departments should be aware and have the right  knowledge to how to fight it.
Changing a few Outlook Options can help keep you organized!
This video shows you how easy it is to boot from ISO images for virtual machines with the ISO images stored on a local datastore on the ESXi host.
This video shows how to remove a single email address from the Outlook 2010 Auto Suggestion memory. NOTE: For Outlook 2016 and 2013 perform the exact same steps. Open a new email: Click the New email button in Outlook. Start typing the address: …

670 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question