Improve company productivity with a Business Account.Sign Up

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 2564
  • Last Modified:

Vmware CPU LATENCY Counter ON ESXI 5.0.

CPU LATENCY Counter ON ESXI 5.0.

I am having trouble understanding the results from the latency counter on the vcentre for our guest VM.

My understanding of the latency counter is that it will increase when there is contention for physical CPU resources and that it is a measurement of percentage of time that a guest has to wait to get access to the CPU resource.  However, all our hosts have more CPU physical cores than the total number of VCPU’s of our guests.  Therefore, if my understanding is correct, there should be minimal latency and there should be no contention for physical CPU resources.  We are seeing a number of guest VM wares with a latency of above 10% which I have read shows significant degradation of performance.

However, our CPU Ready counter are well below 500 milliseconds.  Can anyone shed any light as to why this might be the case?
0
James-Sillett
Asked:
James-Sillett
  • 7
  • 6
  • 3
  • +1
2 Solutions
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
use the esxtop command to determine if the ESX/ESXi server is being overloaded.

Examine the %READY field for the percentage of time that the virtual machine was ready but could not be scheduled to run on a physical CPU.

Under normal operating conditions, this value should remain under 5%

see my EE Article

HOW TO:  Performance Monitor vSphere 4.x or 5.0

also see Page 19

http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf
0
 
vmwarun - ArunCommented:
Are you sure if the VMs are not over-sized with vCPUs? I had earlier experienced this issue when a Windows Server 2008 R2 VM was provisioned with 4 vCPUs and the folks complained about latency. Reducing the vCPUs to 2 finally fixed the issue when we found that it was a P2V conversion and the application within the VM didn't require more than 2 vCPUs.
0
 
James-SillettAuthor Commented:
Hi, Well its an exchange server with 2 vcpus these are both being used to about 80% So I did an esxTop the cpu Rdy % is 0.18 But in Vcentre the Latency CPU is show 20%. I dont why these number are so different
Latency.JPG
exitop.JPG
0
Get expert help—faster!

Need expert help—fast? Use the Help Bell for personalized assistance getting answers to your important questions.

 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Seems acceptable to me, unless you are experiencing Exchange Server performance issues.
0
 
James-SillettAuthor Commented:
The exitop is fine however the latency I have been told if it is between 5 and 10 then you have problems and ours is showing  25% so the question I am asking is why we are seeing such different results between to counters ?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I've not heard of latency between 5 and 10 causing any issues, I'm afraid, I would look at performance long term.
0
 
compdigit44Commented:
How is the performance over all on the host your Exchange VM is running on?

I.E: OverAlll CPU, Memory and disk usage..

DO you have access to Vmware Operations Manager?
0
 
James-SillettAuthor Commented:
Everything is working fine, the we have updated our VMWARE Monitoring software Nworks for Operations Manager and it was this that has picked up a the high latency. They say that his mean there is contention but how can there be contention if we have more cores the VCPU. VMware have said don’t worry about it, but still no one seems to know what it actually means
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Both we and VMware would agree with the findings, check performance long term.
0
 
compdigit44Commented:
I agree with hanccoka, you need to monitor your system and gauge it's long term performance. I have never used NWorks and not sure if it has any performance forecasting abilities.

I am interested to find out more specifics on your ESXi host.

1) Make and Model
2) Type and number of CPU's
3) Amount of memory?
4) Are you assigning vCPU on Guest VM's per socket or per core?
0
 
James-SillettAuthor Commented:
Hi, This is what Nworks say about the counter

After consulting with PM it turns out that this threshold of 10% is gathered from theirs research, and discussions with other experts in VMware monitoring. CPU Latency is the best metric to monitor overall issues with vSphere CPU scheduler, it can be used instead of the older metrics CPU Ready and Wait. CPU latency includes many factors that impact VM CPU performance; and 10% of latency indicates that some investigation should be performed, using the other available counters at VM and also Host level, as detailed in our Knowledge base.


Then

CPU ready is the time a virtual CPU is ready to run but is not being scheduled on a physical CPU. It means that when the OS wants to process something on the processor the hypervisor is saying, ok hang on until it is your turn.
 
A VM in READY state is dispatched by the vmkernel CPU scheduler and enters RUN state. At RUN state, the VM is being served by the hypervisor, and can do what it is expected to do. CPU Ready & CPU Latency are similar
Latency: % of time the VM is unable to run because it is contending for access to the physical CPU.
Ready: % of time that the VM was ready, but could not get scheduled to run on the physical CPU.

So it seems like Ready is after Latency. Latency will go up first, as the preferred NUMA core might not be available, then Ready will follow if there is no core available at all.
0
 
James-SillettAuthor Commented:
3x Dell R720

16 Cores (2 x 8 core) 2.599 Xeron Each

98Gig Memory Each

90% of Guest are 1VCPU but one the servers that have 2 VCPU that virtual Sockets, I read this does not make a difference.

Average of 14 VCPU per Host (less than 16 Available) We do have hyper threading on, not sure this make a difference apart from showing 32cores
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
how many VMs per server?

Total host CPU in use?
0
 
James-SillettAuthor Commented:
9-10 Guest Per Host


2 host adverages @ 15% usage
1 host adverages@ 7% usage

HOST Latency

12% , 15% and 7%
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I would not be concerned with those figures, and concur with VMware.
0
 
compdigit44Commented:
I concur with Hanccooka with the config and stats you provide on your servers, your ESXi host seems to be running fine...
0
 
James-SillettAuthor Commented:
Thanks for your feedback
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

  • 7
  • 6
  • 3
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now