Solved

Vmware CPU LATENCY Counter ON ESXI 5.0.

Posted on 2013-05-10
17
2,132 Views
Last Modified: 2013-05-21
CPU LATENCY Counter ON ESXI 5.0.

I am having trouble understanding the results from the latency counter on the vcentre for our guest VM.

My understanding of the latency counter is that it will increase when there is contention for physical CPU resources and that it is a measurement of percentage of time that a guest has to wait to get access to the CPU resource.  However, all our hosts have more CPU physical cores than the total number of VCPU’s of our guests.  Therefore, if my understanding is correct, there should be minimal latency and there should be no contention for physical CPU resources.  We are seeing a number of guest VM wares with a latency of above 10% which I have read shows significant degradation of performance.

However, our CPU Ready counter are well below 500 milliseconds.  Can anyone shed any light as to why this might be the case?
0
Comment
Question by:James-Sillett
  • 7
  • 6
  • 3
  • +1
17 Comments
 
LVL 117
ID: 39155152
use the esxtop command to determine if the ESX/ESXi server is being overloaded.

Examine the %READY field for the percentage of time that the virtual machine was ready but could not be scheduled to run on a physical CPU.

Under normal operating conditions, this value should remain under 5%

see my EE Article

HOW TO:  Performance Monitor vSphere 4.x or 5.0

also see Page 19

http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf
0
 
LVL 19

Expert Comment

by:vmwarun - Arun
ID: 39157426
Are you sure if the VMs are not over-sized with vCPUs? I had earlier experienced this issue when a Windows Server 2008 R2 VM was provisioned with 4 vCPUs and the folks complained about latency. Reducing the vCPUs to 2 finally fixed the issue when we found that it was a P2V conversion and the application within the VM didn't require more than 2 vCPUs.
0
 

Author Comment

by:James-Sillett
ID: 39161281
Hi, Well its an exchange server with 2 vcpus these are both being used to about 80% So I did an esxTop the cpu Rdy % is 0.18 But in Vcentre the Latency CPU is show 20%. I dont why these number are so different
Latency.JPG
exitop.JPG
0
 
LVL 117
ID: 39161351
Seems acceptable to me, unless you are experiencing Exchange Server performance issues.
0
 

Author Comment

by:James-Sillett
ID: 39161431
The exitop is fine however the latency I have been told if it is between 5 and 10 then you have problems and ours is showing  25% so the question I am asking is why we are seeing such different results between to counters ?
0
 
LVL 117
ID: 39161485
I've not heard of latency between 5 and 10 causing any issues, I'm afraid, I would look at performance long term.
0
 
LVL 19

Expert Comment

by:compdigit44
ID: 39182311
How is the performance over all on the host your Exchange VM is running on?

I.E: OverAlll CPU, Memory and disk usage..

DO you have access to Vmware Operations Manager?
0
 

Author Comment

by:James-Sillett
ID: 39183625
Everything is working fine, the we have updated our VMWARE Monitoring software Nworks for Operations Manager and it was this that has picked up a the high latency. They say that his mean there is contention but how can there be contention if we have more cores the VCPU. VMware have said don’t worry about it, but still no one seems to know what it actually means
0
Backup Your Microsoft Windows Server®

Backup all your Microsoft Windows Server – on-premises, in remote locations, in private and hybrid clouds. Your entire Windows Server will be backed up in one easy step with patented, block-level disk imaging. We achieve RTOs (recovery time objectives) as low as 15 seconds.

 
LVL 117
ID: 39183664
Both we and VMware would agree with the findings, check performance long term.
0
 
LVL 19

Expert Comment

by:compdigit44
ID: 39184114
I agree with hanccoka, you need to monitor your system and gauge it's long term performance. I have never used NWorks and not sure if it has any performance forecasting abilities.

I am interested to find out more specifics on your ESXi host.

1) Make and Model
2) Type and number of CPU's
3) Amount of memory?
4) Are you assigning vCPU on Guest VM's per socket or per core?
0
 

Author Comment

by:James-Sillett
ID: 39184133
Hi, This is what Nworks say about the counter

After consulting with PM it turns out that this threshold of 10% is gathered from theirs research, and discussions with other experts in VMware monitoring. CPU Latency is the best metric to monitor overall issues with vSphere CPU scheduler, it can be used instead of the older metrics CPU Ready and Wait. CPU latency includes many factors that impact VM CPU performance; and 10% of latency indicates that some investigation should be performed, using the other available counters at VM and also Host level, as detailed in our Knowledge base.


Then

CPU ready is the time a virtual CPU is ready to run but is not being scheduled on a physical CPU. It means that when the OS wants to process something on the processor the hypervisor is saying, ok hang on until it is your turn.
 
A VM in READY state is dispatched by the vmkernel CPU scheduler and enters RUN state. At RUN state, the VM is being served by the hypervisor, and can do what it is expected to do. CPU Ready & CPU Latency are similar
Latency: % of time the VM is unable to run because it is contending for access to the physical CPU.
Ready: % of time that the VM was ready, but could not get scheduled to run on the physical CPU.

So it seems like Ready is after Latency. Latency will go up first, as the preferred NUMA core might not be available, then Ready will follow if there is no core available at all.
0
 

Author Comment

by:James-Sillett
ID: 39184146
3x Dell R720

16 Cores (2 x 8 core) 2.599 Xeron Each

98Gig Memory Each

90% of Guest are 1VCPU but one the servers that have 2 VCPU that virtual Sockets, I read this does not make a difference.

Average of 14 VCPU per Host (less than 16 Available) We do have hyper threading on, not sure this make a difference apart from showing 32cores
0
 
LVL 117
ID: 39184157
how many VMs per server?

Total host CPU in use?
0
 

Author Comment

by:James-Sillett
ID: 39184240
9-10 Guest Per Host


2 host adverages @ 15% usage
1 host adverages@ 7% usage

HOST Latency

12% , 15% and 7%
0
 
LVL 117

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 250 total points
ID: 39184262
I would not be concerned with those figures, and concur with VMware.
0
 
LVL 19

Assisted Solution

by:compdigit44
compdigit44 earned 250 total points
ID: 39184387
I concur with Hanccooka with the config and stats you provide on your servers, your ESXi host seems to be running fine...
0
 

Author Closing Comment

by:James-Sillett
ID: 39184412
Thanks for your feedback
0

Featured Post

What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

Join & Write a Comment

HOW TO: Connect to the VMware vSphere Hypervisor 6.5 (ESXi 6.5) using the vSphere (HTML5 Web) Host Client 6.5, and perform a simple configuration task of adding a new VMFS 6 datastore.
Veeam Backup & Replication has added a new integration – Veeam Backup for Microsoft Office 365.  In this blog, we will discuss how you can benefit from Office 365 email backup with the Veeam’s new product and try to shed some light on the needs and …
This tutorial will walk an individual through the steps necessary to enable the VMware\Hyper-V licensed feature of Backup Exec 2012. In addition, how to add a VMware server and configure a backup job. The first step is to acquire the necessary licen…
This video shows you how to use a vSphere client to connect to your ESX host as the root user. Demonstrates the basic connection of bypassing certification set up. Demonstrates how to access the traditional view to begin managing your virtual mac…

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now