Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

ESXi Disk write latency always gone up to above 65 ms every morning between 6 - 7 AM ?

Posted on 2010-11-08
12
Medium Priority
?
4,269 Views
Last Modified: 2012-05-10
Hi All,

I'd like to know what could be the problem or indication if I got this warning every morning between 6 AM - 7 AM ?

Status: Warning (Yellow)
Alarm: Host disk write latency
Time: 6/11/2010 6:26:56 AM
Level of Disk write latency is above 65 Millisecond

Open in new window


FYI:

ESXi 4.0 with local VMFS datastore of RAID-1
2x 1 TB WDC10EARS 64 MB buffer 7200 rpm SATA-II

it happens just in the morning when there is no significant workload in the server (it is still outside of business hours).

any kind of suggestion would be greatly appreciated.

Thanks.
0
Comment
Question by:jjoz
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
12 Comments
 
LVL 2

Expert Comment

by:Slouzer
ID: 34083194
Check for automated antivirus checks or that the backups are not still running?
0
 
LVL 1

Author Comment

by:jjoz
ID: 34083224
Thanks for the reply man,

This is the model of my hard disk drive: http://www.wdc.com/en/products/products.asp?DriveID=866

I don't know why it is happening with a patterns consistently between those time frame.
0
 
LVL 47

Expert Comment

by:David
ID: 34083697
There HAS to be at least one new job running between 6-7AM to account for this.  So you're just going to have to find it.   I'd turn off the virtual machines one by one during this window to see which VM is the culprit as a starting point, then you will know where to look.  

Once you know what machine it is, use the native o/s utilities to see what program is chewing up the most CPU & IO and act accordingly.
0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 
LVL 1

Author Comment

by:jjoz
ID: 34083748
thanks for the suggestion man, this is production web server so it can't be turned off, but that's a good way to isolate the problem.
If this happens during the business hours continuously then i can understand that there is something wrong with the Hard drive unfortunately it isn't.
0
 
LVL 47

Expert Comment

by:David
ID: 34083878
maybe move the clock on each VM up to 6AM briefly to see if anything kicks off?
0
 
LVL 47

Expert Comment

by:David
ID: 34083902
No way is it a hardware problem. There is nothing in way of automated maintenance that kicks off inside an HDD that would run on this sort of schedule.  The HDD has no internal clock.  It works by cumulative power on hours.   So if you turned off the computer for 30 mins, then if it WAS the HDD,  the window it is slow will be from 6:30 - 7:30.
0
 
LVL 47

Assisted Solution

by:David
David earned 668 total points
ID: 34083920
Also, just use perfmon if windows, iostat if UNIX/LINUX, etc ... on each VM during the performance slow down.  Quite simply, it is slow because the HDD is doing more work.  Find the rogue program.
0
 
LVL 1

Author Comment

by:jjoz
ID: 34083967
ok, I'll try that when i get into the office tomorrow morning.
thanks for the info man.
0
 
LVL 28

Assisted Solution

by:bgoering
bgoering earned 668 total points
ID: 34084606
Two things - (1) watch the performance tab on each VM to see what the disk read/write rate is during the problem period - likely only one of your VMs is causing the issue. Once you identify the troublesome vm you can dig deeper into what precisely it is doing during the problem period.

(2) - Make sure you have battery backed write cache on your raid controller, and that it is configured for "write-back" (as opposed to "write-through") caching. Many performance issues with disk wrties can be traced back to raid controller caching configuration and/or the lack of BBWC.

Hope this helps.
0
 
LVL 29

Accepted Solution

by:
Michael Worsham earned 664 total points
ID: 34086711
One thing you stated is that you are using WD Caviar Green drives. This is part of the main issue.

For VMware ESX/ESXi, Server and other virtualized host environments, stick with the WD Caviar Black drives. The 'black' model run at 7200 RPM and are geared to be faster, thus more robust under duress. The 'green' models run at 5400 RPM and have special reduced power consumption capabilities including turning off the drive when not in use. Servers, especially VM-based, will always be active thus not able to take advantage of green-like energy saving modules, thus cause heavy wear and tear on drives that do support them.

Additional reference:
http://www.tomshardware.com/reviews/hitachi-western-digital-terabyte,2017-3.html
0
 
LVL 1

Author Closing Comment

by:jjoz
ID: 34098619
thanks man !
0
 
LVL 1

Author Comment

by:jjoz
ID: 34202940
here's an update to the problem,

I forgot to include the real screenshot from my ESXi, here it is, it averages above 40 ms for both disk command latency and the disk commands issued.

I know that the performance is very slow but i couldn't found anything peculiar in CPU or memory contention.

Is this normal or not OK for a low load web server ?
diskLatencyGraph.jpg
0

Featured Post

Enroll in October's Free Course of the Month

Do you work with and analyze data? Enroll in October's Course of the Month for 7+ hours of SQL training, allowing you to quickly and efficiently store or retrieve data. It's free for Premium Members, Team Accounts, and Qualified Experts!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Each year, investment in cloud platforms grows more than 20% (https://www.immun.io/hubfs/Immunio_2016/Content/Marketing/Cloud-Security-Report-2016.pdf?submissionGuid=a8d80a00-6fee-4b85-81db-a4e28f681762) as an increasing number of companies begin to…
Backups and Disaster RecoveryIn this post, we’ll look at strategies for backups and disaster recovery.
Advanced tutorial on how to run the esxtop command to capture a batch file in csv format in order to export the file and use it for performance analysis. He demonstrates how to download the file using a vSphere web client (or vSphere client) and exp…
This video shows you how easy it is to boot from ISO images for virtual machines with the ISO images stored on a local datastore on the ESXi host.

618 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question