asked on

ESXi Disk write latency always gone up to above 65 ms every morning between 6 - 7 AM ?

Hi All,

I'd like to know what could be the problem or indication if I got this warning every morning between 6 AM - 7 AM ?

Status: Warning (Yellow)
Alarm: Host disk write latency
Time: 6/11/2010 6:26:56 AM
Level of Disk write latency is above 65 Millisecond

Open in new window

FYI:

ESXi 4.0 with local VMFS datastore of RAID-1
2x 1 TB WDC10EARS 64 MB buffer 7200 rpm SATA-II

it happens just in the morning when there is no significant workload in the server (it is still outside of business hours).

any kind of suggestion would be greatly appreciated.

Thanks.

Slouzer

Check for automated antivirus checks or that the backups are not still running?

jjoz

ASKER

Thanks for the reply man,

This is the model of my hard disk drive: http://www.wdc.com/en/products/products.asp?DriveID=866

I don't know why it is happening with a patterns consistently between those time frame.

David

There HAS to be at least one new job running between 6-7AM to account for this. So you're just going to have to find it. I'd turn off the virtual machines one by one during this window to see which VM is the culprit as a starting point, then you will know where to look.

Once you know what machine it is, use the native o/s utilities to see what program is chewing up the most CPU & IO and act accordingly.

jjoz

ASKER

thanks for the suggestion man, this is production web server so it can't be turned off, but that's a good way to isolate the problem.
If this happens during the business hours continuously then i can understand that there is something wrong with the Hard drive unfortunately it isn't.

David

maybe move the clock on each VM up to 6AM briefly to see if anything kicks off?

David

No way is it a hardware problem. There is nothing in way of automated maintenance that kicks off inside an HDD that would run on this sort of schedule. The HDD has no internal clock. It works by cumulative power on hours. So if you turned off the computer for 30 mins, then if it WAS the HDD, the window it is slow will be from 6:30 - 7:30.

SOLUTION

David

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

jjoz

ASKER

ok, I'll try that when i get into the office tomorrow morning.
thanks for the info man.

SOLUTION

bgoering

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

ASKER CERTIFIED SOLUTION

Michael Worsham

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

jjoz

ASKER

thanks man !

jjoz

ASKER

here's an update to the problem,

I forgot to include the real screenshot from my ESXi, here it is, it averages above 40 ms for both disk command latency and the disk commands issued.

I know that the performance is very slow but i couldn't found anything peculiar in CPU or memory contention.

Is this normal or not OK for a low load web server ?
diskLatencyGraph.jpg