intel matrix raid 10 slow performance troubleshooting

You're getting absolutely sensational performance out of four SATA drives. Bandwidth is bursting to 300MB/sec (largely through read-ahead and read caching I suspect). Steady-state sequential read is bang-on where I'd expect it to be. Latency is also excellent at a touch over 13ms

So why is there the big discrepancy between results? The answer is benchmarking software - beware of disappearing into the downward spiral of chasing top-notch benchmark figures. That way madness lies....

I'd expect about 120 write I/O's per second (IOPS), 240 read IOPS and about 16 - 20 MB/sec on write and up to about 40MB/sec on reads for a real-world workload.  Real world I/O profiles are largely random unless you are serving streaming media, manipulating large image files or doing backup-to-disk. All the performance results show sequential read performance which is in the "sweet spot" of disk drive performance (especially for SATA). Re-run your tests for 4K to 16K (which is far more typical of a normal I/O load) random reads and writes, and you'll see the RAID array performance is streets ahead of a single disk.

There are many possible reasons that your sequential read performance is all over the shop - and one of the most important reasons is vibration. When you have four disks next to each other (especially in a lightweight pressed steel drive cage in a PC), rotational vibration from one affects all the others, sending the heads slightly off-track - which has to be corrected by drive firmware. The reason enterprise drives (SCSI and FC) are so much more expensive than personal storage drives is that SCSI drives have two processors on the drive controller board - one handles I/O, the other handles head tracking. SCSI/FC drives are also made with a more ridgid aluminium casing and so on. All this adds up to much higher performance from SCSI/FC drives.

If you are interested in learning more. take a look at this whitepaper from Seagate: http://www.seagate.com/content/docs/pdf/whitepaper/D2c_More_than_Interface_ATA_vs_SCSI_042003.pdf. Microsoft has also published a whitepaper describing the effects I've outlined above: download.microsoft.com/download/9/8/f/98f3fe47-dfc3-4e74-92a3-088782200fe7/TWST05005_WinHEC05.ppt

So: If you want high disk performance, you have to shell out $$$ for the appropriate hardware - you can't polish a poo.  

I think you nailed it RockChalk...  see graph below -- have almost doubled average read rate with that one simple change, and now the curve has very few irregularities in it.   That's much more like what I had hoped this RAID setup would do.   I had seen some references to the write-back cache, but I looked in my BIOS for some place to enable it and didn't find it.  That right click was just too easy once you pointed it out.   Thanks!

hd-tach-cache-enabled.jpg

That's odd to say the least. Write back cache should only affect write performance, not read.

===>>>> "That's odd to say the least. Write back cache should only affect write performance, not read"

agree, that is completely counter-intuitive, and I haven't found any authoritative explanation for why it helps read performance.  But the behavior is consistent with what some other people had reported.  The only reason I hadn't tried it before was that I didn't understand how to change it on my system.

googling around, the one statement I found that sounds like a plausible explanation is this (from the link I posted earlier):
"On the ICH9R (and probably ICH8R, ICH7R etc), enabling the write cache also enables read-ahead cacheing in the driver and/or controller."

also, fyi another thing I tried was booting my system with a Bart PE disk, so that I could run the test with an absolute minimum of other processes running, and the results were about 1/2 way between the original performance and the final one.  The graph still had a sort of binary appearance, i.e. flipping between what looks like raid-0 performance and what looks like single drive performance, but in the Bart PE environment it spent much more time in the higher performance mode.  So for whatever reason, it appears that without the write-back cache turned on, the hard drive system seems to be easily bogged down.   Not sure whether that means main system memory is choking, or something else.  

btw, regarding some of your original info, I think these WD RE2 drives I'm using do incorporate many of the engineering tweaks normally associated with SCSI/FC drives.  They may not be quite as good, but certainly WD markets them as "Enterprise Class" designed specifically for RAID applications, see http://www.westerndigital.com/en/products/products.asp?driveid=399    For my application, I'm mainly dealing with photo and video editing, so my two criteria were high transfer speed for video capture and high reliability.  It seemed to me like Raid 10 with these SATA RE2 drives was a pretty good fit for the purpose.