Unexplainable sudden drop in RAID 5 Throughput

When you are talking about throughput, you are relating to R/W in/out put of your RAID array and network interface and switch port...

You can perform a test to narrow down the problem by trying to copy or backup to an external USB and analyze the speed. Doing this, you can prove either it's network related or not.
If copy to USB is also unexpected slow, you may just need to look in your controller firmware, driver to ensure it's up to date. Reinstall proliant support pack newest version can mitigate the issue.

http://h20000.www2.hp.com/bizsupport/TechSupport/DriverDownload.jsp?lang=en&cc=us&prodNameId=3716247&taskId=135&prodTypeId=18964&prodSeriesId=3716246&lang=en&cc=us

K

Your RAID-5 array may be in a degraded state; you can check in the RAID controller BIOS or RAID management software.  If it is degraded, a drive may need to be replaced and the array will remain degraded until the rebuild is complete.

I thought he said that he already checked the ACU to prove his RAID was healthy no degraded component....

K

Inkevin, if it was network related (controller, switch port, etc)  wouldn't I expect to see similar degradation in throughput on the C: array as well as the D: ? But degredation is only on the D: array.

Sorry, I'm quite sure I used D: instead of D:\ on the CHKDSK command, just a typo on this forum.

--Richard

Good clarification! At least we focus on the RAID array.

I assumed you did check Array Configuration Manager for degraded status... Did you update your firmware and driver on controller?

K

Everything points to the RAID array, so I wanted to make sure - the symptoms certainly sound like a degraded array.

ACU reports:
 Status: OK
Failed Physical Drives: None

HP System Management reports no failed/degraded items

I can certainly update firmware and driver on the controller...is it possible for the controller to cause throughput degradation on C: and not on D: ?

Your C: and D: are not on the same spindle. C: is in RAID 1 and D: is in RAID 5. That said, the load is on RAID 5 container because your Exchange Logs are writting to it. You can certainly relieve the load by migrating all your Exchange logs to C:

K

My logs are already on C:

I will update the controller firmware/driver and see if that makes a difference.

Thank  you.

this happend to me not long ago.  IMO you got a ton of exchange data on just 3 spindles.  Exchange doesnt like that too much.
anyways.. when my exchange backups (580gb datastore) started taking 18 hours instead of the usual 4 hours, i did an offline defrag.  everyone says that you dont NEED to do it (and you dont really), but when i ran it my datastore went down to 230GB but my backup times dropped to 2:30

you may want to investigate that.  and yes, it killed my system, even when exchange svcs were stopped, my system ran like a dog until the offline defrag.

good luck

Has read and write caching been disabled on the RAID array?

PcGod718, thanks for sharing your experience. Again though, even with Exchange out of the picture (dismounted) throughput is still just as slow doing copies or backups of non-exchange files.

Meyersd, write cache has always been disabled on that RAID, and read cache is 100%.

Just an update, we installed all the latest firmware and drivers to all components on that server yesterday. We had been in touch with HP and this is also what they recommended. No change in the throughput. Definitely stumped!

Since it's almost a mystery now, I just suggest to do some basic things hopefully can pull some triggers. Can you first perform a disk defragmentation on your D: in the off hours? After that, if it still does not improve, run chkdsk /F

K

well, since its just an exchange database.. why not go balls to the wall and image/backup the raid5 to an external drive, then completely blowout the raid 5 and re-do it.  should only take a few hours, and you can bypass all this chkdsk stuff.
you can take the shortcut and just kill the partition and recreate, but since it may be something a bit more underlying, i would kill/recreate the entire raid5.


-al