Solved

Does traditional HDD RAID lose performance over time?

Posted on 2013-05-27
8
487 Views
Last Modified: 2013-05-29
So I've been noticing slowdown of my 4-disk RAID10 setup on my ICH10R.  I've never actually benchmarked it, but after recently installing an SSD I started doing so for comparison.

I was getting abysmal performance metrics of my RAID10 volume.  Using CrystalDiskMark 3.0.2 x64, I was seeing the following:

------------------------------------------------------------------------
Volume:  Raid10 at 64k stripe
Test : 500 MB [D: 61% (363.0/596.0 GB)] (x5)
OS : Windows 8 Professional [6.2 Build 9200] (x64)
------------------------------------------------------------------------
Sequential Read :  76.67 MB/s                  Sequential Write :  133.9 MB/s
Random Read 512KB :   26.85 MB/s            Random Write 512KB :  57.95 MB/s
Random Read 4KB (QD=1) :  0.317 MB/s            Random Write 4KB (QD=1) :  2.273 MB/s
Random Read 4KB (QD=32) :  2.040 MB/s      Random Write 4KB (QD=32) :  2.371 MB/s


I intended to rebuild into a RAID0 configuration as my backup scheme has become far more robust, but I wanted to test the RAID10 again after blowing it away and recreating it.  After doing so, my performance increased nearly 3 fold for seq. read, and up to 2 fold for other metrics.  

------------------------------------------------------------------------
Volume:  Raid10 at 64k stripe
Test : 500 MB [D: 0.0% (0.2/596.0 GB)] (x5)
OS : Windows 8 Professional [6.2 Build 9200] (x64)
------------------------------------------------------------------------
Sequential Read :  212.549 MB/s            Sequential Write :  190.535 MB/s
Random Read 512KB :  44.323 MB/s            Random Write 512KB :   80.019 MB/s
Random Read 4KB (QD=1) :  0.652 MB/s            Random Write 4KB (QD=1) :  2.240 MB/s
Random Read 4KB (QD=32) :  4.302 MB/s      Random Write 4KB (QD=32) :  3.216 MB/s


At first I figured it may be due to fragmentation, but Windows did report 0 fragmentation before the first test.  Could it be the disk usage being 61% on the first test and only 0% on the second?  Or does RAID actually lose performance over time?  I've never heard of that.

Outside of that, these numbers don't reflect what I would expect from a 4 drive RAID10.  It is my understanding that the onboard ICH10R chipset has some issues with RAID10, per some searching I found.  Too bad SSD is expensive/GB, otherwise I'd replace this RAID entirely.

For comparison, my SSD:

------------------------------------------------------------------------
Drive:  Samsung 840 PRO SSD
Test : 500 MB [D: 66.0% (79/119.0 GB)] (x5)
OS : Windows 8 Professional [6.2 Build 9200] (x64)
------------------------------------------------------------------------
Sequential Read :  467.2 MB/s            Sequential Write :  333.6 MB/s
Random Read 512KB :  406.2 MB/s            Random Write 512KB :   178.1 MB/s
Random Read 4KB (QD=1) :  24.07 MB/s            Random Write 4KB (QD=1) :  60.98 MB/s
Random Read 4KB (QD=32) :  328.4 MB/s      Random Write 4KB (QD=32) :  176.9 MB/s



Same 4 HDD above in Raid0 128K  (I have large files)

------------------------------------------------------------------------
Volume:  Raid0 at 128k stripe
Test : 500 MB [D: 0.0% (0.2/1192.0 GB)] (x5)
OS : Windows 8 Professional [6.2 Build 9200] (x64)
------------------------------------------------------------------------
Sequential Read :  409.5 MB/s            Sequential Write :  299.6 MB/s
Random Read 512KB :  55.06 MB/s            Random Write 512KB :   124.7 MB/s
Random Read 4KB (QD=1) :  0.702 MB/s            Random Write 4KB (QD=1) :  4.726 MB/s
Random Read 4KB (QD=32) :  4.405 MB/s      Random Write 4KB (QD=32) :  7.088 MB/s
0
Comment
Question by:mcdonamw79
  • 4
  • 4
8 Comments
 
LVL 47

Expert Comment

by:dlethe
ID: 39200180
Well, first, that controller is crap. It doesn't even do read load balancing.  If you want to double read performance (in perfect world), turn off that RAID controller and let the O/S do 2 x RAID1 instead of a single RAID10.  

Your performance hit is most likely bad block recovery, which that controller , (if you want to call that a controller), doesn't handle well. It locks up and doesn't get the data from the other HDD in RAID set in parallel with remapping a bad block.
0
 

Author Comment

by:mcdonamw79
ID: 39200236
Yeah I recently found out about the lack of read load balancing, which was rather discerning.  Crap controller aside, would the bad block recovery you speak of come into play in all of my testing scenarios?  I'm using the same controller in each test, so I would expect its shortcomings to exist regardless.  

Ultimately I'm not understanding why recreating the RAID10 netted me such greater performance.  That's all I'm really looking to understand here.
0
 
LVL 47

Expert Comment

by:dlethe
ID: 39200291
Well bad blocks only happen once per block, as each one is remapped.  Only way to know for sure is to get some decent diagnostic software that maps out all blocks (without attempting recovery) before you run a test.

that software isn't cheap.   Performance has two major metrics, throughput and I/Os per second. They are mutually exclusive and all have different performance characteristics based on block size, reads vs. writes, random vs. sequential, and permutations involving combinations of all of the above.    Suffice to say, you're wasting your time tuning with this fakeraid config. No way can you get better performance no matter what you do with that controller vs. just using native host-based software raid.

Also benchmarking is rarely done right, and don't get hung up by synthetic loads that they generate.  Even many storage pros screw them up.  If your O/S does journaling and if you are benching a mounted filesystem or even a boot device and aren't looking at recovered/unrecovered errors then you can't possibly nail the results.

Spend your time dumping that fakeraid and going to software RAID. It is a better use of your time.
0
Best Practices: Disaster Recovery Testing

Besides backup, any IT division should have a disaster recovery plan. You will find a few tips below relating to the development of such a plan and to what issues one should pay special attention in the course of backup planning.

 

Author Comment

by:mcdonamw79
ID: 39203208
I concede that trying to get better performance from the ICH10R is pointless. I am not trying to do this. I am merely trying to understand why breaking an existing array and recreating it exactly the same yielded 2-3 times performance increase. Am I merely misunderstanding you?
0
 
LVL 47

Expert Comment

by:dlethe
ID: 39203261
Because you likely had RECOVERABLE read errors.  A rebuild forces full reads of all blocks, and it could take up to 30 secs for a single block.   This can be confirmed (well, could have before you began the rebuild).
0
 

Author Comment

by:mcdonamw79
ID: 39205712
Gotcha. Perhaps that is what it was. I do know I ran a verify on the Raid array using the Intel RST before I blew it away. It reported no errors. Perhaps I will never know. Either way I think my primary question is answered. That was: do RAID arrays lose performance over time. Outside of block errors I am assuming this is not possible. Correct?
0
 
LVL 47

Accepted Solution

by:
dlethe earned 500 total points
ID: 39205747
Correct, allowing also for a statistically insignificant factors such as entropy, heat due to crud that builds up everywhere, etc ..
0
 

Author Closing Comment

by:mcdonamw79
ID: 39206269
Thank you for your time.
0

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Storage devices are generally used to save the data or sometime transfer the data from one computer system to another system. However, sometimes user accidentally erased their important data from the Storage devices. Users have to know how data reco…
Create your own, high-performance VM backup appliance by installing NAKIVO Backup & Replication directly onto a Synology NAS!
This video Micro Tutorial explains how to clone a hard drive using a commercial software product for Windows systems called Casper from Future Systems Solutions (FSS). Cloning makes an exact, complete copy of one hard disk drive (HDD) onto another d…
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…

839 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question