Solved

Very slow Array access

Posted on 2012-03-24
17
409 Views
Last Modified: 2012-04-11
I am having an issue with very slow transfer speeds to and from my RAID 5 array.  Drives are WD 1TB Caviar black x3 in a RAID 5 configuration, RAID controller is a 3ware 9650 SATA RAID controller, no battery, so I have write caching disabled.  OS is server 2008.  

I'm moving files from a 2TB single WD Black which is on an onboard SATA controller.  

Transfer speed is ~3MB/s.

However, I have done testing trying to load folders and looking at performance monitor, and I can tell that it is on the array side, not the other drive.

Any ideas how to diagnose such an issue?  RAID controller does not detect any failed drives.
0
Comment
Question by:dbestcomputers
  • 6
  • 5
  • 3
  • +2
17 Comments
 
LVL 38

Expert Comment

by:Aaron Tomosky
Comment Utility
How are you moving the files? If its a sync transfer like Nfs that can get rediculously slow on an unchached raid 5. Are copies from the raid 5 to a single drive fast?
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
3MB a second is about right for this type of I/O in such a config.

You want better performance?  Get a battery and enable cache.

P.S. WD's official policy on WD Caviar Black drives in RAID5 configurations is that this violates the warranty and you risk data loss.  Those disks are simply not suitable for this type of workload and usage patterns.  You need enterprise class drives, especially on Win2K8.
0
 

Author Comment

by:dbestcomputers
Comment Utility
aarontomosky: I tried drag and drop first, now I'm transferring using command line xcopy.  I can see transfer speeds in performance monitor or while drag and dropping.

dlethe: 3MB/s is about right?  That doesn't make sense, should I just enable the write caching then?  RAID5, especially "in this config", as in on a separate hardware controller with it's own cpu and RAM, should perform faster than the a single drive by itself.

Straight off WD:
Transfer Rate (Buffer To Disk)      126 MB/s (Max)

3MB/s is not by any stretch of the imagination OK for a RAID5 array to perform.  I'll let you know how I fixed it after I figure out what the deal is...

And what would 2k8 have to do with whether I need an enterprise level drive or not? As opposed to? What 2003?  Or as opposed to a workstation OS?  They just mean it's a workstation drive and not designed for 24/7 load.  I'm not worried about that right now, I want to fix the ridiculous transfer rate.

Transfers on a machine beside it with an eSata RAID tower using a cheap ass rocketraid card to an 8 drive array with Caviar GREEN drives are performing at over 30MB/s, how does that work if 3MB/s is correct on my nice hardware?
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
It makes perfect sense.  Your I/Os are likely 4KB in size.  Add overhead of filesystem and fact that you have to update NTFS logs, and last-file-access times every time you even look at a file.  Now add fact that there is no I/O queuing, and the RAID5 write hole penalty; and the fact that every drive has to be involved in every I/O (several times) ...

You probably also set your RAID5 with an I/O stripe size that is much greater then the native I/O size that NTFS is doing.  

As for the choice of drives, they are engineered for 2400 hours use per year, not 24x7.  The data recovery logic is tuned for deep recovery in event of a read error and disk locks up as much as 30 seconds in event of a bad block.  One bad block and all IO stops. Furthermore, get a few bad blocks and then the 3Ware card may think the drive died, so it will degrade your RAID.

I could go on about how unsuitable the WDD drive is, but this is from WD's data sheet, but below should convince you ... it is from the Caviar Black data sheet.  If WD says not to use the drive, isn't that enough??


"WD Caviar Black Hard Drives are tested and recommended for use in consumer-type RAID applications (RAID-0 /RAID-1).
 WD Caviar Black Hard Drives are not recommended for and are not warranted for use in RAID environments utilizing Enterprise HBAs and/or
expanders and in multi-bay chassis, as they are not designed for, nor tested in, these specific types of RAID applications. For all Business Critical RAID applications, please consider WD’s Enterprise Hard Drives that are specifically designed with RAID-specific, time-limited error recovery (TLER), are tested extensively in 24x7 RAID applications, and include features like enhanced RAFF technology and thermal extended burn-in testing.
0
 

Author Comment

by:dbestcomputers
Comment Utility
OK fine, so tell tme this? Updated firmware to 9.5.4 on the 9650SE controller, updated driver set to matching 9.5.4, enabled write caching, average speed is now OVER 60MB/s.  

I appreciate your concern about using the black drives, but the fact is, when purchased a WD black was probabably about 5 times less expensive than a true enterprise class drive.  Also, you don't know my application.  We will be shutting down at 5:00 every day, backing up, then shutting the server down, so it's not really getting any more use than a standard workstation, and it's definitely not in a 24/7 environment.

Thanks for the comments anyway though, if you do have any advice for speeding up my array speeds just let me know.  I am using a 64kb stripe, you mentioned something above?
0
 
LVL 47

Assisted Solution

by:dlethe
dlethe earned 200 total points
Comment Utility
Enabled write caching?  Exactly as I said you should do.  

Respectfully, WD knows more about how their HDDs should be used then you do, and you do NOT understand the implications of TLER.   If you did, then this would not be open for discussion.  Please google TLER and RAID.  The issue is how the FIRMWARE is architected to handle errors.

A desktop drive is programmed with expectation that there is only one copy of the wedding photo, so in event of an unrecovered read, it is going to lock up for up to 60 seconds to get it.   An enterprise drive's firmware is designed with expectation that there is redundant data, and that you don't want everything to stop and wait while the disk goes into deep recovery cycle.  So enterprise drives have 10X - 100X better ECC, and they will give up after 2-3 seconds.

Unfortunately, the FIRMWARE in many premium RAID controllers are designed for enterprise drives that have this sort of error recovery timing.   So they will kill a non-responsive drive after 7-10 seconds.   So if your controller's firmware is designed for enterprise drives, and your disk has to go into a deep recovery mode, the controller will think the disk failed, and will kick it off the RAID.
0
 

Author Comment

by:dbestcomputers
Comment Utility
I changed the StorSave setting (don't know what it is) to Performance instead of Balanced, it then said that would make my battery useless (I don't have one, so figured "what the hell"), now I got 100MB/s write speed! Yay!  

The whole server is on a huge rack mounted battery backup, so I'm not too worried about sudden power loss.
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
Of course it will run faster. You have configured your controller to guarantee data loss in event of a drive failure, an ECC error, or unrecoverable error.  (Not just power loss).
0
Want to promote your upcoming event?

Attending an event? Speaking at a conference? Or exhibiting at a tradeshow? Easily inform your contacts by using a promotional banner in your email signature. This will ensure your organization’s most important contacts are in the know.

 
LVL 3

Expert Comment

by:StuWhitby
Comment Utility
dlethe *really* does know his stuff as regards drives from all I've seen on these forums :)  Please take strong note of his advice on this.  Looks like what you've got right now is a RAID system that will perform well but loses all the reliablility benefits of RAID.  Might as well have bought yourself a single SSD instead.  Or just keep it all in RAM.
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
Why not just spend maybe $200 more and get the cache module, and another disk and at least go RAID10?  Then at least you have about 1000X more reliability and less risk of data loss.

If, no WHEN, you have data loss, what do you think would happen if somebody found out that you knowingly put unsupported disks in the system, and effectively turned off all reliability and redundancy, all to save a few hundred dollars?  

If you are doing this for a customer, then you are opening your firm up for litigation.  I have testified as expert witness before, and trust me, it just isn't worth it.

When the WD data sheet TELLS you flat out that your configuration is UNTESTED, AND NOT WARRANTED for the very configuration you are creating, do you think it will go well for you personally?  Or your firm, or your career?

These disks go for about $35 to tier-1s.  Thousands of dollars worth of equipment, licenses and human resources put at risk on $100 worth of disk drives that aren't designed to run the way you configured them.

Think about it!!
0
 
LVL 55

Expert Comment

by:andyalder
Comment Utility
RocketRAID default cache policy is write-back, no wonder it's faster than the other card assuming you haven't changed it to write through.
0
 
LVL 38

Expert Comment

by:Aaron Tomosky
Comment Utility
Think about your setup from a "what if this part dies" position. If a drive goes bad, you can put another one in and it will take about 24 hour to rebuild. As long as a second doesn't fail during that time, you're ok.
If your box loses power during a write, because you have write cache enabled without a battery yOu will lose data and possible corrupt the whole raid.

As for the disks, they may work ok for you or you may get a 60 second pause and they drop from the raid like I did. Seagate constellation drives are 2x regular drives and a much better choice.

Like you, I'm a fan of good'nuf. The really dangerous thing in your setup  is the write cache.
Personally I left the whole $500 raid card $200 battery $800 tler drive world and now hang out in zfs land. Check out zfsguru. It's some reading and learning of new stuff but I'm not lookin back.
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
I'm with you, aarontomosky -- I put EVERYTHING I care about on ZFS.   One of the best things about ZFS is that it flushes on writes, so I just use a bunch of cheap desktop SATA drives in the ZFS equivalent of RAID6

Then I have a pair of SSDs mirrored as the ZIL.

Hot snapshot, filesystem compression, de-dup, RAIDZ2, and consumer drives.  Outperforms and is more reliable then a $1000 RAID board.
0
 

Author Comment

by:dbestcomputers
Comment Utility
"Of course it will run faster. You have configured your controller to guarantee data loss in event of a drive failure, an ECC error, or unrecoverable error.  (Not just power loss)."

Why/How have I guaranteed data loss in the event of drive failure, that's the whole point of having the RAID5 config isn't it?  The striping with parity.

"You probably also set your RAID5 with an I/O stripe size that is much greater then the native I/O size that NTFS is doing. "

I have a 64kb stripe size, should this be changed?

"RocketRAID default cache policy is write-back, no wonder it's faster than the other card assuming you haven't changed it to write through."

Can I change this?  What difference will it make?
0
 
LVL 55

Expert Comment

by:andyalder
Comment Utility
You have two sources of data loss / corruption without even having a hard failure in your current setup, at least not having a failure today...

RAID 5 writes are meant to be atomic, that is the data and parity are meant to be written at exactly the same time. Unless you synchronise the disk spindles (which Compaq did with custom disks on one particular controller/server) that is not possible. So you might update the data before the parity, then get a power failure and restart but you haven't updated the parity. All is well and you can even verify all the data is still good but then a month later a disk fails, you replace it but the data on that disk is rebuilt from incorrect parity. You've fallen into the RAID 5 write hole. None of the data on a RAID 5 volume can be trusted after a power failure, not even stuff that was written years ago. Battery backed cache gets around that problem by writing the missing parity when the power comes back on.

The other situation is separate from the lack of atomicity and is the write-back setting, you probably have the controller set to tell the application that the data's been comitted to disk when really it's only in cache. If it was an ATM you would have given the customer their cache before you updated their bank balance. You can change that with the write-through/write back setting in BIOS, but neither setting covers the write hole.
0
 

Author Comment

by:dbestcomputers
Comment Utility
Thank you Andy.  Makes sense.  I'll look into making this more "safe".  But I hate to sacrifice the speed.  I would've thought with all the cache on this RAID card it would've worked better...  

Again, is there a speed difference by changing the stripe size?
0
 
LVL 55

Accepted Solution

by:
andyalder earned 300 total points
Comment Utility
All that cache but no battery means it is probably just read cache, but you list two servers, one fast because it's got the stops pulled out at a risk of data corruption and the other slow and safe.

As far as the stripe size is concerned your mileage may vary, there isn't a magic figure; if there was the controller manufacture wouldn't bother to give you the option with playing with it.
0

Featured Post

6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

Join & Write a Comment

The 6120xp switches seem to have a bug when you create a fiber port channel when you have a UCS fabric interconnects talking to them.  If you follow the Cisco guide for the UCS, the FC Port channel will never come up and it will say that there are n…
Restoring deleted objects in Active Directory has been a standard feature in Active Directory for many years, yet some admins may not know what is available.
This tutorial will show how to push an installation of Backup Exec to an additional server in both 2012 and 2014 versions of the software. Click on the Backup Exec button in the upper left corner. From here, select Installation and Licensing, then I…
This tutorial will walk an individual through locating and launching the BEUtility application to properly change the service account username and\or password in situation where it may be necessary or where the password has been inadvertently change…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now