Very slow Array access

I am having an issue with very slow transfer speeds to and from my RAID 5 array.  Drives are WD 1TB Caviar black x3 in a RAID 5 configuration, RAID controller is a 3ware 9650 SATA RAID controller, no battery, so I have write caching disabled.  OS is server 2008.  

I'm moving files from a 2TB single WD Black which is on an onboard SATA controller.  

Transfer speed is ~3MB/s.

However, I have done testing trying to load folders and looking at performance monitor, and I can tell that it is on the array side, not the other drive.

Any ideas how to diagnose such an issue?  RAID controller does not detect any failed drives.
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Aaron TomoskySD-WAN SimplifiedCommented:
How are you moving the files? If its a sync transfer like Nfs that can get rediculously slow on an unchached raid 5. Are copies from the raid 5 to a single drive fast?
3MB a second is about right for this type of I/O in such a config.

You want better performance?  Get a battery and enable cache.

P.S. WD's official policy on WD Caviar Black drives in RAID5 configurations is that this violates the warranty and you risk data loss.  Those disks are simply not suitable for this type of workload and usage patterns.  You need enterprise class drives, especially on Win2K8.
dbestcomputersAuthor Commented:
aarontomosky: I tried drag and drop first, now I'm transferring using command line xcopy.  I can see transfer speeds in performance monitor or while drag and dropping.

dlethe: 3MB/s is about right?  That doesn't make sense, should I just enable the write caching then?  RAID5, especially "in this config", as in on a separate hardware controller with it's own cpu and RAM, should perform faster than the a single drive by itself.

Straight off WD:
Transfer Rate (Buffer To Disk)      126 MB/s (Max)

3MB/s is not by any stretch of the imagination OK for a RAID5 array to perform.  I'll let you know how I fixed it after I figure out what the deal is...

And what would 2k8 have to do with whether I need an enterprise level drive or not? As opposed to? What 2003?  Or as opposed to a workstation OS?  They just mean it's a workstation drive and not designed for 24/7 load.  I'm not worried about that right now, I want to fix the ridiculous transfer rate.

Transfers on a machine beside it with an eSata RAID tower using a cheap ass rocketraid card to an 8 drive array with Caviar GREEN drives are performing at over 30MB/s, how does that work if 3MB/s is correct on my nice hardware?
Making Bulk Changes to Active Directory

Watch this video to see how easy it is to make mass changes to Active Directory from an external text file without using complicated scripts.

It makes perfect sense.  Your I/Os are likely 4KB in size.  Add overhead of filesystem and fact that you have to update NTFS logs, and last-file-access times every time you even look at a file.  Now add fact that there is no I/O queuing, and the RAID5 write hole penalty; and the fact that every drive has to be involved in every I/O (several times) ...

You probably also set your RAID5 with an I/O stripe size that is much greater then the native I/O size that NTFS is doing.  

As for the choice of drives, they are engineered for 2400 hours use per year, not 24x7.  The data recovery logic is tuned for deep recovery in event of a read error and disk locks up as much as 30 seconds in event of a bad block.  One bad block and all IO stops. Furthermore, get a few bad blocks and then the 3Ware card may think the drive died, so it will degrade your RAID.

I could go on about how unsuitable the WDD drive is, but this is from WD's data sheet, but below should convince you ... it is from the Caviar Black data sheet.  If WD says not to use the drive, isn't that enough??

"WD Caviar Black Hard Drives are tested and recommended for use in consumer-type RAID applications (RAID-0 /RAID-1).
 WD Caviar Black Hard Drives are not recommended for and are not warranted for use in RAID environments utilizing Enterprise HBAs and/or
expanders and in multi-bay chassis, as they are not designed for, nor tested in, these specific types of RAID applications. For all Business Critical RAID applications, please consider WD’s Enterprise Hard Drives that are specifically designed with RAID-specific, time-limited error recovery (TLER), are tested extensively in 24x7 RAID applications, and include features like enhanced RAFF technology and thermal extended burn-in testing.
dbestcomputersAuthor Commented:
OK fine, so tell tme this? Updated firmware to 9.5.4 on the 9650SE controller, updated driver set to matching 9.5.4, enabled write caching, average speed is now OVER 60MB/s.  

I appreciate your concern about using the black drives, but the fact is, when purchased a WD black was probabably about 5 times less expensive than a true enterprise class drive.  Also, you don't know my application.  We will be shutting down at 5:00 every day, backing up, then shutting the server down, so it's not really getting any more use than a standard workstation, and it's definitely not in a 24/7 environment.

Thanks for the comments anyway though, if you do have any advice for speeding up my array speeds just let me know.  I am using a 64kb stripe, you mentioned something above?
Enabled write caching?  Exactly as I said you should do.  

Respectfully, WD knows more about how their HDDs should be used then you do, and you do NOT understand the implications of TLER.   If you did, then this would not be open for discussion.  Please google TLER and RAID.  The issue is how the FIRMWARE is architected to handle errors.

A desktop drive is programmed with expectation that there is only one copy of the wedding photo, so in event of an unrecovered read, it is going to lock up for up to 60 seconds to get it.   An enterprise drive's firmware is designed with expectation that there is redundant data, and that you don't want everything to stop and wait while the disk goes into deep recovery cycle.  So enterprise drives have 10X - 100X better ECC, and they will give up after 2-3 seconds.

Unfortunately, the FIRMWARE in many premium RAID controllers are designed for enterprise drives that have this sort of error recovery timing.   So they will kill a non-responsive drive after 7-10 seconds.   So if your controller's firmware is designed for enterprise drives, and your disk has to go into a deep recovery mode, the controller will think the disk failed, and will kick it off the RAID.
dbestcomputersAuthor Commented:
I changed the StorSave setting (don't know what it is) to Performance instead of Balanced, it then said that would make my battery useless (I don't have one, so figured "what the hell"), now I got 100MB/s write speed! Yay!  

The whole server is on a huge rack mounted battery backup, so I'm not too worried about sudden power loss.
Of course it will run faster. You have configured your controller to guarantee data loss in event of a drive failure, an ECC error, or unrecoverable error.  (Not just power loss).
dlethe *really* does know his stuff as regards drives from all I've seen on these forums :)  Please take strong note of his advice on this.  Looks like what you've got right now is a RAID system that will perform well but loses all the reliablility benefits of RAID.  Might as well have bought yourself a single SSD instead.  Or just keep it all in RAM.
Why not just spend maybe $200 more and get the cache module, and another disk and at least go RAID10?  Then at least you have about 1000X more reliability and less risk of data loss.

If, no WHEN, you have data loss, what do you think would happen if somebody found out that you knowingly put unsupported disks in the system, and effectively turned off all reliability and redundancy, all to save a few hundred dollars?  

If you are doing this for a customer, then you are opening your firm up for litigation.  I have testified as expert witness before, and trust me, it just isn't worth it.

When the WD data sheet TELLS you flat out that your configuration is UNTESTED, AND NOT WARRANTED for the very configuration you are creating, do you think it will go well for you personally?  Or your firm, or your career?

These disks go for about $35 to tier-1s.  Thousands of dollars worth of equipment, licenses and human resources put at risk on $100 worth of disk drives that aren't designed to run the way you configured them.

Think about it!!
andyalderSaggar maker's framemakerCommented:
RocketRAID default cache policy is write-back, no wonder it's faster than the other card assuming you haven't changed it to write through.
Aaron TomoskySD-WAN SimplifiedCommented:
Think about your setup from a "what if this part dies" position. If a drive goes bad, you can put another one in and it will take about 24 hour to rebuild. As long as a second doesn't fail during that time, you're ok.
If your box loses power during a write, because you have write cache enabled without a battery yOu will lose data and possible corrupt the whole raid.

As for the disks, they may work ok for you or you may get a 60 second pause and they drop from the raid like I did. Seagate constellation drives are 2x regular drives and a much better choice.

Like you, I'm a fan of good'nuf. The really dangerous thing in your setup  is the write cache.
Personally I left the whole $500 raid card $200 battery $800 tler drive world and now hang out in zfs land. Check out zfsguru. It's some reading and learning of new stuff but I'm not lookin back.
I'm with you, aarontomosky -- I put EVERYTHING I care about on ZFS.   One of the best things about ZFS is that it flushes on writes, so I just use a bunch of cheap desktop SATA drives in the ZFS equivalent of RAID6

Then I have a pair of SSDs mirrored as the ZIL.

Hot snapshot, filesystem compression, de-dup, RAIDZ2, and consumer drives.  Outperforms and is more reliable then a $1000 RAID board.
dbestcomputersAuthor Commented:
"Of course it will run faster. You have configured your controller to guarantee data loss in event of a drive failure, an ECC error, or unrecoverable error.  (Not just power loss)."

Why/How have I guaranteed data loss in the event of drive failure, that's the whole point of having the RAID5 config isn't it?  The striping with parity.

"You probably also set your RAID5 with an I/O stripe size that is much greater then the native I/O size that NTFS is doing. "

I have a 64kb stripe size, should this be changed?

"RocketRAID default cache policy is write-back, no wonder it's faster than the other card assuming you haven't changed it to write through."

Can I change this?  What difference will it make?
andyalderSaggar maker's framemakerCommented:
You have two sources of data loss / corruption without even having a hard failure in your current setup, at least not having a failure today...

RAID 5 writes are meant to be atomic, that is the data and parity are meant to be written at exactly the same time. Unless you synchronise the disk spindles (which Compaq did with custom disks on one particular controller/server) that is not possible. So you might update the data before the parity, then get a power failure and restart but you haven't updated the parity. All is well and you can even verify all the data is still good but then a month later a disk fails, you replace it but the data on that disk is rebuilt from incorrect parity. You've fallen into the RAID 5 write hole. None of the data on a RAID 5 volume can be trusted after a power failure, not even stuff that was written years ago. Battery backed cache gets around that problem by writing the missing parity when the power comes back on.

The other situation is separate from the lack of atomicity and is the write-back setting, you probably have the controller set to tell the application that the data's been comitted to disk when really it's only in cache. If it was an ATM you would have given the customer their cache before you updated their bank balance. You can change that with the write-through/write back setting in BIOS, but neither setting covers the write hole.
dbestcomputersAuthor Commented:
Thank you Andy.  Makes sense.  I'll look into making this more "safe".  But I hate to sacrifice the speed.  I would've thought with all the cache on this RAID card it would've worked better...  

Again, is there a speed difference by changing the stripe size?
andyalderSaggar maker's framemakerCommented:
All that cache but no battery means it is probably just read cache, but you list two servers, one fast because it's got the stops pulled out at a risk of data corruption and the other slow and safe.

As far as the stripe size is concerned your mileage may vary, there isn't a magic figure; if there was the controller manufacture wouldn't bother to give you the option with playing with it.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Server Hardware

From novice to tech pro — start learning today.