Expiring Today—Celebrate National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Very slow Array access

Posted on 2012-03-24
17
Medium Priority
?
461 Views
Last Modified: 2012-04-11
I am having an issue with very slow transfer speeds to and from my RAID 5 array.  Drives are WD 1TB Caviar black x3 in a RAID 5 configuration, RAID controller is a 3ware 9650 SATA RAID controller, no battery, so I have write caching disabled.  OS is server 2008.  

I'm moving files from a 2TB single WD Black which is on an onboard SATA controller.  

Transfer speed is ~3MB/s.

However, I have done testing trying to load folders and looking at performance monitor, and I can tell that it is on the array side, not the other drive.

Any ideas how to diagnose such an issue?  RAID controller does not detect any failed drives.
0
Comment
Question by:dbestcomputers
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 5
  • 3
  • +2
17 Comments
 
LVL 39

Expert Comment

by:Aaron Tomosky
ID: 37762183
How are you moving the files? If its a sync transfer like Nfs that can get rediculously slow on an unchached raid 5. Are copies from the raid 5 to a single drive fast?
0
 
LVL 47

Expert Comment

by:David
ID: 37762187
3MB a second is about right for this type of I/O in such a config.

You want better performance?  Get a battery and enable cache.

P.S. WD's official policy on WD Caviar Black drives in RAID5 configurations is that this violates the warranty and you risk data loss.  Those disks are simply not suitable for this type of workload and usage patterns.  You need enterprise class drives, especially on Win2K8.
0
 

Author Comment

by:dbestcomputers
ID: 37762209
aarontomosky: I tried drag and drop first, now I'm transferring using command line xcopy.  I can see transfer speeds in performance monitor or while drag and dropping.

dlethe: 3MB/s is about right?  That doesn't make sense, should I just enable the write caching then?  RAID5, especially "in this config", as in on a separate hardware controller with it's own cpu and RAM, should perform faster than the a single drive by itself.

Straight off WD:
Transfer Rate (Buffer To Disk)      126 MB/s (Max)

3MB/s is not by any stretch of the imagination OK for a RAID5 array to perform.  I'll let you know how I fixed it after I figure out what the deal is...

And what would 2k8 have to do with whether I need an enterprise level drive or not? As opposed to? What 2003?  Or as opposed to a workstation OS?  They just mean it's a workstation drive and not designed for 24/7 load.  I'm not worried about that right now, I want to fix the ridiculous transfer rate.

Transfers on a machine beside it with an eSata RAID tower using a cheap ass rocketraid card to an 8 drive array with Caviar GREEN drives are performing at over 30MB/s, how does that work if 3MB/s is correct on my nice hardware?
0
Three Reasons Why Backup is Strategic

Backup is strategic to your business because your data is strategic to your business. Without backup, your business will fail. This white paper explains why it is vital for you to design and immediately execute a backup strategy to protect 100 percent of your data.

 
LVL 47

Expert Comment

by:David
ID: 37762230
It makes perfect sense.  Your I/Os are likely 4KB in size.  Add overhead of filesystem and fact that you have to update NTFS logs, and last-file-access times every time you even look at a file.  Now add fact that there is no I/O queuing, and the RAID5 write hole penalty; and the fact that every drive has to be involved in every I/O (several times) ...

You probably also set your RAID5 with an I/O stripe size that is much greater then the native I/O size that NTFS is doing.  

As for the choice of drives, they are engineered for 2400 hours use per year, not 24x7.  The data recovery logic is tuned for deep recovery in event of a read error and disk locks up as much as 30 seconds in event of a bad block.  One bad block and all IO stops. Furthermore, get a few bad blocks and then the 3Ware card may think the drive died, so it will degrade your RAID.

I could go on about how unsuitable the WDD drive is, but this is from WD's data sheet, but below should convince you ... it is from the Caviar Black data sheet.  If WD says not to use the drive, isn't that enough??


"WD Caviar Black Hard Drives are tested and recommended for use in consumer-type RAID applications (RAID-0 /RAID-1).
 WD Caviar Black Hard Drives are not recommended for and are not warranted for use in RAID environments utilizing Enterprise HBAs and/or
expanders and in multi-bay chassis, as they are not designed for, nor tested in, these specific types of RAID applications. For all Business Critical RAID applications, please consider WD’s Enterprise Hard Drives that are specifically designed with RAID-specific, time-limited error recovery (TLER), are tested extensively in 24x7 RAID applications, and include features like enhanced RAFF technology and thermal extended burn-in testing.
0
 

Author Comment

by:dbestcomputers
ID: 37762686
OK fine, so tell tme this? Updated firmware to 9.5.4 on the 9650SE controller, updated driver set to matching 9.5.4, enabled write caching, average speed is now OVER 60MB/s.  

I appreciate your concern about using the black drives, but the fact is, when purchased a WD black was probabably about 5 times less expensive than a true enterprise class drive.  Also, you don't know my application.  We will be shutting down at 5:00 every day, backing up, then shutting the server down, so it's not really getting any more use than a standard workstation, and it's definitely not in a 24/7 environment.

Thanks for the comments anyway though, if you do have any advice for speeding up my array speeds just let me know.  I am using a 64kb stripe, you mentioned something above?
0
 
LVL 47

Assisted Solution

by:David
David earned 800 total points
ID: 37762721
Enabled write caching?  Exactly as I said you should do.  

Respectfully, WD knows more about how their HDDs should be used then you do, and you do NOT understand the implications of TLER.   If you did, then this would not be open for discussion.  Please google TLER and RAID.  The issue is how the FIRMWARE is architected to handle errors.

A desktop drive is programmed with expectation that there is only one copy of the wedding photo, so in event of an unrecovered read, it is going to lock up for up to 60 seconds to get it.   An enterprise drive's firmware is designed with expectation that there is redundant data, and that you don't want everything to stop and wait while the disk goes into deep recovery cycle.  So enterprise drives have 10X - 100X better ECC, and they will give up after 2-3 seconds.

Unfortunately, the FIRMWARE in many premium RAID controllers are designed for enterprise drives that have this sort of error recovery timing.   So they will kill a non-responsive drive after 7-10 seconds.   So if your controller's firmware is designed for enterprise drives, and your disk has to go into a deep recovery mode, the controller will think the disk failed, and will kick it off the RAID.
0
 

Author Comment

by:dbestcomputers
ID: 37762749
I changed the StorSave setting (don't know what it is) to Performance instead of Balanced, it then said that would make my battery useless (I don't have one, so figured "what the hell"), now I got 100MB/s write speed! Yay!  

The whole server is on a huge rack mounted battery backup, so I'm not too worried about sudden power loss.
0
 
LVL 47

Expert Comment

by:David
ID: 37762761
Of course it will run faster. You have configured your controller to guarantee data loss in event of a drive failure, an ECC error, or unrecoverable error.  (Not just power loss).
0
 
LVL 3

Expert Comment

by:StuWhitby
ID: 37762949
dlethe *really* does know his stuff as regards drives from all I've seen on these forums :)  Please take strong note of his advice on this.  Looks like what you've got right now is a RAID system that will perform well but loses all the reliablility benefits of RAID.  Might as well have bought yourself a single SSD instead.  Or just keep it all in RAM.
0
 
LVL 47

Expert Comment

by:David
ID: 37763047
Why not just spend maybe $200 more and get the cache module, and another disk and at least go RAID10?  Then at least you have about 1000X more reliability and less risk of data loss.

If, no WHEN, you have data loss, what do you think would happen if somebody found out that you knowingly put unsupported disks in the system, and effectively turned off all reliability and redundancy, all to save a few hundred dollars?  

If you are doing this for a customer, then you are opening your firm up for litigation.  I have testified as expert witness before, and trust me, it just isn't worth it.

When the WD data sheet TELLS you flat out that your configuration is UNTESTED, AND NOT WARRANTED for the very configuration you are creating, do you think it will go well for you personally?  Or your firm, or your career?

These disks go for about $35 to tier-1s.  Thousands of dollars worth of equipment, licenses and human resources put at risk on $100 worth of disk drives that aren't designed to run the way you configured them.

Think about it!!
0
 
LVL 56

Expert Comment

by:andyalder
ID: 37764695
RocketRAID default cache policy is write-back, no wonder it's faster than the other card assuming you haven't changed it to write through.
0
 
LVL 39

Expert Comment

by:Aaron Tomosky
ID: 37766158
Think about your setup from a "what if this part dies" position. If a drive goes bad, you can put another one in and it will take about 24 hour to rebuild. As long as a second doesn't fail during that time, you're ok.
If your box loses power during a write, because you have write cache enabled without a battery yOu will lose data and possible corrupt the whole raid.

As for the disks, they may work ok for you or you may get a 60 second pause and they drop from the raid like I did. Seagate constellation drives are 2x regular drives and a much better choice.

Like you, I'm a fan of good'nuf. The really dangerous thing in your setup  is the write cache.
Personally I left the whole $500 raid card $200 battery $800 tler drive world and now hang out in zfs land. Check out zfsguru. It's some reading and learning of new stuff but I'm not lookin back.
0
 
LVL 47

Expert Comment

by:David
ID: 37766243
I'm with you, aarontomosky -- I put EVERYTHING I care about on ZFS.   One of the best things about ZFS is that it flushes on writes, so I just use a bunch of cheap desktop SATA drives in the ZFS equivalent of RAID6

Then I have a pair of SSDs mirrored as the ZIL.

Hot snapshot, filesystem compression, de-dup, RAIDZ2, and consumer drives.  Outperforms and is more reliable then a $1000 RAID board.
0
 

Author Comment

by:dbestcomputers
ID: 37823471
"Of course it will run faster. You have configured your controller to guarantee data loss in event of a drive failure, an ECC error, or unrecoverable error.  (Not just power loss)."

Why/How have I guaranteed data loss in the event of drive failure, that's the whole point of having the RAID5 config isn't it?  The striping with parity.

"You probably also set your RAID5 with an I/O stripe size that is much greater then the native I/O size that NTFS is doing. "

I have a 64kb stripe size, should this be changed?

"RocketRAID default cache policy is write-back, no wonder it's faster than the other card assuming you haven't changed it to write through."

Can I change this?  What difference will it make?
0
 
LVL 56

Expert Comment

by:andyalder
ID: 37823853
You have two sources of data loss / corruption without even having a hard failure in your current setup, at least not having a failure today...

RAID 5 writes are meant to be atomic, that is the data and parity are meant to be written at exactly the same time. Unless you synchronise the disk spindles (which Compaq did with custom disks on one particular controller/server) that is not possible. So you might update the data before the parity, then get a power failure and restart but you haven't updated the parity. All is well and you can even verify all the data is still good but then a month later a disk fails, you replace it but the data on that disk is rebuilt from incorrect parity. You've fallen into the RAID 5 write hole. None of the data on a RAID 5 volume can be trusted after a power failure, not even stuff that was written years ago. Battery backed cache gets around that problem by writing the missing parity when the power comes back on.

The other situation is separate from the lack of atomicity and is the write-back setting, you probably have the controller set to tell the application that the data's been comitted to disk when really it's only in cache. If it was an ATM you would have given the customer their cache before you updated their bank balance. You can change that with the write-through/write back setting in BIOS, but neither setting covers the write hole.
0
 

Author Comment

by:dbestcomputers
ID: 37829061
Thank you Andy.  Makes sense.  I'll look into making this more "safe".  But I hate to sacrifice the speed.  I would've thought with all the cache on this RAID card it would've worked better...  

Again, is there a speed difference by changing the stripe size?
0
 
LVL 56

Accepted Solution

by:
andyalder earned 1200 total points
ID: 37829520
All that cache but no battery means it is probably just read cache, but you list two servers, one fast because it's got the stops pulled out at a risk of data corruption and the other slow and safe.

As far as the stripe size is concerned your mileage may vary, there isn't a magic figure; if there was the controller manufacture wouldn't bother to give you the option with playing with it.
0

Featured Post

The Ideal Solution for Multi-Display Applications

Check out ATEN’s VS1912 12-Port DP Video Wall Media Player at InfoComm 2017. Kerri describes how easy it is to design creative video walls in asymmetric layouts and schedule detailed playlists ahead of time with its advanced scheduling feature.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Data center, now-a-days, is referred as the home of all the advanced technologies. In-fact, most of the businesses are now establishing their entire organizational structure around the IT capabilities.
Restoring deleted objects in Active Directory has been a standard feature in Active Directory for many years, yet some admins may not know what is available.
This tutorial will walk an individual through locating and launching the BEUtility application and how to execute it on the appropriate database. Log onto the server running the Backup Exec database. In a larger environment, this would generally be …
This tutorial will give a short introduction and overview of Backup Exec 2012 and how to navigate and perform basic functions. Click on the Backup Exec button in the upper left corner. From here, are global settings for the application such as conne…

718 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question