U320 RAID 0 performance as array size grows

I have a dual channel LSI Logic PCI-X 64bit/133mhz U320 scsi adapter.  I'm running RAID 0.  As I added drives to the array I noticed that the performance increase leveled off quickly.  I'm using fujitsu U320 10k drives.

Below is a quick summary of sequential reads in MB/sec of the array:

1 - 67
2 - 120
3 - 170
4 - 190
6 - 225
8 - 245

Array sizes 1, 2 and 3 were on 1 channel.  Array sizes 4, 6 and 8 were spread accross 2 channels.

My main question is this: is this typical?  

I'm new to raid so I wsn't sure what to expect.  I dont know if I have a problem or if this is simply the real world limits of U320.  I knew I wouldnt get 320 MB/sec per channel but I was surprised to get only 245 MB/sec off dual 320 MB channels.

If this is typical, then what is the bottleneck here?  Is the scsi bus getting saturated?  Or is it the SCSI controller itself reaching its limit?  

Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Luc FrankenEMEA Server EngineerCommented:
This is normal, you're getting close to the 320Mb limit.

1) this is the speed one drive can transfer (not what the controller can transfer)
2) see the little overhead 67x2=134
3) same here
8)... now you're close to the maximum transferrate (note that your motherboard also gets to be a limited factor now as it's getting slow compared to your transferrate, they use big drivecaches to overcome this)

There's a lot more to consider beyond the disk channel:
If the two disk channels are on the same PCI bus they must contend with each other and the other devices on that bus. You're at about saturation point for 1 PCI-X bus with your 8 disks.
If this is software RAID, the CPU must construct the FAT and the more disks you have the more intensive this task is.
You also have to get the data into or out of something, so memory and processor speed and the other tasks going on play a part in the equation as well.
Remember your large stripe set is much more likely to fail than a single disk as you have to aggregate the failure rate of the individual drives along with their bandwidth :)
Luc FrankenEMEA Server EngineerCommented:
I agree with chicagoan, a striped set is likely to fail, consider using Raid5 for some redundancy or if you can affort it Raid 10(0+1, a mirrored striped set)

Protecting & Securing Your Critical Data

Considering 93 percent of companies file for bankruptcy within 12 months of a disaster that blocked access to their data for 10 days or more, planning for the worst is just smart business. Learn how Acronis Backup integrates security at every stage

markhortonAuthor Commented:
Thanks both LucF and chicagoan.  You've comfirmed my suspicions for the most part but I'm still wondering where the bottleneck is.  I'm giving you both credit.  I'm also posting more comments in case you have any additonal ideas on this.

The system I have is a dual opteron with a hypertransport link from the PCI-X bus to the CPU.  The hypertransport bus is capable of 6.4 GB of throughput (theoretically).  The PCI-X bus is 64-bit 133mhz which translates to a theoretical bandwidth of 1GB per sec.  There are 2 slots on this PCI bus with the controller card in 1 slot.  The other slot is empty.  Its a dual channel card.  Each channel is U320 so this has a theoretical bandwidth of 640 MB/sec on the card.

These opterons have a large amount of memory bandwith so I dont think thats the bottleneck.  The PCI-X bus should have enough bandwidth.  That leaves the scsi controller itself or the scsi bus.  

I kinda felt that 245 MB/sec was not saturating a 640MB bus.  So that leaves the scsi controller.  I guess 1 way to test this would be to get another controller and spread the array accross 2 controllers.

Luc FrankenEMEA Server EngineerCommented:
I guess 1 way to test this would be to get another controller and spread the array accross 2 controllers.
Nope, won't work, the controllers won't communicate with each other so you can't make it a striped set (unless you use software raid, but that will explain the low speed even more, everythink you'll read/write has to go trough the processor and the internal memmory)

Luc FrankenEMEA Server EngineerCommented:
And for the rest:
don't you think that 3 times "theoretically" in your post doesn't answer your question.... nothing in a computer system can take advantage of "theoretical" speeds, it has a lot of components between your harddrive and the output, lots of times "almost" theoretical speed" still will give you a performance drop.

Luc FrankenEMEA Server EngineerCommented:
doesn't <=> does (sorry, typo)
So we have a couple of variables solved. Your disks are 320, properly terminated and cabled and negotiating 320 on the SCSI bus. Your adaptor is capable of 320*2. You PCI-X bus maximum throughput is 64*133/8 million bytes/second or about
1064 MBytes/sec. Your chipset, memory, processor, etc. is apparently not the culprit, so what do we have left?
1st off, the bus isn't 100% efficient, so all you can do is approach some barrier.
In the few adapter comparisons I've seen there hasn't beeen much difference in performance with one or two disks, when one goes beyond that the drivers can make a big difference. While you have to expect some overhead and will see diminishing returns associated with adding disks, one wonders what throttling mechanisms the vendors use to avoid the disk subsysttem from eating all your CPU clicks to feed the bus. Then you have your software raid libraries and the same issues. I think the biggest difference in big-name adapters in going to be implementation on the driver side.
Off-loading the array management to RAID card means the vendor can punch that little I960 (or whatever processor is onboard) up into the redline and the customer isn't going to bitch.
So you can see if LSI has OEM drivers and compare them, you can change your performance settings to background and see how that affects things, you can reformat and up the size of your disk blocks if your applications aren't generating billions of tiny files, and / or you can look for a RAID card (or a fibre channel card and offload the whole array processing mess to a dedicated RAID controller.)

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
markhortonAuthor Commented:
BTW I just remembered that I was going to give both of you points, but when I accepted the answer it did not let me split the points.  I could have sworn that was possible in the past.  

LucF, I would ike to give you points (300) if there is a way to do that.  Let me know and I'll do it.

Luc FrankenEMEA Server EngineerCommented:
There was a split button just below the question just before you accepted chicagoans comment as an answer. But that possibility wont work now.....

You can create a new question, something like:

In the header => Points for LucF
In the border => For your help at http:Q_20783767.html

and give it the amount of points you want to give to me.

Luc FrankenEMEA Server EngineerCommented:
>I could have sworn that was possible in the past.  

It's still possible => http://www.experts-exchange.com/help/closing.jsp#3

But it won't work after you've accepted a comment as an answer. That's why I posted the workaround above.

markhortonAuthor Commented:
ok, I must have missed the split since they redesigned their website.  

I created a new question to give you points.

Luc FrankenEMEA Server EngineerCommented:
Just posted there.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.