Solved

U320 RAID 0 performance as array size grows

Posted on 2003-10-30
13
745 Views
Last Modified: 2010-04-03
I have a dual channel LSI Logic PCI-X 64bit/133mhz U320 scsi adapter.  I'm running RAID 0.  As I added drives to the array I noticed that the performance increase leveled off quickly.  I'm using fujitsu U320 10k drives.

Below is a quick summary of sequential reads in MB/sec of the array:

1 - 67
2 - 120
3 - 170
4 - 190
6 - 225
8 - 245

Array sizes 1, 2 and 3 were on 1 channel.  Array sizes 4, 6 and 8 were spread accross 2 channels.

My main question is this: is this typical?  

I'm new to raid so I wsn't sure what to expect.  I dont know if I have a problem or if this is simply the real world limits of U320.  I knew I wouldnt get 320 MB/sec per channel but I was surprised to get only 245 MB/sec off dual 320 MB channels.

If this is typical, then what is the bottleneck here?  Is the scsi bus getting saturated?  Or is it the SCSI controller itself reaching its limit?  

Mark
0
Comment
Question by:markhorton
  • 8
  • 3
  • 2
13 Comments
 
LVL 32

Expert Comment

by:Luc Franken
Comment Utility
This is normal, you're getting close to the 320Mb limit.

1) this is the speed one drive can transfer (not what the controller can transfer)
2) see the little overhead 67x2=134
3) same here
4)...
6)...
8)... now you're close to the maximum transferrate (note that your motherboard also gets to be a limited factor now as it's getting slow compared to your transferrate, they use big drivecaches to overcome this)

LucF
0
 
LVL 18

Expert Comment

by:chicagoan
Comment Utility
There's a lot more to consider beyond the disk channel:
If the two disk channels are on the same PCI bus they must contend with each other and the other devices on that bus. You're at about saturation point for 1 PCI-X bus with your 8 disks.
If this is software RAID, the CPU must construct the FAT and the more disks you have the more intensive this task is.
You also have to get the data into or out of something, so memory and processor speed and the other tasks going on play a part in the equation as well.
Remember your large stripe set is much more likely to fail than a single disk as you have to aggregate the failure rate of the individual drives along with their bandwidth :)
0
 
LVL 32

Expert Comment

by:Luc Franken
Comment Utility
I agree with chicagoan, a striped set is likely to fail, consider using Raid5 for some redundancy or if you can affort it Raid 10(0+1, a mirrored striped set)

LucF
0
 

Author Comment

by:markhorton
Comment Utility
Thanks both LucF and chicagoan.  You've comfirmed my suspicions for the most part but I'm still wondering where the bottleneck is.  I'm giving you both credit.  I'm also posting more comments in case you have any additonal ideas on this.

The system I have is a dual opteron with a hypertransport link from the PCI-X bus to the CPU.  The hypertransport bus is capable of 6.4 GB of throughput (theoretically).  The PCI-X bus is 64-bit 133mhz which translates to a theoretical bandwidth of 1GB per sec.  There are 2 slots on this PCI bus with the controller card in 1 slot.  The other slot is empty.  Its a dual channel card.  Each channel is U320 so this has a theoretical bandwidth of 640 MB/sec on the card.

These opterons have a large amount of memory bandwith so I dont think thats the bottleneck.  The PCI-X bus should have enough bandwidth.  That leaves the scsi controller itself or the scsi bus.  

I kinda felt that 245 MB/sec was not saturating a 640MB bus.  So that leaves the scsi controller.  I guess 1 way to test this would be to get another controller and spread the array accross 2 controllers.

Mark
0
 
LVL 32

Expert Comment

by:Luc Franken
Comment Utility
I guess 1 way to test this would be to get another controller and spread the array accross 2 controllers.
Nope, won't work, the controllers won't communicate with each other so you can't make it a striped set (unless you use software raid, but that will explain the low speed even more, everythink you'll read/write has to go trough the processor and the internal memmory)

LucF
0
 
LVL 32

Expert Comment

by:Luc Franken
Comment Utility
And for the rest:
don't you think that 3 times "theoretically" in your post doesn't answer your question.... nothing in a computer system can take advantage of "theoretical" speeds, it has a lot of components between your harddrive and the output, lots of times "almost" theoretical speed" still will give you a performance drop.

LucF
0
Backup Your Microsoft Windows Server®

Backup all your Microsoft Windows Server – on-premises, in remote locations, in private and hybrid clouds. Your entire Windows Server will be backed up in one easy step with patented, block-level disk imaging. We achieve RTOs (recovery time objectives) as low as 15 seconds.

 
LVL 32

Expert Comment

by:Luc Franken
Comment Utility
doesn't <=> does (sorry, typo)
0
 
LVL 18

Accepted Solution

by:
chicagoan earned 300 total points
Comment Utility
So we have a couple of variables solved. Your disks are 320, properly terminated and cabled and negotiating 320 on the SCSI bus. Your adaptor is capable of 320*2. You PCI-X bus maximum throughput is 64*133/8 million bytes/second or about
1064 MBytes/sec. Your chipset, memory, processor, etc. is apparently not the culprit, so what do we have left?
1st off, the bus isn't 100% efficient, so all you can do is approach some barrier.
In the few adapter comparisons I've seen there hasn't beeen much difference in performance with one or two disks, when one goes beyond that the drivers can make a big difference. While you have to expect some overhead and will see diminishing returns associated with adding disks, one wonders what throttling mechanisms the vendors use to avoid the disk subsysttem from eating all your CPU clicks to feed the bus. Then you have your software raid libraries and the same issues. I think the biggest difference in big-name adapters in going to be implementation on the driver side.
Off-loading the array management to RAID card means the vendor can punch that little I960 (or whatever processor is onboard) up into the redline and the customer isn't going to bitch.
So you can see if LSI has OEM drivers and compare them, you can change your performance settings to background and see how that affects things, you can reformat and up the size of your disk blocks if your applications aren't generating billions of tiny files, and / or you can look for a RAID card (or a fibre channel card and offload the whole array processing mess to a dedicated RAID controller.)

0
 

Author Comment

by:markhorton
Comment Utility
BTW I just remembered that I was going to give both of you points, but when I accepted the answer it did not let me split the points.  I could have sworn that was possible in the past.  

LucF, I would ike to give you points (300) if there is a way to do that.  Let me know and I'll do it.

Mark
0
 
LVL 32

Expert Comment

by:Luc Franken
Comment Utility
There was a split button just below the question just before you accepted chicagoans comment as an answer. But that possibility wont work now.....

You can create a new question, something like:

In the header => Points for LucF
In the border => For your help at http:Q_20783767.html

and give it the amount of points you want to give to me.

LucF
0
 
LVL 32

Expert Comment

by:Luc Franken
Comment Utility
>I could have sworn that was possible in the past.  

It's still possible => http://www.experts-exchange.com/help/closing.jsp#3

But it won't work after you've accepted a comment as an answer. That's why I posted the workaround above.

LucF
0
 

Author Comment

by:markhorton
Comment Utility
ok, I must have missed the split since they redesigned their website.  

I created a new question to give you points.


0
 
LVL 32

Expert Comment

by:Luc Franken
Comment Utility
Just posted there.
0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

Suggested Solutions

A bootable USB key can be very handy now-a-days. My favorite USB key consists of our Windows 7 image, network card drivers (to connect up to a Ghost server), the latest BIOS updates for all of our PCs and CopyWipe (to erase a retired PC) Creating…
How to update Firmware and Bios in Dell Equalogic PS6000 Arrays and Hard Disks firmware update.
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now