Go Premium for a chance to win a PS4. Enter to Win

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 799
  • Last Modified:

U320 RAID 0 performance as array size grows

I have a dual channel LSI Logic PCI-X 64bit/133mhz U320 scsi adapter.  I'm running RAID 0.  As I added drives to the array I noticed that the performance increase leveled off quickly.  I'm using fujitsu U320 10k drives.

Below is a quick summary of sequential reads in MB/sec of the array:

1 - 67
2 - 120
3 - 170
4 - 190
6 - 225
8 - 245

Array sizes 1, 2 and 3 were on 1 channel.  Array sizes 4, 6 and 8 were spread accross 2 channels.

My main question is this: is this typical?  

I'm new to raid so I wsn't sure what to expect.  I dont know if I have a problem or if this is simply the real world limits of U320.  I knew I wouldnt get 320 MB/sec per channel but I was surprised to get only 245 MB/sec off dual 320 MB channels.

If this is typical, then what is the bottleneck here?  Is the scsi bus getting saturated?  Or is it the SCSI controller itself reaching its limit?  

Mark
0
markhorton
Asked:
markhorton
  • 8
  • 3
  • 2
1 Solution
 
LucFCommented:
This is normal, you're getting close to the 320Mb limit.

1) this is the speed one drive can transfer (not what the controller can transfer)
2) see the little overhead 67x2=134
3) same here
4)...
6)...
8)... now you're close to the maximum transferrate (note that your motherboard also gets to be a limited factor now as it's getting slow compared to your transferrate, they use big drivecaches to overcome this)

LucF
0
 
chicagoanCommented:
There's a lot more to consider beyond the disk channel:
If the two disk channels are on the same PCI bus they must contend with each other and the other devices on that bus. You're at about saturation point for 1 PCI-X bus with your 8 disks.
If this is software RAID, the CPU must construct the FAT and the more disks you have the more intensive this task is.
You also have to get the data into or out of something, so memory and processor speed and the other tasks going on play a part in the equation as well.
Remember your large stripe set is much more likely to fail than a single disk as you have to aggregate the failure rate of the individual drives along with their bandwidth :)
0
 
LucFCommented:
I agree with chicagoan, a striped set is likely to fail, consider using Raid5 for some redundancy or if you can affort it Raid 10(0+1, a mirrored striped set)

LucF
0
Visualize your virtual and backup environments

Create well-organized and polished visualizations of your virtual and backup environments when planning VMware vSphere, Microsoft Hyper-V or Veeam deployments. It helps you to gain better visibility and valuable business insights.

 
markhortonAuthor Commented:
Thanks both LucF and chicagoan.  You've comfirmed my suspicions for the most part but I'm still wondering where the bottleneck is.  I'm giving you both credit.  I'm also posting more comments in case you have any additonal ideas on this.

The system I have is a dual opteron with a hypertransport link from the PCI-X bus to the CPU.  The hypertransport bus is capable of 6.4 GB of throughput (theoretically).  The PCI-X bus is 64-bit 133mhz which translates to a theoretical bandwidth of 1GB per sec.  There are 2 slots on this PCI bus with the controller card in 1 slot.  The other slot is empty.  Its a dual channel card.  Each channel is U320 so this has a theoretical bandwidth of 640 MB/sec on the card.

These opterons have a large amount of memory bandwith so I dont think thats the bottleneck.  The PCI-X bus should have enough bandwidth.  That leaves the scsi controller itself or the scsi bus.  

I kinda felt that 245 MB/sec was not saturating a 640MB bus.  So that leaves the scsi controller.  I guess 1 way to test this would be to get another controller and spread the array accross 2 controllers.

Mark
0
 
LucFCommented:
I guess 1 way to test this would be to get another controller and spread the array accross 2 controllers.
Nope, won't work, the controllers won't communicate with each other so you can't make it a striped set (unless you use software raid, but that will explain the low speed even more, everythink you'll read/write has to go trough the processor and the internal memmory)

LucF
0
 
LucFCommented:
And for the rest:
don't you think that 3 times "theoretically" in your post doesn't answer your question.... nothing in a computer system can take advantage of "theoretical" speeds, it has a lot of components between your harddrive and the output, lots of times "almost" theoretical speed" still will give you a performance drop.

LucF
0
 
LucFCommented:
doesn't <=> does (sorry, typo)
0
 
chicagoanCommented:
So we have a couple of variables solved. Your disks are 320, properly terminated and cabled and negotiating 320 on the SCSI bus. Your adaptor is capable of 320*2. You PCI-X bus maximum throughput is 64*133/8 million bytes/second or about
1064 MBytes/sec. Your chipset, memory, processor, etc. is apparently not the culprit, so what do we have left?
1st off, the bus isn't 100% efficient, so all you can do is approach some barrier.
In the few adapter comparisons I've seen there hasn't beeen much difference in performance with one or two disks, when one goes beyond that the drivers can make a big difference. While you have to expect some overhead and will see diminishing returns associated with adding disks, one wonders what throttling mechanisms the vendors use to avoid the disk subsysttem from eating all your CPU clicks to feed the bus. Then you have your software raid libraries and the same issues. I think the biggest difference in big-name adapters in going to be implementation on the driver side.
Off-loading the array management to RAID card means the vendor can punch that little I960 (or whatever processor is onboard) up into the redline and the customer isn't going to bitch.
So you can see if LSI has OEM drivers and compare them, you can change your performance settings to background and see how that affects things, you can reformat and up the size of your disk blocks if your applications aren't generating billions of tiny files, and / or you can look for a RAID card (or a fibre channel card and offload the whole array processing mess to a dedicated RAID controller.)

0
 
markhortonAuthor Commented:
BTW I just remembered that I was going to give both of you points, but when I accepted the answer it did not let me split the points.  I could have sworn that was possible in the past.  

LucF, I would ike to give you points (300) if there is a way to do that.  Let me know and I'll do it.

Mark
0
 
LucFCommented:
There was a split button just below the question just before you accepted chicagoans comment as an answer. But that possibility wont work now.....

You can create a new question, something like:

In the header => Points for LucF
In the border => For your help at http:Q_20783767.html

and give it the amount of points you want to give to me.

LucF
0
 
LucFCommented:
>I could have sworn that was possible in the past.  

It's still possible => http://www.experts-exchange.com/help/closing.jsp#3

But it won't work after you've accepted a comment as an answer. That's why I posted the workaround above.

LucF
0
 
markhortonAuthor Commented:
ok, I must have missed the split since they redesigned their website.  

I created a new question to give you points.


0
 
LucFCommented:
Just posted there.
0

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

  • 8
  • 3
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now