Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

U320 RAID 0 performance as array size grows

Posted on 2003-10-30
13
Medium Priority
?
788 Views
Last Modified: 2010-04-03
I have a dual channel LSI Logic PCI-X 64bit/133mhz U320 scsi adapter.  I'm running RAID 0.  As I added drives to the array I noticed that the performance increase leveled off quickly.  I'm using fujitsu U320 10k drives.

Below is a quick summary of sequential reads in MB/sec of the array:

1 - 67
2 - 120
3 - 170
4 - 190
6 - 225
8 - 245

Array sizes 1, 2 and 3 were on 1 channel.  Array sizes 4, 6 and 8 were spread accross 2 channels.

My main question is this: is this typical?  

I'm new to raid so I wsn't sure what to expect.  I dont know if I have a problem or if this is simply the real world limits of U320.  I knew I wouldnt get 320 MB/sec per channel but I was surprised to get only 245 MB/sec off dual 320 MB channels.

If this is typical, then what is the bottleneck here?  Is the scsi bus getting saturated?  Or is it the SCSI controller itself reaching its limit?  

Mark
0
Comment
Question by:markhorton
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 8
  • 3
  • 2
13 Comments
 
LVL 32

Expert Comment

by:LucF
ID: 9656308
This is normal, you're getting close to the 320Mb limit.

1) this is the speed one drive can transfer (not what the controller can transfer)
2) see the little overhead 67x2=134
3) same here
4)...
6)...
8)... now you're close to the maximum transferrate (note that your motherboard also gets to be a limited factor now as it's getting slow compared to your transferrate, they use big drivecaches to overcome this)

LucF
0
 
LVL 18

Expert Comment

by:chicagoan
ID: 9657743
There's a lot more to consider beyond the disk channel:
If the two disk channels are on the same PCI bus they must contend with each other and the other devices on that bus. You're at about saturation point for 1 PCI-X bus with your 8 disks.
If this is software RAID, the CPU must construct the FAT and the more disks you have the more intensive this task is.
You also have to get the data into or out of something, so memory and processor speed and the other tasks going on play a part in the equation as well.
Remember your large stripe set is much more likely to fail than a single disk as you have to aggregate the failure rate of the individual drives along with their bandwidth :)
0
 
LVL 32

Expert Comment

by:LucF
ID: 9657779
I agree with chicagoan, a striped set is likely to fail, consider using Raid5 for some redundancy or if you can affort it Raid 10(0+1, a mirrored striped set)

LucF
0
Visualize your virtual and backup environments

Create well-organized and polished visualizations of your virtual and backup environments when planning VMware vSphere, Microsoft Hyper-V or Veeam deployments. It helps you to gain better visibility and valuable business insights.

 

Author Comment

by:markhorton
ID: 9658675
Thanks both LucF and chicagoan.  You've comfirmed my suspicions for the most part but I'm still wondering where the bottleneck is.  I'm giving you both credit.  I'm also posting more comments in case you have any additonal ideas on this.

The system I have is a dual opteron with a hypertransport link from the PCI-X bus to the CPU.  The hypertransport bus is capable of 6.4 GB of throughput (theoretically).  The PCI-X bus is 64-bit 133mhz which translates to a theoretical bandwidth of 1GB per sec.  There are 2 slots on this PCI bus with the controller card in 1 slot.  The other slot is empty.  Its a dual channel card.  Each channel is U320 so this has a theoretical bandwidth of 640 MB/sec on the card.

These opterons have a large amount of memory bandwith so I dont think thats the bottleneck.  The PCI-X bus should have enough bandwidth.  That leaves the scsi controller itself or the scsi bus.  

I kinda felt that 245 MB/sec was not saturating a 640MB bus.  So that leaves the scsi controller.  I guess 1 way to test this would be to get another controller and spread the array accross 2 controllers.

Mark
0
 
LVL 32

Expert Comment

by:LucF
ID: 9658800
I guess 1 way to test this would be to get another controller and spread the array accross 2 controllers.
Nope, won't work, the controllers won't communicate with each other so you can't make it a striped set (unless you use software raid, but that will explain the low speed even more, everythink you'll read/write has to go trough the processor and the internal memmory)

LucF
0
 
LVL 32

Expert Comment

by:LucF
ID: 9658824
And for the rest:
don't you think that 3 times "theoretically" in your post doesn't answer your question.... nothing in a computer system can take advantage of "theoretical" speeds, it has a lot of components between your harddrive and the output, lots of times "almost" theoretical speed" still will give you a performance drop.

LucF
0
 
LVL 32

Expert Comment

by:LucF
ID: 9658831
doesn't <=> does (sorry, typo)
0
 
LVL 18

Accepted Solution

by:
chicagoan earned 1200 total points
ID: 9660102
So we have a couple of variables solved. Your disks are 320, properly terminated and cabled and negotiating 320 on the SCSI bus. Your adaptor is capable of 320*2. You PCI-X bus maximum throughput is 64*133/8 million bytes/second or about
1064 MBytes/sec. Your chipset, memory, processor, etc. is apparently not the culprit, so what do we have left?
1st off, the bus isn't 100% efficient, so all you can do is approach some barrier.
In the few adapter comparisons I've seen there hasn't beeen much difference in performance with one or two disks, when one goes beyond that the drivers can make a big difference. While you have to expect some overhead and will see diminishing returns associated with adding disks, one wonders what throttling mechanisms the vendors use to avoid the disk subsysttem from eating all your CPU clicks to feed the bus. Then you have your software raid libraries and the same issues. I think the biggest difference in big-name adapters in going to be implementation on the driver side.
Off-loading the array management to RAID card means the vendor can punch that little I960 (or whatever processor is onboard) up into the redline and the customer isn't going to bitch.
So you can see if LSI has OEM drivers and compare them, you can change your performance settings to background and see how that affects things, you can reformat and up the size of your disk blocks if your applications aren't generating billions of tiny files, and / or you can look for a RAID card (or a fibre channel card and offload the whole array processing mess to a dedicated RAID controller.)

0
 

Author Comment

by:markhorton
ID: 9682036
BTW I just remembered that I was going to give both of you points, but when I accepted the answer it did not let me split the points.  I could have sworn that was possible in the past.  

LucF, I would ike to give you points (300) if there is a way to do that.  Let me know and I'll do it.

Mark
0
 
LVL 32

Expert Comment

by:LucF
ID: 9682091
There was a split button just below the question just before you accepted chicagoans comment as an answer. But that possibility wont work now.....

You can create a new question, something like:

In the header => Points for LucF
In the border => For your help at http:Q_20783767.html

and give it the amount of points you want to give to me.

LucF
0
 
LVL 32

Expert Comment

by:LucF
ID: 9682118
>I could have sworn that was possible in the past.  

It's still possible => http://www.experts-exchange.com/help/closing.jsp#3

But it won't work after you've accepted a comment as an answer. That's why I posted the workaround above.

LucF
0
 

Author Comment

by:markhorton
ID: 9682184
ok, I must have missed the split since they redesigned their website.  

I created a new question to give you points.


0
 
LVL 32

Expert Comment

by:LucF
ID: 9682225
Just posted there.
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

AWS Glacier is Amazons cheapest storage option and is their answer to a ‘Cold’ storage service.  Customers primarily use this service for archival purposes and storage of infrastructure backups.  Its unlimited storage potential and low storage cost …
The article will include the best Data Recovery Tools along with their Features, Capabilities, and their Download Links. Hope you’ll enjoy it and will choose the one as required by you.
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…
Suggested Courses

662 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question