[Last Call] Learn how to a build a cloud-first strategyRegister Now


Nested RAID 50 Linux Raid + hardware raid question

Posted on 2010-11-21
Medium Priority
Last Modified: 2012-05-10
Hi all...

I need help to figure out something...

Before banging my head on the wall with a problem I want to know if anybody would throw in a bit of advice on this one...

I'm killing an old win 2003 backup server. reinstalling in linux of course :P ... here is the problem I have...

I have 2 RAID controllers in there with 3X1TB in RAID 5 on each...
I did put those 2 raid 5 in stripe using software RAID...

I know that this easy and possible with Linux here is the real question...

I will be receiving 4 extra drives in a couple of weeks to add to this server... 2 drives for each Raid 5 pool... extending those with the hp cards will be easy as hell... I'm just not sure of how to do the rest in linux... BTW the server is not converted yet...

My plan was: 2X raid5 in nested Stripe with linux raid (Raid 50) -> then LVM over that -> then ext3 or xfs on top...

Thanks in advance for the comments!!!!

Question by:ismog_tech
  • 4
  • 3
  • 2
  • +1
LVL 47

Expert Comment

ID: 34184298
What, exactly, is the question?


Author Comment

ID: 34184308
ya sorry for that I just re-read myself that's not that clear...

basically once my setup is made with the original hardware... what would be the steps I take to add the additionnal 4 drives of storage let's say past the expansion on the 2 raid cards (thios I know how to do)... How do I expand the raid volumes in linux then expand the LVM then expand the xfs or ext3...

better??? :P
LVL 47

Accepted Solution

David earned 668 total points
ID: 34184399
Well, there are a lot of online articles,  this points to a few threads that are as good as any, but let me add my own advice before you even start ..

1.  Upgrade kernel.  LOTS of bugs always being found with md, so might as well make sure you are current.
2.  Do full backup first.
3.  Do not expand RAID5 by 2 disks, I strongly recommend going to a RAID6 config, which means effectively adding only one usable disk for data.  It could very well take 3-4 days, or longer to rebuild those arrays, you don't want to be exposed that long.  So best way to do that is to boot to a non-RAID disk, or USB flash drive. Use the 4 new disks to create a 4-drive RAID6, copy from the existing RAID5.

It takes maybe 1/3rd as long to copy from the old array to a new array as it takes to expand an existing array.  You could do this overnight if you build the new array, then copied data to it.  Then when the disks are freed up from the first source array, you would probably have enough free space on that new array to save a compressed copy of the data from the other array, then repeat the process.

But here are some links for doing it the way you wanted to originally ..


Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.


Author Comment

ID: 34185000
ok I think you didn't figure out what I meant...

first the setup insn't important it's only transitory storage for D2D2T...

2nd ...

2 raids cards with 2 raid 5 so how do you expect to build a raid 6 with the 4 news drives since those will be plugged on 2 different raid cards???

so RAID5 array #1 =ra1
RAID5 array #2 =ra2

final storage at startup = final

final = stripe (ra1+ ra2) initial setup

what I need to know is....

final = stripe ( (ra1+ expansion) + (ra2 + expansion) )

how will I expand the 2 raid partition composing my stripe so they reflect the RAID array expansions?

LVL 47

Expert Comment

ID: 34185470
So you are using md driver to stripe the 2 hardware-based RAID devices?   (Well, actually you could still use md to do this, as the LUN presented by each controller could still be part of a RAID6, but obviously that is not what you want).

First, you have to expand ra1 and ra2 to use the new drives per each raid controller's documentation.   Do all of this when system is booted to another device so ra1, ra2 and the md0 (assuming md0 = f(ra1,ra2).

Then you have to tell mdadm to discover the resized devices, and expand the volume ...  look at 2.2 here

Having said that, unless you have premium controllers with cache battery backup and channel load balancing, then you are most likely creating a big bottleneck with your design.  Every write is going to require at least one I/O on every disk drive.

If you care about performance, you should consider just using md RAID.
LVL 81

Assisted Solution

arnold earned 664 total points
ID: 34186041
Adding to the above discussion depends on your initial setup.

I.e. do you use MD to completely RAID i.e. first RAID5 is /dev/sda and the second /dev/sdb
are you using software raid as /dev/md0 which is made up of /dev/sda /dev/sdb
Or are you first breaking each array and mirroring the individual partitions?
/dev/md0 /dev/sda1 /dev/sdb1 etc.

The per-partition software raid provides additional flexibility dealing with shifting partitions if/when issues arise with the underlying RAID controller.

As dlethe pointed out.
You would have to go through the steps.
expand each of the HW RAID groups
adjust the md create software raid
extend the physical volume to reflect the added space
adjust the logical volume/volumes to take advantage of the new space or create additional logical partitions.


Author Comment

ID: 34187294
but is that possible???

since the expansion of the 1st array will be right in the middle of the stripe....

another thing I don't understand how does my nested Raid 50 setup is underperforming compared to a raid6 software??? my card are hp P600 with batteries and 256 cache each...
LVL 56

Expert Comment

ID: 34187397
Can't see how the shrink and grow link helps since it's talking about RAID 1 and just breaks the mirror, resizes and then re-mirrors. Not valid for this setup as you want to extend software RAID 0.

From what I read mdadm --grow isn't supported for RAID 0, but since it's just D2D2T you might as well blow it away and start again when you get the new disks.
LVL 56

Assisted Solution

andyalder earned 668 total points
ID: 34187417
On second thoughts http://man-wiki.net/index.php/8:raidreconf says you can expand RAID 0.

Author Closing Comment

ID: 34286179
Point for all ... good examples and links but finally took the terminator approach... kill and rebuild and rekill and rebuild... like I said it's temporary storage... and I like to have my weekends off :P but overall that was a good brain twister. thanks to all.

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The question appears often enough, how do I transfer my data from my old server to the new server while preserving file shares, share permissions, and NTFS permisions.  Here are my tips for handling such a transfer.
How much do you know about the future of data centers? If you're like 50% of organizations, then it's probably not enough. Read on to get up to speed on this emerging field.
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
This video teaches viewers how to encrypt an external drive that requires a password to read and edit the drive. All tasks are done in Disk Utility. Plug in the external drive you wish to encrypt: Make sure all previous data on the drive has been …
Suggested Courses
Course of the Month17 days, 14 hours left to enroll

830 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question