Solved

How to recover a three disk Ubuntu software RAID5 array

Posted on 2011-03-10
8
1,489 Views
Last Modified: 2012-05-11
Hey there,

I'm building an Ubuntu 10.04 virtual machine server.  I have three 500GB SATA drives which I setup in a RAID5 configuration.  The setup went fairly easily and I installed the OS without a problem.  

Before I start setting everything up, I want to make sure that if I lose one of the drives I know how to recover.  For my dry run, I removed one of the drives, connected it to my laptop with a SATA USB docking station, and deleted the partitions from it.  

I physically reinstalled the drive in my server and tried to boot.  During boot I'm prompted to start the degraded RAID.  I say yes to that and I get the following:

mdadm: CREATE user root not found
mdadm: CREATE group disk not found
raid5: raid level 5 set md0 active with 2 out of 3 devices, algorithm 2
mdadm: /dev/md0 has been started with 2 drives (out of 3)
raid5: failed to run raid set md1
mdadm: failed to RUN-ARRAY /dev/md1: Input.output error
mdadm: Not enough devices to start the array while not clean - consider --force
Started the RAID in degraded mode
Gave up waiting for root device:
*then a list of common problems and then:

ALERT! /dev/disk/by-uuid/bunchanumbersnad letters does not exist.  Dropping to a shell!
Then I get a busybox shell and a prompt of "initramfs".  

So, the question is, how would I go about rebuilding the third drive in the array?  Can I boot to a live CD and do something there?  Is there something I can do at the initramfs prompt?

Glad I'm not trying to figure this out on a failed production box!  Yay for dry runs! :P

Thanks in advance for any help.  
0
Comment
Question by:ttist25
  • 5
  • 2
8 Comments
 
LVL 47

Assisted Solution

by:dlethe
dlethe earned 100 total points
ID: 35101480
Looks like you never set it up properly to begin with.   /dev/md0 looks golden, it sees 2 out of 3 disks.

/dev/md1 (which would be the other partition) should have had exactly the same output.   So you must have done something differently when you set that up, or perhaps you never formatted it.  Look at the config files.  What is different between the two?
0
 
LVL 1

Author Comment

by:ttist25
ID: 35101893
Config file?  I didn't see no stinking config file!

:D

Where would I find them?  Is it accessible via the initramfs thing?  I'm just knowledgeable enough about Linux to be dangerous.  I guess I've spread my wings and left the nest but I'm getting jacked up by some branches on the way down!  

When I set it up I used the instructions here.

Everything went as described in that document except that it states:
If you choose to place the root partition on a RAID array, the installer will then ask if you would like to boot in a degraded state. See the section called “Degraded RAID” for further details.
and I didn't get asked about booting in a degraded state at that point.  

Maybe I should try starting over to make sure I didn't miss something?

Thanks for the help.
0
 
LVL 12

Expert Comment

by:mccracky
ID: 35110083
do you have a copy of your /etc/fstab?  that would help us.

If you did follow the guide that you pointed to, the md0 that came up is just the swap partition.  The md1 is the root partition that doesn't want to boot in a degraded state.  When GRUB comes up, hit "e" to edit and then choose the "kernel" line and "e" to edit it.  Append "bootdegraded=true" (without the quotes) to the kernel line and then boot the box.  Does it come up now?
0
 
LVL 1

Author Comment

by:ttist25
ID: 35115832
Sorry - I started from scratch again before I saw your response.  I followed these instructions to the letter except that I selected RAID5 rather than RAID1 and I formatted three drives with a swap and / partition instead of two.  

Also this time, after I got the OS installed, I opened up a terminal and did "sudo dpkg-reconfigure mdadm" to set bootdegraded to true and also set the notification email.  

I rebooted, shutdown, and disconnected power to drive 1 (sdb).  Then I started the box and tried hitting "e" after POST but I don't ever see GRUB.  I'm not sure I see grub menu's even with all three drives running normally.  Although I know grub is installed - there's no other OS or bootloader.  

Maybe I'm missing something fundamental.  For instance, all of the RAID5 tutorials I've seen reference at least 4 drives.  One is for the OS (and is sometimes mirrored with a 5th drive in RAID1) and the other three are used for the array.  I'm installing the OS on my RAID5 array.  Is this the reason it can't boot degraded?

I've got to have this server be able to run in a degraded state as an entire satellite office (6 users) will be connecting via RDP to the VM's on it.  The office is about 1-1/2 hours from me so I need to have that redundancy so they can continue to run until I can get there and fix it.  

If I reconnect the drive and boot would the fstab still help you?  Just let me know and I'll post it right away.

Thanks SO much for your help.  I'm really getting frustrated trying to get this working and it's nice to have you guys there for support.  


Thanks again.  
0
How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

 
LVL 12

Accepted Solution

by:
mccracky earned 400 total points
ID: 35116356
I think by default on the new Ubuntu installs, the GRUB menu is hidden.  Hold down the <SHIFT> key while it is booting to see the GRUB menu.

To answer your question about the tutorials, Linux before had trouble booting from RAID, so many would put at least the /boot partition on a non-RAID portion of the drive.  Also, many would consider the data as the most important for the redundancy.

After you set the configuration of bootdegraded=true did it come up?
0
 
LVL 1

Author Comment

by:ttist25
ID: 35117184
Nope.  Didn't come up even with bootdegraded set to true.  That said, even without setting bootdegraded to true, I get an option that asks if I want to boot degraded.

Right now it's sitting with sdb disconnected and sitting at an initramfs prompt.  I'll try rebooting and see if I can get the grub menu.  

Is there anything else I can provide for you?

thanks
0
 
LVL 1

Author Comment

by:ttist25
ID: 35117453
I was in an Ubuntu IRC chat and someone there stated that the OS can't be on the array in RAID5.  Do you know if that's true?
0
 
LVL 1

Author Comment

by:ttist25
ID: 35121700
Hey guys,

Thanks for the help but I've given up on RAID5.  For my application I think a RAID1 setup is the way to go.  It provides sufficient fault tolerance for my application and will keep the server running long enough for me to add a new drive.  

RAID5 in theory (better read times, greater storage space, etc) would've been better for my application but, there doesn't seem to be strong consensus in the community in the ability to install the OS on the array and recover and I see more nightmares than solutions in the available resources.  

I used this tutorial to set it up and test it.  Worked well.  

I setup RAID1, disconnected sdb, removed sdb, booted, added sdb and it built and runs fine.  

Keep it simple I guess.  Time to move on.

Thanks again for your efforts.
0

Featured Post

Complete VMware vSphere® ESX(i) & Hyper-V Backup

Capture your entire system, including the host, with patented disk imaging integrated with VMware VADP / Microsoft VSS and RCT. RTOs is as low as 15 seconds with Acronis Active Restore™. You can enjoy unlimited P2V/V2V migrations from any source (even from a different hypervisor)

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
Acronis True Image is Reporting a Bad Sector! 12 87
VMWare 6 crashing 14 54
Linux Scripting 3 49
Anti-virus for Linux Server 15 85
VM backups can be lost due to a number of reasons: accidental backup deletion, backup file corruption, disk failure, lost or stolen hardware, malicious attack, or due to some other undesired and unpredicted event. Thus, having more than one copy of …
Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
This tutorial will walk an individual through the steps necessary to enable the VMware\Hyper-V licensed feature of Backup Exec 2012. In addition, how to add a VMware server and configure a backup job. The first step is to acquire the necessary licen…
This tutorial will walk an individual through configuring a drive on a Windows Server 2008 to perform shadow copies in order to quickly recover deleted files and folders. Click on Start and then select Computer to view the available drives on the se…

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now