?
Solved

Linux RAID with MDADM not staying in sync.

Posted on 2007-07-22
33
Medium Priority
?
1,192 Views
Last Modified: 2013-12-16
SUSE LINUX ENTERPRISE DESKTOP 10.
I had heard good reports about the performance of RAID on Suse, so I decided to RAID my main pc.  
As I was fully  aware that the a mistake could easily hose my system, I took it real slow, but in the end I couldn't get the arrays to stay synced.

The system was originally running for many months on a single disk, so I simply installed an identical 320GB disk and created identical partitions on it.  I synced the array partitions which appeared to complete successfully, but after a reboot,  the array appears broken again.  Greater detail is explained below.

To sync the array partitions I ran:  

srv1:/home/gordonm # mdadm /dev/md2 -a /dev/sda2

which completed successfully as shown by...

srv1:/home/gordonm # cat /proc/mdstat
Personalities : [raid1]
md5 : active raid1 sdb5[0]
      24121472 blocks [2/1] [U_]

md3 : active raid1 sdb3[0]
      276832000 blocks [2/1] [U_]

md1 : active raid1 sdb1[0]
      104320 blocks [2/1] [U_]

md2 : active raid1 sda2[1] sdb2[0]
      10490368 blocks [2/2] [UU]

md6 : active raid1 sdb6[0] sda6[1]
      1020032 blocks [2/2] [UU]

unused devices: <none>


As a double check of the just synced array MD2, the below shows no problem that I can see.
srv1:/home/gordonm # mdadm --detail /dev/md2
/dev/md2:
        Version : 00.90.03
  Creation Time : Sun Jun 24 21:29:45 2007
     Raid Level : raid1
     Array Size : 10490368 (10.00 GiB 10.74 GB)
    Device Size : 10490368 (10.00 GiB 10.74 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 2
    Persistence : Superblock is persistent

    Update Time : Sun Jul 22 23:47:16 2007
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : 9bc64366:61602784:e2438f93:b2779bdc
         Events : 0.558886

    Number   Major   Minor   RaidDevice State
       0       8       18        0      active sync   /dev/sdb2
       1       8        2        1      active sync   /dev/sda2



The problem is when I reboot.  The arrays are broken again as if I havn't added the second disk at all.

Here is my current setup...

Disk /dev/sda: 320.0 GB, 320072933376 bytes
255 heads, 63 sectors/track, 38913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   fd  Linux raid autodetect
/dev/sda2              14        1319    10490445   fd  Linux raid autodetect
/dev/sda3            1320       35783   276832080   fd  Linux raid autodetect
/dev/sda4           35784       38913    25141725    f  W95 Ext'd (LBA)
/dev/sda5           35784       38786    24121566   fd  Linux raid autodetect
/dev/sda6           38787       38913     1020096   fd  Linux raid autodetect

Disk /dev/sdb: 320.0 GB, 320072933376 bytes
255 heads, 63 sectors/track, 38913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          13      104391   fd  Linux raid autodetect
/dev/sdb2              14        1319    10490445   fd  Linux raid autodetect
/dev/sdb3            1320       35783   276832080   fd  Linux raid autodetect
/dev/sdb4           35784       38913    25141725    f  W95 Ext'd (LBA)
/dev/sdb5           35784       38786    24121566   fd  Linux raid autodetect
/dev/sdb6           38787       38913     1020096   fd  Linux raid autodetect

#############
#  MDADM.CONF     #
#############
srv1:/home/gordonm # cat /etc/mdadm.conf
DEVICE /dev/sdb1 /dev/sdb2 /dev/sdb3 /dev/sdb5 /dev/sdb6 /dev/sda1 /dev/sda2 /dev/sda3 /dev/sda5 /dev/sdb6
ARRAY /dev/md1 devices=/dev/sdb1,/dev/sda1
ARRAY /dev/md2 devices=/dev/sdb2,/dev/sda2
ARRAY /dev/md3 devices=/dev/sdb3,/dev/sda3
ARRAY /dev/md5 devices=/dev/sdb5,/dev/sda5
ARRAY /dev/md6 devices=/dev/sdb6,/dev/sda6

Something during reboot is breaking it, or the state of the array is not saving at shutdown.  
Either way - "HELP!" - I am a linux newb, learning fast, but I am truly stumped and am not sure where to start.
Thanks in advance.
0
Comment
Question by:zanotech
  • 14
  • 9
  • 6
  • +1
33 Comments
 
LVL 19

Expert Comment

by:alextoft
ID: 19542985
Did you add the raid module to /etc/sysconfig/kernel and run a mkinitrd so the kernel has the correct module available at boot time?

I've implemented 10+ production software raid1 SuSE systems and all of them are rock solid.
0
 
LVL 19

Expert Comment

by:alextoft
ID: 19542992
Just for example, the entry in my home desktop sysconfig/kernel looks like this:
INITRD_MODULES="via82cxxx processor thermal fan jbd ext3 raid1"

That's literally all you need to do - add the correct module, which is also raid1 in your case, then run mkinitrd. Then the kernel has the module available in the ramdrive before it mounts the root filesystem.
0
 
LVL 19

Expert Comment

by:alextoft
ID: 19542997
Oh, and don't forget do run grub-install on your second hard disc. It's not relevant to this question particularly, but it means you'll still be able to boot if your primary drive fails. Using dd to copy from 1 drive to another is not always sufficient.
0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 

Author Comment

by:zanotech
ID: 19543063
Crikey!  That was a fast reponse.  It is 0020hrs here in Perth (West Australia) and I didn't expect a comment so soon.  Thanks.

I checked what you suggested about the raid module and it looks OK :

srv1:/home/gordonm # cat /etc/sysconfig/kernel | grep INITRD_MODULES
INITRD_MODULES="raid1 amd74xx sata_nv sata_sil reiserfs processor thermal fan edd"
DOMU_INITRD_MODULES="xennet xenblk"
srv1:/home/gordonm #
0
 

Author Comment

by:zanotech
ID: 19543109
Since my last comment I just rebooted and found yet again that MD2 broke!
They all broke after my first reboot, and so I am just working/testing with MD2.

srv1:/home/gordonm # cat /proc/mdstat
Personalities : [raid1]
md5 : active raid1 sdb5[0]
      24121472 blocks [2/1] [U_]

md3 : active raid1 sdb3[0]
      276832000 blocks [2/1] [U_]

md1 : active raid1 sdb1[0]
      104320 blocks [2/1] [U_]

md2 : active raid1 sdb2[0]
      10490368 blocks [2/1] [U_]

md6 : active raid1 sdb6[0] sda6[1]
      1020032 blocks [2/2] [UU]

unused devices: <none>
srv1:/home/gordonm #

Where would the best log be to pursue as the system starts?
Does MDADM have a log file as it initialises
0
 
LVL 43

Expert Comment

by:ravenpl
ID: 19543169
what partition type You marked for /dev/sda2? It should be linux raid autodetect.
fdisk -l /dev/sda

Otherwise please bring here output from
dmesg
command after rebooted (at least the part from raid startup)
0
 
LVL 7

Expert Comment

by:ezaton
ID: 19543204
You need to mark the partition as "Linux raid autodetect" (fd), else it will not be detected during boot time.
Check the raid-howto on how to convert an existing ext2/3/reiserfs/whatever partition to md device.
0
 

Author Comment

by:zanotech
ID: 19543232
###################
PARTITION TYPE FOR /dev/sda
###################
fdisk -l /dev/sda

Disk /dev/sda: 320.0 GB, 320072933376 bytes
255 heads, 63 sectors/track, 38913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   fd  Linux raid autodetect
/dev/sda2              14        1319    10490445   fd  Linux raid autodetect
/dev/sda3            1320       35783   276832080   fd  Linux raid autodetect
/dev/sda4           35784       38913    25141725    f  W95 Ext'd (LBA)
/dev/sda5           35784       38786    24121566   fd  Linux raid autodetect
/dev/sda6           38787       38913     1020096   fd  Linux raid autodetect

######################
DMESG
######################
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
md: raid1 personality registered for level 1
NFORCE3-250: IDE controller at PCI slot 0000:00:08.0
NFORCE3-250: chipset revision 162
NFORCE3-250: not 100% native mode: will probe irqs later
NFORCE3-250: 0000:00:08.0 (rev a2) UDMA133 controller
    ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:pio, hdb:pio
    ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:DMA, hdd:DMA
Probing IDE interface ide0...
Probing IDE interface ide1...
hdc: SONY DVD RW DRU-510A, ATAPI CD/DVD-ROM drive
hdd: _NEC DV-5700A, ATAPI CD/DVD-ROM drive
ide1 at 0x170-0x177,0x376 on irq 15
libata version 1.20 loaded.
sata_sil 0000:02:0c.0: version 0.9
ata1: SATA max UDMA/100 cmd 0xF883A880 ctl 0xF883A88A bmdma 0xF883A800 irq 177
ata2: SATA max UDMA/100 cmd 0xF883A8C0 ctl 0xF883A8CA bmdma 0xF883A808 irq 177
ata3: SATA max UDMA/100 cmd 0xF883AA80 ctl 0xF883AA8A bmdma 0xF883AA00 irq 177
ata4: SATA max UDMA/100 cmd 0xF883AAC0 ctl 0xF883AACA bmdma 0xF883AA08 irq 177
ata1: SATA link up 1.5 Gbps (SStatus 113)
ata1: dev 0 cfg 00:0c5a 49:2f00 82:346b 83:7d01 84:4023 85:3469 86:3c01 87:4023 88:207f 93:0000
ata1: dev 0 ATA-7, max UDMA/133, 625142448 sectors: LBA48
sata_get_dev_handle: SATA dev addr=0xc0000, handle=0x00000000
ata1: dev 0 configured for UDMA/100
sata_get_dev_handle: SATA dev addr=0xc0000, handle=0x00000000
scsi0 : sata_sil
ata2: SATA link up 1.5 Gbps (SStatus 113)
ata2: dev 0 cfg 00:0c5a 49:2f00 82:346b 83:7d01 84:4023 85:3469 86:3c01 87:4023 88:207f 93:0000
ata2: dev 0 ATA-7, max UDMA/133, 625142448 sectors: LBA48
sata_get_dev_handle: SATA dev addr=0xc0000, handle=0x00000000
ata2: dev 0 configured for UDMA/100
sata_get_dev_handle: SATA dev addr=0xc0000, handle=0x00000000
scsi1 : sata_sil
ata3: SATA link down (SStatus 0)
scsi2 : sata_sil
ata4: SATA link down (SStatus 0)
scsi3 : sata_sil
  Vendor: ATA       Model: ST3320620AS       Rev: 3.AA
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sda: 625142448 512-byte hdwr sectors (320073 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: drive cache: write back
SCSI device sda: 625142448 512-byte hdwr sectors (320073 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: drive cache: write back
 sda: sda1 sda2 sda3 sda4 < sda5 sda6 >
sd 0:0:0:0: Attached scsi disk sda
  Vendor: ATA       Model: ST3320620AS       Rev: 3.AA
  Type:   Direct-Access                      ANSI SCSI revision: 05
sd 0:0:0:0: Attached scsi generic sg0 type 0
SCSI device sdb: 625142448 512-byte hdwr sectors (320073 MB)
sdb: Write Protect is off
sdb: Mode Sense: 00 3a 00 00
SCSI device sdb: drive cache: write back
SCSI device sdb: 625142448 512-byte hdwr sectors (320073 MB)
sdb: Write Protect is off
sdb: Mode Sense: 00 3a 00 00
SCSI device sdb: drive cache: write back
 sdb: sdb1 sdb2 sdb3 sdb4 < sdb5 sdb6 >
sd 1:0:0:0: Attached scsi disk sdb
sd 1:0:0:0: Attached scsi generic sg1 type 0
BIOS EDD facility v0.16 2004-Jun-25, 2 devices found
md: md6 stopped.
md: bind<sda6>
md: bind<sdb6>
raid1: raid set md6 active with 2 out of 2 mirrors
md: md2 stopped.
md: bind<sdb2>
raid1: raid set md2 active with 1 out of 2 mirrors
Attempting manual resume
md: md2 stopped.
md: unbind<sdb2>
md: export_rdev(sdb2)
md: md6 still in use.
md: md6 still in use.
md: md6 still in use.
md: md1 stopped.
md: bind<sdb1>
raid1: raid set md1 active with 1 out of 2 mirrors
md: md2 stopped.
md: bind<sdb2>
raid1: raid set md2 active with 1 out of 2 mirrors
md: md3 stopped.
md: bind<sdb3>
raid1: raid set md3 active with 1 out of 2 mirrors
md: md5 stopped.
md: bind<sdb5>
raid1: raid set md5 active with 1 out of 2 mirrors
ReiserFS: md2: found reiserfs format "3.6" with standard journal
ReiserFS: md2: using ordered data mode
reiserfs: using flush barriers
ReiserFS: md2: journal params: device md2, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30
ReiserFS: md2: checking transaction log (md2)
ReiserFS: md2: Using r5 hash to sort names
Adding 1020024k swap on /dev/md6.  Priority:-1 extents:1 across:1020024k
ReiserFS: md3: found reiserfs format "3.6" with standard journal
ReiserFS: md3: using ordered data mode
reiserfs: using flush barriers
ReiserFS: md3: journal params: device md3, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30
ReiserFS: md3: checking transaction log (md3)
ReiserFS: md3: Using r5 hash to sort names

I should mention that the only raid device that is working, md6, is a swap partition.
0
 
LVL 7

Expert Comment

by:ezaton
ID: 19543268
Do not raid your swap! This is a complete waste of CPU power, and by all means not recommended at all!

Also - listen - converting an existing partitions into software raid is a complicated task, and I can tell you it is not simple. Not at all (follow the raid howto to do it). If you have the ability to extract your data out of the system, do so, and rebuild everything, this time with raid setup during install time. It will save you time - lots of time.
0
 

Author Comment

by:zanotech
ID: 19543351
The swap was mirrored in the excellent article from the Novell web site called "Migrating and existing Suse Linux Enterprise Server 9 System to RAID1"  It all seemed to go nicely step by step, and all the reader comments gave it excellent opinions.  If swap wasn't mirrored, then the system may not boot right if there was a disk failure?

In any case, I don't think that mirroring the swap is causing my problems.  I don't fancy rebuilding because that would be a total pain.  I hope that a few hours trying to fix this will do it.   Plus you only learn by trying to crack problems. Surely I am not the only person to have ever come across pesky md's that would stay synced after a reboot.  Any good ideas on the best way to fix this (without rebuilding).

It is 0210hrs, and work beckons tomorrow!! :-o
I'll check back tomorrow.
Good night!
0
 
LVL 7

Expert Comment

by:ezaton
ID: 19543364
Your system will boot even without swap partitions at all. Mirroring swap is wasting cpu, and since I'm tired myself, I would recommend google on the issue.
0
 
LVL 19

Expert Comment

by:alextoft
ID: 19545457
That Novell article is sound. I used it as reference a year ago when I migrated a customer server to a raid1 mirror when it went into production from a dev environment. It's been fine ever since :/ That said, I did modify the procedure as he used ext3 which whilst technically inferior in a number of ways, I use in preference to Reiser.

There's no reason why you shouldn't raid your swap if you want true redundancy, the CPU overhead is minimal (I benchmarked it with/without and unless your box is getting seriously hammered you'd never even notice a difference).

When you're using the machine on a daily basis (do stuff that involves a lot of file transactions...) do you see any output to dmesg?

0
 
LVL 7

Expert Comment

by:ezaton
ID: 19545498
When your system will actually swap, as it sometimes happen, your overhead will be terrible. Look at it that way - each I/O operation is a n expensive operation. When you invoke software mirror for any normal data, you pay with the additional (minor) delay of the kernel daemon [md_raid1] (or [raid1] working. Your swap operations run in nice -10, which means they have a higher priority. It means that during swapping, your system is less responsive, and processes take longer to respond (which is natural), and, of course, you've just added the additional overhead of the kernel daemon.

This is not recommended, it affects performance under heavy load (and I've seen several such setups which were overloaded, and when the raid for the swap was stopped, started "breathing"), and since software Raid is as good as your hardware would allow (if your disk will not report corrupted data, what happens next? - and again, I've had such cases), several minor "dents" on the data side can easily be fixed, while corrupted swap will cause kernel panic or segfault of programs.

Buy hey - you learn from your own experience. You try and you test. It's a bit more tricky to test a faulty but still existing disk.
0
 
LVL 43

Expert Comment

by:ravenpl
ID: 19545520
ezaton: there was a time that I was thinking exatly like You, but then someone told me:
I prefer to pay the minimal performance loose, but gain system stability in case of emergency.
Remember - if You loose swap, any swapped process will be killed immediately. What worse, kernel can be killed as well. The vmalloced memory may be swapped as well! If You have swap mirrored, above errors are recoverable.

So as You see it's user's choice ans saying it's pointless is not in place.

However You can say that for raid0. There are people who put it's swap on raid0. Since kernel swap code can do striping by itself, it's pointless to put swap on raid0.

Let's back to the main question. Shall we?
0
 

Author Comment

by:zanotech
ID: 19547787
Hi Alextoft, Ezaton and Revenpl,
That was an interesting read about SWAP mirroring, and it is obvious that your are experienced linux people.  I can see arguments for and against mirroring swap, though I do like the "I prefer to pay the minimal performance loose, but gain system stability in case of emergency."

On the subject of swap partitions, my swap is on sdb6.  Is this a big no-no?  Someone told me that it should be sdb1.

In any case, my problem still looms, and I am still no closer to cracking it, so as with many things IT related, if it doesn't work, undo it, then redo.  

So what I'd like to propose as a course of action is to break the mirror so my system just runs on md1,md2,md3... etc, where it is just using /dev/sdb.  Then just mirror one partition at a time, then reboot to see if it stays mirrored.  Is this worth doing?  Does anyone have clear steps to do this or any warnings?

My original partitions were:
/dev/sdb1 /boot
/dev/sdb2 /
/dev/sdb3 /home
/dev/sdb5 /windows/C
/dev/sdb6 swap

I am not even using /dev/sdb5 at the moment so I thougt that would be a good partition to practise with.
How does one break a mirror safely, then rebuild it?
0
 
LVL 7

Assisted Solution

by:ezaton
ezaton earned 600 total points
ID: 19547914
You can do this by running for failing a disk
mdadm --manage /dev/md1 -f /dev/sda1
for removing the failed disk
mdadm  --manage /dev/md1 -r /dev/sda1
and to add the device
mdadm  --manage /dev/md1 -a /dev/sda1
I would also consider using the -U to be on the safe side:
mdadm --assemble /dev/md1 -U

Also, to check that things work correctly, you cat attempt to stop one of the arrays and re-start it, for example:
mdadm --manage /dev/md5 -S
mdadm --manage /dev/md5 -R

Good luck.
0
 
LVL 43

Expert Comment

by:ravenpl
ID: 19548095
> On the subject of swap partitions, my swap is on sdb6.  Is this a big no-no?  Someone told me that it should be sdb1.
Again, it's Your choice.
sdb1 if faster than sda6 (hdd is fastest at first sectors, ans slowest at last sectors).
Also it may matter if You want to repartition. It's way easier to short/extend partition while keeping startOffset untouched, instead of moving startOffset of partition.

One more Q, cause it's not clear to me.
The mirrors were crated from scratch, or You have had /dev/sda partitioned, then plugged /dev/sdb and changed to mirror?
0
 

Author Comment

by:zanotech
ID: 19548750
ezaton:
Thanks for that,  I will look to give it a go.  Will backup important data first though.  

Ravenpl:
My original system was running with just /dev/sda, then I installed /dev/sdb.  Set up the md with just the /dev/sdb disk, then later added the /dev/sda partitions to the md's.

To err on the side of caution, is it recommended to backup my disk partition structure? So if things go south say with the partition tables, I can recover.  Is the fdisk -l command good enough for this?

0
 
LVL 7

Expert Comment

by:ezaton
ID: 19549032
zanotech - it should be good enough, as long as you have not initiated a resync or formated any of the modified partitions.

About volume changes, ravenpl - I myself use LVM above software raid1.
My setup is:
/boot -> /dev/md0 (sda1 & sdb1)
swap#1 /dev/sda2
swap#2 /dev/sdb2
pv for LVM -> /dev/md1 (sda3 and sdb3)
On the LVM:
/ as 10GB /dev/VG00/rootLV
/home 50GB /dev/VG00/homeLV

etc.

This allows me to alter partition layout and modify sizes, where changing existing md devices is quite complicated, and changes to a used disk partition layout will not update kernel (yes, I am familiar with partprobe). I wouldn't recommend this.
The disadvantages of this setup is the time it takes to rebuild the raid in case of an unclean state (happens at power outage, sometimes). I wanted to use LVM only, but LVM requires three PVs for normal operations, and not just two (the 3rd uses for PV clear/dirty mapping). I do not know what is the effect of using two of the PVs (mirror target and bitmap) on the same physical disk, so I didn't want to risk it.

Back to MD - these are my experience and my suggestions accordingly.
0
 
LVL 43

Expert Comment

by:ravenpl
ID: 19549432
I'm not sure what happened, and I don't know why it will not ork for You (hot adding device).
But there 's one thing which is disallowed. Was it Your case?
The thing is You can't upgrade filesystem on raw partition to md partition.
Eg. You have /dev/sdb1 formatted as reiserfs. You decided to mirror it. Attached /dev/sda, unmounted /dev/sdb1, created mirror with just /dev/sdb1, hot-added /dev/sda1 to the mirror. This will not work! Mirror will save persistent superblock at the end of partitions corrupting reiserfs' data. But You will mount it OK. After it's mnounted, reiserfs will corrupt mirror's data. On reboot Mirrors will not match one against other, and only one will be choosen. And so on.
0
 
LVL 7

Expert Comment

by:ezaton
ID: 19549835
ravenpl, I agree. Changing the partition type using fdisk won't solve it. I was under the impression that the size of the partition was changed according to "Migrating and existing Suse Linux Enterprise Server 9 System to RAID1" guide, which I assumed to include this data. In the original linux software raid howto this issue is being discussed.
0
 

Author Comment

by:zanotech
ID: 19552215
Hi
I am a little lost in the last 3 posts...It looks like there is a suggestion to where my problem is ??  Is there something I should check?
0
 
LVL 7

Expert Comment

by:ezaton
ID: 19552363
Although some of it is a bit old, it is still relevant:
http://tldp.org/HOWTO/Software-RAID-HOWTO-7.html#ss7.4
Check method 2 - this is the method you were supposed to use.
It goes, in general (and logic) like that:
1. Set partitions to raid autodetect (already done by you)
2. If your / is on /dev/sda2, build an md device with setting your existing root (/dev/sda2) drive as failed.
3. Format the md device (this will not affect your existing /, as it is still mounted, and your md device will not overwrite it). You can reboot and verify that the raid device (for the md device setup just now) starts correctly - again - with a failed disk.
4. this is covered there with a simple sentence - "Copy the system files". It means you mkdir some /mnt/temp
mount /dev/md0 (for the matter) /mnt/temp
cd /
tar cf - --one-file-system . | ( cd /mnt/temp ; tar xvf - )
When you're done, you can edit the "new" fstab file inside /mnt/temp/etc and direct it to use a different root (your /dev/md0, in my example). Remember to update your /boot/grub/menu.lst file with your root= directive.
5. umount /mnt/temp and reboot.

Notice that I do not know how does /etc/raidtab is affecting the system nowadays. When the guide was written, it was important, however, on my system I do not have such a file. As far as I know all modern md systems use the superblock to contain the raid information.

I hope it solves your problem.
0
 

Author Comment

by:zanotech
ID: 19589509
Hi Gents

Here is my latest foray into this issue.  I am becoming more comfortable with mdadm BUT still have an issue.

As usual all my /dev/mdx's except /dev/md6 revert to a failed state after a reboot.

# HERE IS THE RAID STATUS AFTER A REBOOT
cat /proc/mdstat
Personalities : [raid1]
md5 : active raid1 sdb5[0]
      24121472 blocks [2/1] [U_]

md3 : active raid1 sdb3[0]
      276832000 blocks [2/1] [U_]

md1 : active raid1 sdb1[0]
      104320 blocks [2/1] [U_]

md2 : active raid1 sdb2[0]
      10490368 blocks [2/1] [U_]

md6 : active raid1 sdb6[0] sda6[1]
      1020032 blocks [2/2] [UU]

unused devices: <none>

I proposed to test stopping and starting one of the raid devices without rebooting to see if the raid device reverts to a failed state similar to a reboot.  I used /dev/md5 as it contains no data.

I have grouped related output between hash lines.

################################################
# ADD THE NON-RAID DISK PARTITION TO IT'S RESPECTIVE RAID ARRAY.
mdadm /dev/md5 -a /dev/sda5

# DMESG
md: bind<sda5>
RAID1 conf printout:
 --- wd:1 rd:2
 disk 0, wo:0, o:1, dev:sdb5
 disk 1, wo:1, o:1, dev:sda5
md: syncing RAID array md5
md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc.
md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction.
md: using 128k window, over a total of 24121472 blocks.

# THE RECOVERY FOR MD5 PROCEEDS HAPPILY TO 100%.  SO FAR SO GOOD.
cat /proc/mdstat
Personalities : [raid1]
md5 : active raid1 sda5[1] sdb5[0]
      24121472 blocks [2/2] [UU]

md3 : active raid1 sdb3[0]
      276832000 blocks [2/1] [U_]

md1 : active raid1 sdb1[0]
      104320 blocks [2/1] [U_]

md2 : active raid1 sdb2[0]
      10490368 blocks [2/1] [U_]

md6 : active raid1 sdb6[0] sda6[1]
      1020032 blocks [2/2] [UU]

unused devices: <none>

###############################################
#STOP THE ARRAY

mdadm -S /dev/md5
cat /proc/mdstat
Personalities : [raid1]
md3 : active raid1 sdb3[0]
      276832000 blocks [2/1] [U_]

md1 : active raid1 sdb1[0]
      104320 blocks [2/1] [U_]

md2 : active raid1 sdb2[0]
      10490368 blocks [2/1] [U_]

md6 : active raid1 sdb6[0] sda6[1]
      1020032 blocks [2/2] [UU]

#DMESG
RAID1 conf printout:
 --- wd:2 rd:2
 disk 0, wo:0, o:1, dev:sdb5
 disk 1, wo:0, o:1, dev:sda5
md: md5 stopped.
md: unbind<sda5>
md: export_rdev(sda5)
md: unbind<sdb5>
md: export_rdev(sdb5)

##############################################

START THE ARRAY
mdadm --assemble --scan /dev/md5
mdadm: /dev/md5 has been started with 2 drives.

#DMESG
md: md5 stopped.
md: bind<sda5>
md: bind<sdb5>
raid1: raid set md5 active with 2 out of 2 mirrors

# ARRAY MD5 STARTED SUCCESSFULLY!
cat /proc/mdstat
Personalities : [raid1]
md5 : active raid1 sdb5[0] sda5[1]
      24121472 blocks [2/2] [UU]

md3 : active raid1 sdb3[0]
      276832000 blocks [2/1] [U_]

md1 : active raid1 sdb1[0]
      104320 blocks [2/1] [U_]

md2 : active raid1 sdb2[0]
      10490368 blocks [2/1] [U_]

md6 : active raid1 sdb6[0] sda6[1]
      1020032 blocks [2/2] [UU]

unused devices: <none>

##############################################

So the array can be stopped and started online without a problem.
Now for a reboot to see if  /dev/md5 remains intact.

Reboot...

# SAME AS EVER AFTER A REBOOT.  ALL md DEVICES EXCEPT md6 ARE BROKEN!! :-(
cat /proc/mdstat
Personalities : [raid1]
md5 : active raid1 sdb5[0]
      24121472 blocks [2/1] [U_]

md3 : active raid1 sdb3[0]
      276832000 blocks [2/1] [U_]

md1 : active raid1 sdb1[0]
      104320 blocks [2/1] [U_]

md2 : active raid1 sdb2[0]
      10490368 blocks [2/1] [U_]

md6 : active raid1 sdb6[0] sda6[1]
      1020032 blocks [2/2] [UU]

unused devices: <none>

# DMESG EXTRACT
md: md6 stopped.
md: bind<sda6>
md: bind<sdb6>
raid1: raid set md6 active with 2 out of 2 mirrors
md: md2 stopped.
md: bind<sdb2>
raid1: raid set md2 active with 1 out of 2 mirrors
Attempting manual resume
md: md2 stopped.
md: unbind<sdb2>
md: export_rdev(sdb2)
md: md6 still in use.
md: md6 still in use.
md: md6 still in use.
md: md1 stopped.
md: bind<sdb1>
raid1: raid set md1 active with 1 out of 2 mirrors
md: md2 stopped.
md: bind<sdb2>
raid1: raid set md2 active with 1 out of 2 mirrors
md: md3 stopped.
md: bind<sdb3>
raid1: raid set md3 active with 1 out of 2 mirrors
md: md5 stopped.
md: bind<sdb5>
raid1: raid set md5 active with 1 out of 2 mirrors

##############################################

From my testing it can be seen that the test md device /dev/md5 could be manually stopped and started OK.
Yet a reboot breaks it!!!

I have spent hours on this and even though I have learnt more about mdadm, I am no closer to solving my problem.
Where can I possible go to from here??

Why would the swap md (md6) remain intact?  Could it be causing the issue?

What could possibly cause the md's to break at reboot?!
0
 
LVL 7

Expert Comment

by:ezaton
ID: 19590293
As said before - your filesystem corrupts the superblock. It means that until you mount the device, you will have a good looking md device. When you mount it, you will overwrite the superblock.
Try, before the stopping stage to mount the filesystem, and then unmount it, and then continue the test from that point (stopping, etc). Let us know what were the results.
0
 
LVL 43

Expert Comment

by:ravenpl
ID: 19590360
Since md5 has no data, please destroy it, then build again, reformat.
umount /dev/md5
mdadm -S /dev/md5
mdadm -C /dev/md5 -l1 -n2 /dev/sd[ab]5 # will propably warn about existing filesystem
mke2fs -O dir_index,has_journal /dev/md5 # if it should have label, add: -L TheLabel
mount /dev/md5
reboot

if now it's fine, then Yuu did as I guessed. One can't upgrade existing filesystem to mdX filesystem.
0
 

Author Comment

by:zanotech
ID: 19590433
Hi guys

Thanks for the feedback...

ravenpl - "One can't upgrade existing filesystem to mdX filesystem."
I think what you wrote can't be correct, because according to the document I used to assist me mirror the disks it is possible. See http://www.novell.com/coolsolutions/appnote/15087.html 
In that doc, search for "*WARNING*: This is the point of no return." where the instruction are to add the original data partition to the new array.
-----------------------------------------------------
linux:~ # mdadm /dev/md1 -a /dev/sda1
mdadm: hot added /dev/sda1
----------------------------------------------------

Strangely I don't recall see "hot added" when I carried out my test.  Not sure if that matters, could just be different versions of mdadm.

ezaton - Thanks also, I'll test what you say.
0
 
LVL 43

Accepted Solution

by:
ravenpl earned 900 total points
ID: 19590450
> "One can't upgrade existing filesystem to mdX filesystem."
It is true. The link You provided says the same. You have to copy all data from old (raw) partition to new(degradated raid). Was it Your case? If so, then You did right, and I'm just checking.
0
 

Author Comment

by:zanotech
ID: 19590477
I did copy all the data from the old partitions to the new degraded md ones.  I did run the raid in a degraded state for about a week before I attempted to do the final steps:
mdadm /dev/mdx -a /dev/sday
Could this weeks delay have caused my problems, because then the partitions being added to the array would not resemble the original data that was copied to the degraded raid devices??  I didn't think that this would matter because in for example
mdadm /dev/md1 -a /dev/sda1
sda1 will be simply overwritten would it not?
0
 

Author Comment

by:zanotech
ID: 19681067
Sorry for the delay with this...I will test again ASAP.  Work...kids...you know how it is!
0
 

Author Comment

by:zanotech
ID: 19845186
I will definately be back on to this unresolved issue after work tonight...
0
 

Author Comment

by:zanotech
ID: 19852422
I spent about 3hrs on my problem last night...reading and googling.  I was still at a loss.  I thought that maybe there is some lowlevel problem with the partitions that I am trying to add to the md devices.  So I did this:

1.  Fail the sda[1-6] devices from the /dev/md[1-6] devices. eg.  madam --manage /dev/md1 -f /dev/sda1
2.  Remove the failed sda[1-6] devices.  eg madam --manage /dev/md1 -r /dev/sda1
3.  Deleted all partions on /dev/sda with fdisk d command.
4.  Recreated all partitions on /dev/sda
5.  Added /dev/sda partitions back to the /dev/md devices. eg mdadm --manage /dev/md1 -a /dev/sda1

Then I checked for mirroring with cat /proc/mdstat and it looked fine, all got to 100% mirrored.
But then a reboot broke all the mirrors again, except for (as usual) /dev/sda6 which is the swap partition.

I have spent hours on this and have exactly the same problem that I started with. A little frustrated, but I think I am at the point where my only option is to backup my data and wipe the system clean.

I think that it would be good if mdadm had some vebose logging capability to see exactly what is happening at boot time, as it appears that this is where my problem is and dmesg doesn't give enough clues.

If anyone has any last minute advice, please be quick, because in about 2hrs I should be ready to wipe the system and start again.  A relieving thoughth in some strange way :-)
0
 

Author Comment

by:zanotech
ID: 19857110
Nothing like wiping a system and rebuilding it!!
All works OK now and I have redundancy - woohoo!
Thanks for sticking with me along the way ravenpl and ezaton, I have become that much wiser from your assistance.
I'll split the points between you both.

Thanks
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Setting up Secure Ubuntu server on VMware 1.      Insert the Ubuntu Server distribution CD or attach the ISO of the CD which is in the “Datastore”. Note that it is important to install the x64 edition on servers, not the X86 editions. 2.      Power on th…
The purpose of this article is to demonstrate how we can use conditional statements using Python.
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.
Suggested Courses
Course of the Month14 days, 13 hours left to enroll

840 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question