need help to replace faulty disk using lv mirroring

Hi All,

I am using RHEL 5 .3 installed on HP hardware . its on cluster veritas on which one node works fine and second node disk is on predective disk faliure.

now i want to replace it with second hardisk using lv mirroring.

My OS installed on LV's. means all partition including root is on LV . apart from boot partition.


i am planning to add one new harddisk in a slot so that i can create lv mirroring of predective faliure disk than transfer the os to the new one after creation.

So pls guide with the right steps so that no problem occurs and with very less downtime.

Pls also guide any risk involvement in this ? is this the right way for doing it..?
The RockAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

The RockAuthor Commented:
I have checked this link:

https://www.centos.org/docs/5/html/Cluster_Logical_Volume_Manager/mirrorrecover.html

please let me know if it will works ? on rhel?

also any risk involve or should i go for any other method to replace my bad disk?

please give the right set of steps?

thank you!
0
arnoldCommented:
Presumably you currently do not have your setup with a mirror.

How many disks can you add into the system? Does your HP a system have RAID controller? 8pm it does, it might be .......

Trying to mirror an LV after the fact might not be possible.

What does the system do?

One option available to you is to add the second drive.  Set it up as a Raid Volume consisting of a single device. Make sure to install the boot record on the new drive Then use dump | restore to clone while online the partitions.

While configuring the new drive/volume as bootable, as close to a mirror of the existing environment,

Partitionn the new drive, 100-500 MB for /dev/sdb1 of type automatic raid
 mdadm would initial the mirror initially consisting of a single device the new drive.

Discussion
http://www.redhat.com/archives/fedora-list/2006-March/msg04349.html

See if transitioning to a mirrored setup using software raid is the path you want to go down.
Or just clone the existing disk into a new one with the knowledge that some data will be lost during the transition from the old disk to the new disk.


Presumably, you've taken care to backup your data.
0
The RockAuthor Commented:
what i tried till yet -

added one disk to server then created lvmirror of first disk to this one . using

lvs -a -o +devices
  LV                   VG      Attr      LSize   Pool Origin Data%  Move Log            Cpy%Sync Convert Devices
  images_lv            infravg -wi-ao---   4.00g                                                         /dev/sda3(0)
  home_lv              rootvg  mwi-aom--   1.00g                         home_lv_mlog     100.00         home_lv_mimage_0(0),home_lv_mimage_1(0)
  [home_lv_mimage_0]   rootvg  iwi-aom--   1.00g                                                         /dev/sda2(0)
  [home_lv_mimage_1]   rootvg  iwi-aom--   1.00g                                                         /dev/sdb1(96)
  [home_lv_mlog]       rootvg  lwi-aom--  32.00m                                                         /dev/sda2(629)
  opt_lv               rootvg  mwi-aom-- 512.00m                         opt_lv_mlog      100.00         opt_lv_mimage_0(0),opt_lv_mimage_1(0)
  [opt_lv_mimage_0]    rootvg  iwi-aom-- 512.00m                                                         /dev/sda2(32)
  [opt_lv_mimage_1]    rootvg  iwi-aom-- 512.00m                                                         /dev/sdb1(549)
  [opt_lv_mlog]        rootvg  lwi-aom--  32.00m                                                         /dev/sda2(632)
  root_lv              rootvg  mwi-aom--   3.00g                         root_lv_mlog     100.00         root_lv_mimage_0(0),root_lv_mimage_1(0)
  [root_lv_mimage_0]   rootvg  iwi-aom--   3.00g                                                         /dev/sda2(48)
  [root_lv_mimage_1]   rootvg  iwi-aom--   3.00g                                                         /dev/sdb1(0)
  [root_lv_mlog]       rootvg  lwi-aom--  32.00m                                                         /dev/sda2(628)
  swap_lv              rootvg  mwi-aom--   1.00g                         swap_lv_mlog     100.00         swap_lv_mimage_0(0),swap_lv_mimage_1(0)
  [swap_lv_mimage_0]   rootvg  iwi-aom--   1.00g                                                         /dev/sda2(144)
  [swap_lv_mimage_1]   rootvg  iwi-aom--   1.00g                                                         /dev/sdb1(565)
  [swap_lv_mlog]       rootvg  lwi-aom--  32.00m                                                         /dev/sda2(633)
  system_lv            rootvg  mwi-aom-- 128.00m                         system_lv_mlog   100.00         system_lv_mimage_0(0),system_lv_mimage_1(0)
  [system_lv_mimage_0] rootvg  iwi-aom-- 128.00m                                                         /dev/sda2(176)
  [system_lv_mimage_1] rootvg  iwi-aom-- 128.00m                                                         /dev/sdb1(416)
  [system_lv_mlog]     rootvg  lwi-aom--  32.00m                                                         /dev/sdb1(420)
  tmp_lv               rootvg  mwi-aom--   9.00g                         tmp_lv_mlog      100.00         tmp_lv_mimage_0(0),tmp_lv_mimage_1(0)
  [tmp_lv_mimage_0]    rootvg  iwi-aom--   9.00g                                                         /dev/sda2(180)
  [tmp_lv_mimage_0]    rootvg  iwi-aom--   9.00g                                                         /dev/sda2(372)
  [tmp_lv_mimage_1]    rootvg  iwi-aom--   9.00g                                                         /dev/sdb1(128)
  [tmp_lv_mlog]        rootvg  lwi-aom--  32.00m                                                         /dev/sda2(630)
  usr_lv               rootvg  mwi-aom--   4.00g                         usr_lv_mlog      100.00         usr_lv_mimage_0(0),usr_lv_mimage_1(0)
  [usr_lv_mimage_0]    rootvg  iwi-aom--   4.00g                                                         /dev/sda2(212)
  [usr_lv_mimage_1]    rootvg  iwi-aom--   4.00g                                                         /dev/sdb1(421)
  [usr_lv_mlog]        rootvg  lwi-aom--  32.00m                                                         /dev/sda2(631)
  var_lv               rootvg  -wi-ao---   1.00g                                                         /dev/sda2(340)

then i have verified if mirroring works or not with below command:

lvconvert -m0 /dev/rootvg/opt_lv /dev/sda2
  Logical volume opt_lv converted.

it works fine looks like

lvs -a -o +devices
  LV                   VG      Attr      LSize   Pool Origin Data%  Move Log            Cpy%Sync Convert Devices
  images_lv            infravg -wi-ao---   4.00g                                                         /dev/sda3(0)
  home_lv              rootvg  mwi-aom--   1.00g                         home_lv_mlog     100.00         home_lv_mimage_0(0),home_lv_mimage_1(0)
  [home_lv_mimage_0]   rootvg  iwi-aom--   1.00g                                                         /dev/sda2(0)
  [home_lv_mimage_1]   rootvg  iwi-aom--   1.00g                                                         /dev/sdb1(96)
  [home_lv_mlog]       rootvg  lwi-aom--  32.00m                                                         /dev/sda2(629)
  opt_lv               rootvg  -wi-ao--- 512.00m                                                         /dev/sdb1(549)
  root_lv              rootvg  mwi-aom--   3.00g                         root_lv_mlog     100.00         root_lv_mimage_0(0),root_lv_mimage_1(0)
  [root_lv_mimage_0]   rootvg  iwi-aom--   3.00g                                                         /dev/sda2(48)
  [root_lv_mimage_1]   rootvg  iwi-aom--   3.00g                                                         /dev/sdb1(0)
  [root_lv_mlog]       rootvg  lwi-aom--  32.00m                                                         /dev/sda2(628)
  swap_lv              rootvg  mwi-aom--   1.00g                         swap_lv_mlog     100.00         swap_lv_mimage_0(0),swap_lv_mimage_1(0)
  [swap_lv_mimage_0]   rootvg  iwi-aom--   1.00g                                                         /dev/sda2(144)
  [swap_lv_mimage_1]   rootvg  iwi-aom--   1.00g                                                         /dev/sdb1(565)
  [swap_lv_mlog]       rootvg  lwi-aom--  32.00m                                                         /dev/sda2(633)
  system_lv            rootvg  mwi-aom-- 128.00m                         system_lv_mlog   100.00         system_lv_mimage_0(0),system_lv_mimage_1(0)
  [system_lv_mimage_0] rootvg  iwi-aom-- 128.00m                                                         /dev/sda2(176)
  [system_lv_mimage_1] rootvg  iwi-aom-- 128.00m                                                         /dev/sdb1(416)
  [system_lv_mlog]     rootvg  lwi-aom--  32.00m                                                         /dev/sdb1(420)
  tmp_lv               rootvg  mwi-aom--   9.00g                         tmp_lv_mlog      100.00         tmp_lv_mimage_0(0),tmp_lv_mimage_1(0)
  [tmp_lv_mimage_0]    rootvg  iwi-aom--   9.00g                                                         /dev/sda2(180)
  [tmp_lv_mimage_0]    rootvg  iwi-aom--   9.00g                                                         /dev/sda2(372)
  [tmp_lv_mimage_1]    rootvg  iwi-aom--   9.00g                                                         /dev/sdb1(128)
  [tmp_lv_mlog]        rootvg  lwi-aom--  32.00m                                                         /dev/sda2(630)
  usr_lv               rootvg  mwi-aom--   4.00g                         usr_lv_mlog      100.00         usr_lv_mimage_0(0),usr_lv_mimage_1(0)
  [usr_lv_mimage_0]    rootvg  iwi-aom--   4.00g                                                         /dev/sda2(212)
  [usr_lv_mimage_1]    rootvg  iwi-aom--   4.00g                                                         /dev/sdb1(421)
  [usr_lv_mlog]        rootvg  lwi-aom--  32.00m                                                         /dev/sda2(631)
  var_lv               rootvg  -wi-ao---   1.00g                                                         /dev/sda2(340)



pls have look and suggest if above things will work to replace faulty disk.??

now problem comes with bootable disk /dev/sda1?

How to fix it do that system can boot too from new disk - ??

is it possible to go with above solution for root and other partition and for boot partition i can create raid mirroring md0 solution ??

or any other recommended option?

if md0 could work please advice the recommended steps for the same?


thank you for reply!!
0
Acronis Data Cloud 7.8 Enhances Cyber Protection

A closer look at five essential enhancements that benefit end-users and help MSPs take their cloud data protection business further.

arnoldCommented:
You added before partitioning the new disk to match the old disk.
Your mirror includes /dev/sdb

Run fdisk -l to display the current partitions of each disk.

There is a way to install boot on /dev/sdb but it should have an identical partitioning layout.

The boot partition /boot is a raw device and for mirroring has to use software raid, mdadm.
0
The RockAuthor Commented:
So you mean to say I can't use lv mirroring with it or u r saying madam need to use for boot mirroring ? Please guide me with some steps to configure boot on /Dev/sdb ?

Thanks again
0
The RockAuthor Commented:
Just to add you are saying mdadm for all partition is recommended including boot or only for boot and rest can be with lvmirroring ?

If I can create only boot mirroring using mdadm then how to create partition for boot as we need to have different partition for boot ? Or do I need one more disk ?
0
arnoldCommented:
Before you start down this road, you should take a test system (VM if available) setup with a single drive to mimic the current physical system. and then try to convert it into the redundant setup you want.

first thing first, to provide redundancy on the /boot, you have to use mdadm
when partitioning the disk, make sure
/dev/sdb1 would be of size 100-500 MB of type linux RAID auto (fd) and active(bootable).
mdadm  -C md0 -n 2 -l 1 missing /dev/sdb1
this will create /dev/md0 RAID1 with a single device.
mkfs.ext3 /dev/md0 To create ext3 filesystem on the RAID device.
mount /dev/md0 /mnt
cd /boot
find . | cpio -pdvmu /mnt #this will get the current boot components.  You have to look at /boot/grub/grub.conf to make sure the bootup line does not reference partition UUID or it will fail given the two partitions have different UUID. and should now reflect #boot=/dev/md0
copying the MBR many sites point to dd if=/dev/sda of=/dev/sdb bs=512 count=1

http://wiki.centos.org/HowTos/Install_On_Partitionable_RAID1

you can use lvmirror, as you have it setup. to minimize the amount of transitions.
the trick with /dev/sdb1 now being a raid device making /dev/md0 is to boot the system from the second drive.
during boot, use the key needed to bring up the boot menu, at which point you should select the second drive to boot from (/dev/sdb).
Once the system boots without an issue, you would change /dev/sda1 from linux (83) to fd
mdadm /dev/md0 -a /dev/sda1
cat /proc/mdstat should reflect /dev/md0 is now made up of two drives and is being rebuilt/synchronized.
....


RedHat List serve article on the matter
https://listman.redhat.com/archives/redhat-list/2003-July/msg00173.html

there are several step by step guides on converting an existing system to a software raid....
0
The RockAuthor Commented:
Hi tried but systems crashed in my lab ..says partiton with bad sector . I have added one more disk and tried to mirror the boot to that disk.
0
The RockAuthor Commented:
Hi Arnold Thanks to help - but i tried your solution but whenever booting from new harddisk it crashed.

could you please help with grub.conf file modification also.

I am testing on Virtual machine

so when i am adding it gives me /dev/sdb so is it hd(1,0) or hd(0,1) as at the time of adding on vcenter it shows scsi(0:1).

please help
0
arnoldCommented:
the entry on the hd(x,y) does not change provided both drives are partitioned identically.
/dev/md0 if that is the /boot RAID is what the rule is. If sda dies, /dev/md0 will point to /dev/sdb and will still be the hd(x,y) since sdb is the "only" bootable drive in the system.


one option is to copy the MBR portion from /dev/sda to /dev/sdb deals with making both sdb and sda bootable.


post fdisk -l for your VM

In your VM test, did you disable the /dev/sda drive to test /dev/md0?

you are going through a transition from /dev/sda to /dev/md0 with /dev/sda and /dev/sdb as mirrors such that the transition to /dev/md0 is what fails?
0
The RockAuthor Commented:
Here it is :

fdisk -l

Disk /dev/sda: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          33      262144   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2              33        2644    20971520   8e  Linux LVM
/dev/sda3            2644        5222    20708352   8e  Linux LVM

Disk /dev/sdb: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          33      262144   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sdb2              33        2644    20971520   8e  Linux LVM
/dev/sdb3            2644        5222    20708352   8e  Linux LVM

Disk /dev/sdc: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1          32      257008+  fd  Linux raid autodetect

Disk /dev/md0: 263 MB, 263061504 bytes
2 heads, 4 sectors/track, 64224 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md0 doesn't contain a valid partition table

one more thing what does "missing" mean in your above mdadm command?
0
arnoldCommented:
Please post cat /proc/mdstat
The only device you have as raid is /dev/sdc1 it should be of the same size as /dev/sda1 as well as marked as bootable..

the missing entry in the mdadm command means that you are creating a raid 1 volume when you define /dev/md0 with one disk missing from the get go.
i.e. RAID1 with only one disk as a member. This way when another disk is added into the system, you can add a partion of it to the /dev/md0 RAID group gaining redundancy.  MDADM will auto start the rebuild of the RAID Array as soon as the second member is added.
0
The RockAuthor Commented:
I have also tried this below:

 mdadm --create --metadata=0.90 --level=1 --raid-devices=2 /dev/md_d0 /dev/sdb missing
mdadm: array /dev/md_d0 started.

 mdadm --add /dev/md_d0 /dev/sda
mdadm: Cannot open /dev/sda: Device or resource busy


 cat /proc/mdstat
Personalities : [raid1]
md_d0 : active raid1 sdb[0]
      41942976 blocks [2/1] [U_]

md0 : active raid1 sdc1[1]
      256896 blocks [2/1] [_U]

unused devices: <none>


 mount /dev/md_d0p1 /mnt/sysimage
mount: /dev/md_d0p1 is not a valid block device
0
The RockAuthor Commented:
I want to use /dev/sda and /dev/sdb for mirroring

for practice i tried with sdc also but dont know how to fix it.
0
The RockAuthor Commented:
current OS version is RHEL 5.10 i have for testing and the actual will be 5.3
0
arnoldCommented:
md0 is the device it can not be part of the you are creating it should not be part of the device list.

You must partition the drives before and use the partition when using devices.
note your second attempt created a new raid device md_d0 with /dev/sdb as ....

you could use the whole drive /dev/sdb as a member of a RAID group.

You can not add /dev/sda since it is your boot device.

The process is:
1) having a running system, you first have to partition the second disk (d/ev/sdb) with the first partition of equivalent size as the boot partition /dev/sda1 with the type of Raid Auto (fd) and bootable
2) you create a RAID device /dev/md0 or anyname you prefer of raid-level 1, with 2 device /dev/sdb1 and missing
3) you then use mkfs.extx to create a filesystem on /dev/md0 either the same filesystem that you have on /dev/sda1 or if you prefer to change it to ext4 do so.
4) mount /dev/md0 /mnt
cd /boot (presumably this is /dev/sda1)
find . -print | cpio -pdvmu /mnt # this will copy the content of /dev/sdb
5) grub
root (hd1) #for /dev/sdb when /dev/sda is runing
setup (hd1)
quit
The /mnt/grub/grub.conf you may have to play with the hd(1,0) to boot from /dev/md0 while /dev/sda is plugged in.
Duplicate the entry.
Look at /etc/fstab /boot would need to be changed to use /dev/md0 instead of /dev/sda1
only when the system is booting from /dev/md0 with /dev/sdb.
Only then you can change the /dev/sda1 partition type to fd (raid auto)
And use mdadm /dev/md0 -a /dev/sda1
cat /proc/mdstat
it will show /dev/md0 as rebuilding with /dev/sdb1 and /dev/sda1 ....
With an estimate on % complete if I am not mistaken.  The /boot partition is fairly small (100-500MB) depending on your choice.
0
arnoldCommented:
your md_do attempt includes the whole disk which means you have to partition /dev/md_d0 i.e. added it to LVm pvcreate /dev/md_d0
alternatively, you can delete /dev/md_d0
And work with /dev/sdc
or create /dev/sdb1 partiion. add it as a second drive to /dev/md0
then you will be able to replace /dev/sdc1 with /dev/sda.......
.
.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
The RockAuthor Commented:
Sorry for delay but thank you so much for your help it works
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.