[x]
Posted via EE Mobile

Search, ask, and monitor your questions on the go with EE Mobile. Visit Experts Exchange from your mobile device and never be out of touch again.

Question
[x]
Attachment Details
[x]
The Solution Rating System

With so many solutions, how can you tell which solutions are most likely to help you and which ones are not? To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.

  • The Grade of the Solution
  • The Zone Rank of the Expert Providing the Solution
  • The Number of Author and Expert Comments
  • The Number of Experts Contributing
  • The Feedback of the Community

Your Input Matters
Because of the way the system is set up, the most important variable in this equation is you. As a member of Experts Exchange, you are able to cast your vote on the quality of the solutions in regard to how complete, accurate, helpful and easy to understand each solution is. When you provide your feedback, each rating is adjusted accordingly. So, if you see a solution that has a poor rating that you think is a good solution, let us know by rating it. As you do, the rating will be adjusted and will become more accurate for other members of our site.

If you have any suggestions that you would like to make for our rating system, please ask a question in the Suggestions Zone of Community Support.

Thank you!

8.0

Linux software RAID 1 - root filesystem becomes read-only after a fault on one disk

Asked by DrStalker in Linux, Disaster Recovery, Storage Technology

Tags: linux software-raid raid

Linux software RAID 1 locking to read-only mode

The setup:  
Centos 5.2, 2x 320 GB sata drives in RAID 1.

 - /dev/md0 (/dev/sda1 + /dev/sdb1) is
   /boot
 - /dev/md1 (/dev/sda1 +
   /dev/sdb1) is an LVM partition which
   contains /, /data and swap partitions

All filesystems other than swap are ext3

We've had problem on several systems where a fault on one drive has locked the root filesystem as readonly, which obviously causes problems.
    [root@myserver /]# mount | grep Root
    /dev/mapper/VolGroup00-LogVolRoot on / type ext3 (rw)
    [root@myserver /]# touch /foo
    touch: cannot touch `/foo': Read-only file system

I can see that one of the partitions in the array is faulted:

    [root@myserver /]# mdadm --detail /dev/md1
    /dev/md1:
    [...]
              State : clean, degraded
     Active Devices : 1
    Working Devices : 1
     Failed Devices : 1
      Spare Devices : 0
    [...]
        Number   Major   Minor   RaidDevice State
           0       0        0        0      removed
           1       8       18        1      active sync   /dev/sdb2
           2       8        2        -      faulty spare   /dev/sda2


Remounting as rw fails:

    [root@myserver /]# mount -n -o remount /
    mount: block device /dev/VolGroup00/LogVolRoot is write-protected, mounting read-only

The LVM tools give an error unless --ignorelockingfailure is used (because they can't write to /var) but show the volume group as rw:

    [root@myserver /]# lvm vgdisplay
    Locking type 1 initialisation failed.
    [root@myserver /]# lvm pvdisplay --ignorelockingfailure
      --- Physical volume ---
      PV Name               /dev/md1
      VG Name               VolGroup00
      PV Size               279.36 GB / not usable 15.56 MB
      Allocatable           yes (but full)
      [...]
   
    [root@myserver /]# lvm vgdisplay --ignorelockingfailure
      --- Volume group ---
      VG Name               VolGroup00
      System ID
      Format                lvm2
      Metadata Areas        1
      Metadata Sequence No  4
      VG Access             read/write
      VG Status             resizable
      [...]
   
    [root@myserver /]# lvm lvdisplay /dev/VolGroup00/LogVolRoot --ignorelockingfailure
      --- Logical volume ---
      LV Name                /dev/VolGroup00/LogVolRoot
      VG Name                VolGroup00
      LV UUID                PGoY0f-rXqj-xH4v-WMbw-jy6I-nE04-yZD3Gx
      LV Write Access        read/write
      [...]

In this case /boot (seperate RAID meta-device) and /data (a different logical volume in the same volume group) are still writtable.  From the previous occurances I know that a restart will bring the system back up with a read/write root filesystem and a properly degraded RAID array.  


So, I have two questions:

1) When this occurs, how can I get the root filesystem back to read/write without a system restart?

2) What needs to be changed to stop this filesystem locking?  With a RAID 1 failure on a single disk we don't want the filesystems to lockup, we want the system to keep running until we can replace the bad disk.

[+][-]08/10/09 01:51 AM, ID: 25058087Accepted Solution

View this solution now by starting your 30-day free trial. Setting up your free trial is quick, easy, and secure. We will return you to this solution, unlocked, when you're done.

About this solution

Zones: Linux, Disaster Recovery, Storage Technology
Tags: linux software-raid raid
Sign Up Now!
Solution Provided By: diepes
Participating Experts: 1
Solution Grade: A
 
[+][-]08/09/09 07:59 PM, ID: 25056927Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
 
Loading Advertisement...
20091111-EE-VQP-92 - Hierarchy / EE_QW_3_20080625