Link to home
Start Free TrialLog in
Avatar of blentz
blentz

asked on

ext3 read-only after SAN timeout

I've had storage system issues (SAN / IBM VIO) on a couple of my linux systems, where the kernel has detected that the EXT3 file system has errors and has forced the block device into read-only mode.

I know that I can clear this up by rebooting (perhaps w/fsck), but I need a solution to get these file systems back into read-write mode *without rebooting*.

I want to keep error-behavior set to remount-ro, but once the storage issue has been resolved, I want the file system available for read-write again.

Nothing seems to work. The file system is read-write at the LVM level and at the blockdev level, but the kernel refuses to cooperate.
Deleted by TieFighterMod, 250 points refunded. - 5/10/2009 1:40:39 PM
# mount / -o remount,rw
mount: block device /dev/rootvg/rootlv is write-protected, mounting read-only
 
]# lvdisplay --ignorelockingfailure /dev/rootvg/rootlv | grep "LV Write Access"
  LV Write Access        read/write
 
# blockdev --report /dev/rootvg/rootlv
RO    RA   SSZ   BSZ   StartSec     Size    Device
rw   256   512  1024          0     786432  /dev/rootvg/rootlv

Open in new window

Avatar of yuzh
yuzh

Please have a look at  the folloqing page, pay attention to the section:

"Why does the ext3 filesystems on my Storage Area Network (SAN) repeatedly become read-only?"


http://www.redhat.com/magazine/026dec06/features/tips_tricks/
Avatar of blentz

ASKER

Thanks for the link, but it really doesn't help. I'm running RHEL 5.2 with all updates, so the kernel is at least 2.6.18. I still need to know how to recover the block devices in read-write without a reboot.
can you manually umount and remount the filesystem?
Avatar of blentz

ASKER

Yes, that works for /var (if I quiet the system enough) but the /usr and / file systems are also having this problem, and I can't think of a way to dismount / without building chroot system somewhere else or using some other trickery.

It begs to question, who does an umount/mount work but a remount is stuck read-only?
If /usr/ and / located in the SAN, you have to reboot, their is not other options till Linux can fix the kernel bug for this case.
ASKER CERTIFIED SOLUTION
Avatar of blentz
blentz

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Did anyone find a solution for this issue in the end? We have exactly the same problem - just with Centos and SAN - ext3 going read only when the SAN cluster fails over.

How do you re-mount the LUN to read / write without a reboot?

Anyone?

O
Avatar of blentz

ASKER

I don't think you can. The options for an ext3 mount are either read-only or panic, the latter of which is actually worse!!

For us the solution was to ensure that all disks were under the control of multipath. However, there's yet another problem in RHEL where multipath, LVM, and md (RAID1 the operating system volume group) do NOT play together at all.

I have a feature request open with Red Hat on this but nobody's worked on it yet.

It's heartbreaking because I work in an AIX shop where mutlipath vscsi (VIOS) disks along with LVM mirroring for the operating system volume group (rootvg) is standard issue.. we do it on all AIX builds without any issue. :-( boo Red Hat.