blentz
asked on
ext3 read-only after SAN timeout
I've had storage system issues (SAN / IBM VIO) on a couple of my linux systems, where the kernel has detected that the EXT3 file system has errors and has forced the block device into read-only mode.
I know that I can clear this up by rebooting (perhaps w/fsck), but I need a solution to get these file systems back into read-write mode *without rebooting*.
I want to keep error-behavior set to remount-ro, but once the storage issue has been resolved, I want the file system available for read-write again.
Nothing seems to work. The file system is read-write at the LVM level and at the blockdev level, but the kernel refuses to cooperate.
Deleted by TieFighterMod, 250 points refunded. - 5/10/2009 1:40:39 PM
I know that I can clear this up by rebooting (perhaps w/fsck), but I need a solution to get these file systems back into read-write mode *without rebooting*.
I want to keep error-behavior set to remount-ro, but once the storage issue has been resolved, I want the file system available for read-write again.
Nothing seems to work. The file system is read-write at the LVM level and at the blockdev level, but the kernel refuses to cooperate.
Deleted by TieFighterMod, 250 points refunded. - 5/10/2009 1:40:39 PM
# mount / -o remount,rw
mount: block device /dev/rootvg/rootlv is write-protected, mounting read-only
]# lvdisplay --ignorelockingfailure /dev/rootvg/rootlv | grep "LV Write Access"
LV Write Access read/write
# blockdev --report /dev/rootvg/rootlv
RO RA SSZ BSZ StartSec Size Device
rw 256 512 1024 0 786432 /dev/rootvg/rootlv
ASKER
Thanks for the link, but it really doesn't help. I'm running RHEL 5.2 with all updates, so the kernel is at least 2.6.18. I still need to know how to recover the block devices in read-write without a reboot.
can you manually umount and remount the filesystem?
ASKER
Yes, that works for /var (if I quiet the system enough) but the /usr and / file systems are also having this problem, and I can't think of a way to dismount / without building chroot system somewhere else or using some other trickery.
It begs to question, who does an umount/mount work but a remount is stuck read-only?
It begs to question, who does an umount/mount work but a remount is stuck read-only?
If /usr/ and / located in the SAN, you have to reboot, their is not other options till Linux can fix the kernel bug for this case.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Did anyone find a solution for this issue in the end? We have exactly the same problem - just with Centos and SAN - ext3 going read only when the SAN cluster fails over.
How do you re-mount the LUN to read / write without a reboot?
Anyone?
O
How do you re-mount the LUN to read / write without a reboot?
Anyone?
O
ASKER
I don't think you can. The options for an ext3 mount are either read-only or panic, the latter of which is actually worse!!
For us the solution was to ensure that all disks were under the control of multipath. However, there's yet another problem in RHEL where multipath, LVM, and md (RAID1 the operating system volume group) do NOT play together at all.
I have a feature request open with Red Hat on this but nobody's worked on it yet.
It's heartbreaking because I work in an AIX shop where mutlipath vscsi (VIOS) disks along with LVM mirroring for the operating system volume group (rootvg) is standard issue.. we do it on all AIX builds without any issue. :-( boo Red Hat.
For us the solution was to ensure that all disks were under the control of multipath. However, there's yet another problem in RHEL where multipath, LVM, and md (RAID1 the operating system volume group) do NOT play together at all.
I have a feature request open with Red Hat on this but nobody's worked on it yet.
It's heartbreaking because I work in an AIX shop where mutlipath vscsi (VIOS) disks along with LVM mirroring for the operating system volume group (rootvg) is standard issue.. we do it on all AIX builds without any issue. :-( boo Red Hat.
"Why does the ext3 filesystems on my Storage Area Network (SAN) repeatedly become read-only?"
http://www.redhat.com/magazine/026dec06/features/tips_tricks/