[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

ext3 read-only after SAN timeout

Posted on 2009-02-12
11
Medium Priority
?
3,733 Views
Last Modified: 2012-06-21
I've had storage system issues (SAN / IBM VIO) on a couple of my linux systems, where the kernel has detected that the EXT3 file system has errors and has forced the block device into read-only mode.

I know that I can clear this up by rebooting (perhaps w/fsck), but I need a solution to get these file systems back into read-write mode *without rebooting*.

I want to keep error-behavior set to remount-ro, but once the storage issue has been resolved, I want the file system available for read-write again.

Nothing seems to work. The file system is read-write at the LVM level and at the blockdev level, but the kernel refuses to cooperate.
Deleted by TieFighterMod, 250 points refunded. - 5/10/2009 1:40:39 PM
# mount / -o remount,rw
mount: block device /dev/rootvg/rootlv is write-protected, mounting read-only
 
]# lvdisplay --ignorelockingfailure /dev/rootvg/rootlv | grep "LV Write Access"
  LV Write Access        read/write
 
# blockdev --report /dev/rootvg/rootlv
RO    RA   SSZ   BSZ   StartSec     Size    Device
rw   256   512  1024          0     786432  /dev/rootvg/rootlv

Open in new window

0
Comment
Question by:blentz
  • 4
  • 3
8 Comments
 
LVL 38

Expert Comment

by:yuzh
ID: 23628363
Please have a look at  the folloqing page, pay attention to the section:

"Why does the ext3 filesystems on my Storage Area Network (SAN) repeatedly become read-only?"


http://www.redhat.com/magazine/026dec06/features/tips_tricks/
0
 
LVL 2

Author Comment

by:blentz
ID: 23629021
Thanks for the link, but it really doesn't help. I'm running RHEL 5.2 with all updates, so the kernel is at least 2.6.18. I still need to know how to recover the block devices in read-write without a reboot.
0
 
LVL 38

Expert Comment

by:yuzh
ID: 23630076
can you manually umount and remount the filesystem?
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 2

Author Comment

by:blentz
ID: 23632849
Yes, that works for /var (if I quiet the system enough) but the /usr and / file systems are also having this problem, and I can't think of a way to dismount / without building chroot system somewhere else or using some other trickery.

It begs to question, who does an umount/mount work but a remount is stuck read-only?
0
 
LVL 38

Expert Comment

by:yuzh
ID: 23646756
If /usr/ and / located in the SAN, you have to reboot, their is not other options till Linux can fix the kernel bug for this case.
0
 
LVL 2

Accepted Solution

by:
blentz earned 0 total points
ID: 24348894
I am not satisfied with any answered provided so far. Our RHEL systems need to be highly available, regardless of redundant SAN failover, and since this question has not assisted, I would like the question closed and points refunded.
0
 

Expert Comment

by:otisjones
ID: 32714655
Did anyone find a solution for this issue in the end? We have exactly the same problem - just with Centos and SAN - ext3 going read only when the SAN cluster fails over.

How do you re-mount the LUN to read / write without a reboot?

Anyone?

O
0
 
LVL 2

Author Comment

by:blentz
ID: 32770215
I don't think you can. The options for an ext3 mount are either read-only or panic, the latter of which is actually worse!!

For us the solution was to ensure that all disks were under the control of multipath. However, there's yet another problem in RHEL where multipath, LVM, and md (RAID1 the operating system volume group) do NOT play together at all.

I have a feature request open with Red Hat on this but nobody's worked on it yet.

It's heartbreaking because I work in an AIX shop where mutlipath vscsi (VIOS) disks along with LVM mirroring for the operating system volume group (rootvg) is standard issue.. we do it on all AIX builds without any issue. :-( boo Red Hat.
0

Featured Post

 [eBook] Windows Nano Server

Download this FREE eBook and learn all you need to get started with Windows Nano Server, including deployment options, remote management
and troubleshooting tips and tricks

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Setting up Secure Ubuntu server on VMware 1.      Insert the Ubuntu Server distribution CD or attach the ISO of the CD which is in the “Datastore”. Note that it is important to install the x64 edition on servers, not the X86 editions. 2.      Power on th…
Every server (virtual or physical) needs a console: and the console can be provided through hardware directly connected, software for remote connections, local connections, through a KVM, etc. This document explains the different types of consol…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
Suggested Courses
Course of the Month20 days, 1 hour left to enroll

872 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question