Link to home
Start Free TrialLog in
Avatar of Nozmoking
NozmokingFlag for United States of America

asked on

Fedora Core 5 Linux server root file system drops into protected read-only mode randomly or when cron jobs run shell scripts since power outage.

I have a Linux server (FC5) that has developed problems since a power outage. The root file system become read-only periodically when altering files or shell scripts that write files run. The machine reboots clean sometimes, other times when I reboot it finds disk errors and runs a forced disk check. I'm not very well versed with Linux file systems  - it's an older Compaq Proliant 5500 and the file system is probably striped across multiple drives. I have not seen anything obvious in the logs that indicate what files or sectors may have been affected. Any help would be appreciated.
Avatar of ravenpl
ravenpl
Flag of Poland image

after the filesystem is remounted R/O, kernel should tell the reason
dmesg
command should show it.
For previous reasons scan /var/log/messages
grep kernel /var/log/messages
Does "mount -o remount,rw" helps?
ASKER CERTIFIED SOLUTION
Avatar of Ibrahim Bazarwala
Ibrahim Bazarwala
Flag of Kuwait image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of Nozmoking

ASKER

Thanks  - I'll run the disk check and post what I find in the messages log file from after the last restart.

-Noz
Log file attached -
Thanks,
-Noz
messages.txt
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
I have to check and see if the root volume is part of the striped scsi array - I'm hoping that the person that put the thing together kept the root vol on its own physical drive. This appears to be data corruption from a power failure. Since I cannot unmount the root volume to run fsck I will try and run it from the FC5 rescue cd when I get  there.

Thanks,
-Noz
I have not used the rescue cd before - since it mounts the file system but does not provide a map for fsck to read, do I run it against the mount point the recscue cd creates in rw mode? Or, do I have to unmouint the root volume first? The rescue cd mounts it here:
/dev/VolGroup00/LogVol00 on /mnt/sysimage type ext3 (rw,data=ordered)

Any advice would be appreciated,
- Noz
 
I am unable to unmount the file system VolGroup00/LogVol00 after loading the FC5 rescue cd. I can only unmount/remount in rerad-only. How can I run fsck on the root volume? I would appreciate any input.

Thanks,
- Noz
You can run fsck on read-only mounted filesystem
e2fsck -f -n /dev/VolGroup00/LogVol00
-f == force
-n == don't try repairing anything, change to -y to repair automatically, but then, don't remount it R/W! Reboot, or umount/mount.
Well...
Ran e2fsck in forced mode from FC5 recovery cd and it came back clean but there were still errors when running fsck against the root volume in read-only (nofix) mode from the terminal wioth the system running. I loaded the rescue cd again and ran e2fsck again with -c and it noted and fixed bad blocks butr now I get a kernel panic after loading the system volumes. It successfully mounts the two volumes in VolGroup00 then I get an error saying "Kernel panic - not syncing: Attempted to kill init!". Is there anything else I can do at this point or is it a dead horse?

Thanks,
- Noz
One more thought - since the root file system seems to have been damaged beyon repair at this point could I run an FC 5 upgrade and possible restore the file system to some level of functionality? This machine was task oriented so I have not really lost any data...
Thanks,
- Noz
Just to bring some resolution to this issue - can someone advise as to the easiest way to boot this server with either the fc5 rescue CD or another recovery CD (I have several CD images of recovery solution collections I've scrounged), configure the network connections, mount the bad volume and copy some "critical" data to another machine? If I could save some select data then I can go ahead and rebuild it from scratch. I  cannot load any of the current Fedora live CD images because the bugger only has 256MB of RAM, and being a Compaq 5500 we'd have to spend a few bucks to increase it. If anyone can point me to the best utility and procedure for doing that I can close this out.

Thanks!
- Noz
Just one final  follow-up; any suggestions for pulling some files off the root volume before I send the beast to the bone yard? I'd just like to load up a bare-bones OS, mount the old root volume and copy a few things over the network. Basically, I need to know what to load that would give me either secure or even insecure file transfer after starting the network interfaces and mounting the old volume, since the kernel on the old sys-vol will not start.

Thanks very much,

-Noz
Forced accept.

Computer101
EE Admin
I was able to eliminate the kernel panic and restore the server to a bootable state by using the Fedora Core 5 installation CD #1 to perform an upgrade on the OS. Although this is not the most successful of repair methods, it replaced the damaged system files caused by the power failure and then running fsck with the sysvol mounted. The system is now usable again.

- Noz.
I was able to eliminate the kernel panic and restore the server to a bootable state by using the Fedora Core 5 installation CD #1 to perform an upgrade on the OS. Although this is not the most successful of repair methods, it replaced the damaged system files caused by the power failure and running fsck with the sysvol mounted.

The system is now usable again.

- Noz.