Fedora Core 5 Linux server root file system drops into protected read-only mode randomly or when cron jobs run shell scripts since power outage.

I have a Linux server (FC5) that has developed problems since a power outage. The root file system become read-only periodically when altering files or shell scripts that write files run. The machine reboots clean sometimes, other times when I reboot it finds disk errors and runs a forced disk check. I'm not very well versed with Linux file systems  - it's an older Compaq Proliant 5500 and the file system is probably striped across multiple drives. I have not seen anything obvious in the logs that indicate what files or sectors may have been affected. Any help would be appreciated.
NozmokingAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

ravenplCommented:
after the filesystem is remounted R/O, kernel should tell the reason
dmesg
command should show it.
0
ravenplCommented:
For previous reasons scan /var/log/messages
grep kernel /var/log/messages
0
ibu1System AdministratorCommented:
Does "mount -o remount,rw" helps?
0
Newly released Acronis True Image 2019

In announcing the release of the 15th Anniversary Edition of Acronis True Image 2019, the company revealed that its artificial intelligence-based anti-ransomware technology – stopped more than 200,000 ransomware attacks on 150,000 customers last year.

ibu1System AdministratorCommented:
If the kernel finds fatal corruption on the disk or if certain key IOs like journal writes start failing, the kernel may remount the filesystem as read-only. This is because the filesystem can no longer maintain write integrity under these conditions. Any such behavior will be thoroughly logged in /var/log/messages.

Should this happen, backup your recent data as this may be a symptom of an impending disk failure. Perform filesystem checks on the disk using e2fsck as soon as possible and use the -c option to enable badblock checking. The normal fsck may not detect all the errors and return clean. For example:

e2fsck -c /path to the disk
eg
e2fsck -c /dev/sda3
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
NozmokingAuthor Commented:
Thanks  - I'll run the disk check and post what I find in the messages log file from after the last restart.

-Noz
0
NozmokingAuthor Commented:
Log file attached -
Thanks,
-Noz
messages.txt
0
ravenplCommented:
Looks like /dev/sdc drive is failing - Do You have Your root filesystem on sdc?
0
NozmokingAuthor Commented:
I have to check and see if the root volume is part of the striped scsi array - I'm hoping that the person that put the thing together kept the root vol on its own physical drive. This appears to be data corruption from a power failure. Since I cannot unmount the root volume to run fsck I will try and run it from the FC5 rescue cd when I get  there.

Thanks,
-Noz
0
NozmokingAuthor Commented:
I have not used the rescue cd before - since it mounts the file system but does not provide a map for fsck to read, do I run it against the mount point the recscue cd creates in rw mode? Or, do I have to unmouint the root volume first? The rescue cd mounts it here:
/dev/VolGroup00/LogVol00 on /mnt/sysimage type ext3 (rw,data=ordered)

Any advice would be appreciated,
- Noz
 
0
NozmokingAuthor Commented:
I am unable to unmount the file system VolGroup00/LogVol00 after loading the FC5 rescue cd. I can only unmount/remount in rerad-only. How can I run fsck on the root volume? I would appreciate any input.

Thanks,
- Noz
0
ravenplCommented:
You can run fsck on read-only mounted filesystem
e2fsck -f -n /dev/VolGroup00/LogVol00
-f == force
-n == don't try repairing anything, change to -y to repair automatically, but then, don't remount it R/W! Reboot, or umount/mount.
0
NozmokingAuthor Commented:
Well...
Ran e2fsck in forced mode from FC5 recovery cd and it came back clean but there were still errors when running fsck against the root volume in read-only (nofix) mode from the terminal wioth the system running. I loaded the rescue cd again and ran e2fsck again with -c and it noted and fixed bad blocks butr now I get a kernel panic after loading the system volumes. It successfully mounts the two volumes in VolGroup00 then I get an error saying "Kernel panic - not syncing: Attempted to kill init!". Is there anything else I can do at this point or is it a dead horse?

Thanks,
- Noz
0
NozmokingAuthor Commented:
One more thought - since the root file system seems to have been damaged beyon repair at this point could I run an FC 5 upgrade and possible restore the file system to some level of functionality? This machine was task oriented so I have not really lost any data...
Thanks,
- Noz
0
NozmokingAuthor Commented:
Just to bring some resolution to this issue - can someone advise as to the easiest way to boot this server with either the fc5 rescue CD or another recovery CD (I have several CD images of recovery solution collections I've scrounged), configure the network connections, mount the bad volume and copy some "critical" data to another machine? If I could save some select data then I can go ahead and rebuild it from scratch. I  cannot load any of the current Fedora live CD images because the bugger only has 256MB of RAM, and being a Compaq 5500 we'd have to spend a few bucks to increase it. If anyone can point me to the best utility and procedure for doing that I can close this out.

Thanks!
- Noz
0
NozmokingAuthor Commented:
Just one final  follow-up; any suggestions for pulling some files off the root volume before I send the beast to the bone yard? I'd just like to load up a bare-bones OS, mount the old root volume and copy a few things over the network. Basically, I need to know what to load that would give me either secure or even insecure file transfer after starting the network interfaces and mounting the old volume, since the kernel on the old sys-vol will not start.

Thanks very much,

-Noz
0
Computer101Commented:
Forced accept.

Computer101
EE Admin
0
NozmokingAuthor Commented:
I was able to eliminate the kernel panic and restore the server to a bootable state by using the Fedora Core 5 installation CD #1 to perform an upgrade on the OS. Although this is not the most successful of repair methods, it replaced the damaged system files caused by the power failure and then running fsck with the sysvol mounted. The system is now usable again.

- Noz.
0
NozmokingAuthor Commented:
I was able to eliminate the kernel panic and restore the server to a bootable state by using the Fedora Core 5 installation CD #1 to perform an upgrade on the OS. Although this is not the most successful of repair methods, it replaced the damaged system files caused by the power failure and running fsck with the sysvol mounted.

The system is now usable again.

- Noz.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.