CentOS 5 forcefsck loop

A client's POS server was crashing intermittently, and as part of the diagnostic I ran fsck. Of course the problems were on the root drive, so we had to use forcefsck to make the system check on boot.

The problem now is that forcefsck on boot finds errors that need to be addressed manually (not automatically), but it's loading the FS in read-only.

End result is that the machine is stuck in a loop. It boots and runs fsck, fails out to read-only, and we can't delete the forcefsck command.

The only solution I've found online so far is to try booting via a CD (like damn small linux) and accessing the files from there - but the root file system is inaccessible when I boot from CD.

The server has 2 SSD drives running in RAID 1 with a hardware controller.
RKFcomputersAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

farzanjCommented:
You can do linux rescue.  You can do it by using CentOS CD1 or DVD.  At the boot prompt

boot:

Type
linux rescue askmethod

this enables options to mount some volumes over the network or if you don't need network availability option, just type

linux rescue

And hit enter.

It then mounts your actual drive under /mnt/sysimage.  So now your /root is /mnt/sysimage/root
And you /boot is /mnt/sysimage/boot.

You can do fsck as well if you enable chroot jail.  This you can do by issuing command

cd /mnt/sysimage
chroot /mnt/sysimage
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
RKFcomputersAuthor Commented:
@Farzanj Thanks! I'll try that today and see how it works out.
0
farzanjCommented:
Good luck!!  And for your reference, here is the documentation of various modes to boot, including rescue.  You can also boot at run level emergency, or single user or run level 1, all of these are different.

http://www.centos.org/docs/5/html/Installation_Guide-en-US/ch-rescuemode.html
0
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

RKFcomputersAuthor Commented:
I just received the server from the client this evening, and it's actually CentOS 6, not 5.

When I boot from a CentOS 6.1 DVD 1 I get the following options:
Install or upgrade...
Install system with basic video
Rescue installed system
Boot from local drive
Memory test

I chose Rescue Installed System from the menu, but it started an installation wizard that didn't resemble your instructions so I bailed. Is there a trick to get the command line boot like what you're describing?
0
farzanjCommented:
You chose the right thing.  It will start like a wizard.  Will ask you if you have network options like NFS, HTTP or FTP. It takes some time and then it would tell you where it mounted your current OS.  One thing that it would not do is trying to partition or trying to install software.  Other than these options, you should not worry.
0
RKFcomputersAuthor Commented:
Great answer, with prompt follow up
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.