CentOS 5 forcefsck loop
Posted on 2012-03-12
A client's POS server was crashing intermittently, and as part of the diagnostic I ran fsck. Of course the problems were on the root drive, so we had to use forcefsck to make the system check on boot.
The problem now is that forcefsck on boot finds errors that need to be addressed manually (not automatically), but it's loading the FS in read-only.
End result is that the machine is stuck in a loop. It boots and runs fsck, fails out to read-only, and we can't delete the forcefsck command.
The only solution I've found online so far is to try booting via a CD (like damn small linux) and accessing the files from there - but the root file system is inaccessible when I boot from CD.
The server has 2 SSD drives running in RAID 1 with a hardware controller.