We have a server that is running Red Hat Enterprise Linux. Recently the server became unresponsive and had to be rebooted manually, by holding down the power button until it shut off.
We were able to get the server back up and running, but after 24 hours it became unresponsive again and had to be manually rebooted. This time it took several attempts to get the server up and running and now the file system is in RO mode and we can not do anything with it.
We do not have much experience with Linux and inherited this system from the previous IT members. This system runs an Oracle instance with several schemas on it. The Oracle databases live on a LUN on our SAN unit, an EMC Clarion CX300.
We do have an rsync that was running and it appears to have a copy of most of the file system that was on the local disks for the linux install. We also have the original install disks for Red Hat and also have a Linux Live CD, KNOPTIX I believe.
Is there anyway to repair or restore the file system from the rsync copy? How can we determine if it is a hardware failure that is causing the issues? I am leaning toward a failing hard drive, but do not know enough about linux to troubleshoot.
We do have a second Linux server that i believe is running the same version of Red Hat. We could install Oracle on that server and then attempt to attach the LUN to the new server and bring the databases online that way, but would prefer to get the original server online if possible.