Link to home
Start Free TrialLog in
Avatar of Jason Yu
Jason YuFlag for United States of America

asked on

Redhat system unable to boot

I have a redhat linux server working as a backup server. After my coworker worked on this system, for some reason it couldn't start. Please see the following output for diagnose.

......
Activating Logical volumes
   4 logical volume(s) in volume group "VolGroup00" now active.

Creating root device. Mounting root filesystem.
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
Setting up other filesystems.
Setting up new root fs
no fstab.sys, mounting internal defaults
Switching to new root and runing init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
/bin/sh: ro: No such file or directory
Kernel panic - not syncing: Attempted to kill init!


After these steps, the system just keep rebooting. Please advise what caused this failure and how to remedy it. Thank you experts!
pic1.JPG
photo-1.JPG
photo-3.JPG
Avatar of Zephyr ICT
Zephyr ICT
Flag of Belgium image

I think you will need to "rescue" the system using the installation media... When you start the installation media you should have an option to rescue the system.

What kind of work was done on this system?
Avatar of Jason Yu

ASKER

The problem is I don't know what he did on the system before failure. Also I only know it's a redhat linux, I don't know the version, which rescue disk should I use to recover?

Thanks.
Do you still have the original install media which was used to install the server? That should be enough to use for rescue...

There's more info on rescue from Red Hat here.

Though you might need to create a new initrd file ...
thanks for reply, may I ask what this initrd file for?

could you list some occasions it got lost?

thanks.
initrd is part of the Linux boot process, stand for initial ramdisk, to load a temporary root file system into memory...

It might just be corrupt/broken, a new initrd needs to be created using rescue, this can happen when something went wrong with the disk config or maybe the controller
Avatar of Julian Parker
It looks like you have a raid setup, there is a message on the top of the screen with what looks like v0.25941 which at a guess (and I really mean guess) could be 5.x

If you dont know you can use any live distro type disk but you might have to manually start the raid and lvm. Having the rescue disk will do this for you and will save hassle.

When the system boots you can get more information from the kernel version in the grub screen and RHEL/Centos usually says something like "starting RHEL <version>"

If you get the disks mounted check the logs.
It looks like your your fstab is messed up
EXT3 - i assume you have RHEL5
Get 1st CD orDVD of RHEL5 or CentOS5 and follow the onscreen menus to mount system from the disk, chroot and you can edit files on your disk (mounting will be problematic with broken fstab, so maybe you end up scanning LVM,  mounting / and checking fstab)
I burned two cds and will use the linux rescue mode to recover.

I will update you guys once I have progress.
I entered into rescue mode and ran "chroot /mnt/sysimage" command. it indeed gave me a shell. However, when I went into etc/ folder and want to use "ls -alt" command, it says couldn't find the command. Please advise.

Which file may be changed incorrectly? /etc/fstab?  

thanks.
I couldn't run lots of standard linux commands, why?

Here are the pictures from the screen outputs.
photo-1.jpg
photo-2.jpg
photo-3.jpg
Did you try with "/sbin/fdisk" ?
since fstab is damaged it mounts nothing
try mounting lvm devices without chroot looking for signs of /etc...
To spravtek: I will try this command tomorrow, do I need run chroot before this command?

gheist:

You mean I can try to mount lvm without running chroot command?

thanks.
You should be able to Scan and mount volume groups yes ...

> lvm vgscan -v

Activate all available volume groups with something like this ...
> lvm vgchange -a y

List the logical volumes ...
> lvm lvs –all

with the volumes activated, you should be able to mount the volumes now ... If all worked without a problem that is

> mount /dev/volumegroup/logicalvolume  /mountingpoint
I successfully mounted the /dev/VolGroup00/Logvol00 to /mnt/lv00.

And I was able to open /etc/fstab file and review the file, but i didn't see any changes has been made on it.

I commented out the last line for mounting an external drive and tried to reboot, but I got the same error. could you guys please help?

Any comments will be great apprecaited, thank you in advance.
Here are the pictures before it reboots itself.

From picture, I can see a line "no fstab.sys, mounting internal defaults". Was this file lost?

What is the function of this file and could I copy it from another system? Please advise.

thank you.
I searched online for the lost fstab.sys error and found two links, all of them rebuild the init kernel. Is this the solution?

1. http://troysunix.blogspot.com/2011/12/fixing-broken-initrd-in-linux.html
2. https://www.centos.org/forums/viewtopic.php?t=17504

With our filesystems mounted, we need 'chroot' to "/mnt/sysimage", back up
the existing "initrd" file, and finally create a new one with 'mkinitrd':

        sh-3.2# /usr/sbin/chroot /mnt/sysimage
        sh-3.2# /bin/ls /boot/initrd*
        /boot/initrd-2.6.18-238.el5.img
        sh-3.2# /bin/mv /boot/initrd-2.6.18-238.el5.img /boot/old-initrd-2.6.18-238.el5.img-old
        sh-3.2# /sbin/mkinitrd /boot/initrd-2.6.18-238.el5.img 2.6.18-238.el5



If this is the problem you're having and the old system isn't working anymore you'll probably need to boot up a rescue cd, mount/chroot the fs on the disk(s), update etc/modprobe.conf, rebuild initrd, then reboot. If you're not sure what module is needed you can look at the output of lsmod when up on the rescue cd.
Here are the pictures, I forgot to attach them.
pic1.JPG
photo-2.JPG
Yes, that was my thought from the start ...
Though you might need to create a new initrd file ...
...

So I would definitely try that ... The first link you posted seems like a good howto ... Did you try that?
What do you think about the cause of this failure?

Is it a hardware failure? This is an old server.
It can be hardware yes, sometimes this can be caused by a storage controller (RAID card) or something... But for that you'll need to investigate further I think.

Did you find out what it was that your co-worker did on the system!
where it chokes on "ro" it starts running /etc/rc.sysinit
compare the file you have with healthy system.
Thanks for replies. My coworker doesn't want to tell me the changes he has made on the system.

I will try to check rc.sysinit file first. If it doesn't work, I will go recreat that initrd file.

Thank you guys, I will update you once I make progress.
recreate initrd is 10s...
what does this mean?

"recreate initrd is 10s..."
suggestion to try quickest thing first. (you can as well choose other boot option that has known good initrd)
It's very strange I couldn't see any initid file under /root. After i ran "chroot /mnt/sysimage" , it didn't let me use "ls" command anymore. I had to quite root mode to use those basic commands.
photo-4.jpg
photo-3.jpg
initrd's are in /Boot not in /Root
Sorry, was away for the day, keep us posted.
show us fstab...
Here is hte fstab file, it doesn't look abnormal. I commented out the last line because we didn't connect the external drive now.
photo.jpg
I am sorry, it was a typo. I indeed search the initid file under /boot, please see the attached pictures for detail. If there is no initid files, what should I do? thanks.


"It's very strange I couldn't see any initid file under /root. After i ran "chroot /mnt/sysimage" , it didn't let me use "ls" command anymore. I had to quite root mode to use those basic commands."  /root is a typo.
If I couldn't find any initrd file under /boot, what should I do? do I need copy this file from a healthy system with the same distro version?

Please help, thank you.
So you chroot into sysimage -- $ /usr/sbin/chroot /mnt/sysimage

Can you get any result from following? -- $ /bin/ls /boot/initrd*
ASKER CERTIFIED SOLUTION
Avatar of gheist
gheist
Flag of Belgium image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
do I need manuly mount this /boot partition by issuing command :

mount /dev/sda1 /boot
I tried to run the command /bin/ls /boot/initrd, it says "no such file or directory". I noticed the /dev/sda1 was mounted on /mnt/sysimage/boot ( it is an CentOS system, that may be different than the standard redhat linux ).

Then I went to the /mnt/sysimage/boot folder and found the original initrd file "initrd-2.6.18-194.17.1.el5.centos.pulsxen.img". I mv this file with XXX.orig, and tried to run mkinitrd command to rebuild the mkinitrd. However, it couldn't find the "mkinitrd" command.

I went to the /mnt/sysimage/sbin folder to search this command, but still couldn't find it.

How could I find the "mkinitrd" command to rebuild my kernel. Please advise, thank you.
photo-1.jpg
photo-2.jpg
photo-3.jpg
photo-4.jpg
It could be in the /usr/bin/ directory ... Can you try use the command "locate mkinitrd" ...

Can maybe also try "rpm -qi mkinitrd"
I couldn't use locate command to find it, in fact, the locate command was not configured. I used "find" command. but still couldn't find it.

I will try to use rpm -qi mkinitrd tomorrow, do I need network to run this rpm command.

Please advise, thank you.
No network access will not be needed ... It will search for the package on the system, you might need to download it and make it available, but in the first place we'll do a query (-q) for it ...

It's strange though, that it's not available ...
I tried to use "rpm -qa mkinird" but it says package is not install, how could i install this package?

thanks.
Any help for this? How could I install mkinird package? Would I install it from the CD-ROM?

thanks.
initrd is a file containing initial filesystem and mkinitrd is on the cd usually in rescue system.
But if you dont mount all filesystems you cannot mkinitrd.
have you tried booting other kernel versions present on the system?
It looks like I only have one initrd file under the /boot. please review the pictures I uploaded in my former posts.

How could I find the mkinitrd command to make a new initrd file?
It is close to impossible to repair xen boot chain.
Can you check command line of boot that it uses initrd*.img.orig ?
I renamed that .img file to .img.orig, some experts above suggested me to backup the original kernel file and use mkinitrd command to make a new one. Did I understand the solution correctly?

thanks.
xen is not really centos. we need to take different approach. btw rhel6 does not use xen anymore...

my /etc/grub.conf has following line:

kernel=/vmlinuz-...... ro root=/dev/sd....

maybe this ro pops up somewhere?
HI, Gheist:

First, thank you for your reply. However, I have some difficulty in understanding your method. Are you asking my to change the kernel file?

By following the above posts, I have already made some progress by finding the problematic kernel file. The next step I need do is to use mkinitrd command to recreate the initrd file. The problem is that I don't have this command installed.
It just takes too long.
What is the boot command line you use?
Does it boot in single mode?
Without them halw-way-into-xen somehow does not help at all.
I don't think I can boot it into single mode.

I already renamed the initrd*.img to initrd*.img.orig.

How could I check which command line of boot uses this file? Any instructions on how to do it.

This server is down for nearly a month, If i couldn't bring it back, I have to copy all the scripts files out and reinstall, please help.

"It is close to impossible to repair xen boot chain.
Can you check command line of boot that it uses initrd*.img.orig ? "

What is teh "xen boot chain", thanks.
Boot from install CD (1st DVD 3rd boot option, centos is fine)
Mount system and chroot to /mnt/sysimage, so you are in your system
Copy needed files out and reinstall. (so you use xen or not? if not dont install it)