Solved

Redhat system unable to boot

Posted on 2014-02-20
51
1,380 Views
Last Modified: 2014-05-01
I have a redhat linux server working as a backup server. After my coworker worked on this system, for some reason it couldn't start. Please see the following output for diagnose.

......
Activating Logical volumes
   4 logical volume(s) in volume group "VolGroup00" now active.

Creating root device. Mounting root filesystem.
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
Setting up other filesystems.
Setting up new root fs
no fstab.sys, mounting internal defaults
Switching to new root and runing init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
/bin/sh: ro: No such file or directory
Kernel panic - not syncing: Attempted to kill init!


After these steps, the system just keep rebooting. Please advise what caused this failure and how to remedy it. Thank you experts!
pic1.JPG
photo-1.JPG
photo-3.JPG
0
Comment
Question by:Jason Yu
  • 26
  • 13
  • 11
  • +1
51 Comments
 
LVL 25

Expert Comment

by:Zephyr ICT
ID: 39876086
I think you will need to "rescue" the system using the installation media... When you start the installation media you should have an option to rescue the system.

What kind of work was done on this system?
0
 

Author Comment

by:Jason Yu
ID: 39876144
The problem is I don't know what he did on the system before failure. Also I only know it's a redhat linux, I don't know the version, which rescue disk should I use to recover?

Thanks.
0
 
LVL 25

Expert Comment

by:Zephyr ICT
ID: 39876156
Do you still have the original install media which was used to install the server? That should be enough to use for rescue...

There's more info on rescue from Red Hat here.

Though you might need to create a new initrd file ...
0
 

Author Comment

by:Jason Yu
ID: 39876323
thanks for reply, may I ask what this initrd file for?

could you list some occasions it got lost?

thanks.
0
 
LVL 25

Expert Comment

by:Zephyr ICT
ID: 39876350
initrd is part of the Linux boot process, stand for initial ramdisk, to load a temporary root file system into memory...

It might just be corrupt/broken, a new initrd needs to be created using rescue, this can happen when something went wrong with the disk config or maybe the controller
0
 
LVL 19

Expert Comment

by:jools
ID: 39876474
It looks like you have a raid setup, there is a message on the top of the screen with what looks like v0.25941 which at a guess (and I really mean guess) could be 5.x

If you dont know you can use any live distro type disk but you might have to manually start the raid and lvm. Having the rescue disk will do this for you and will save hassle.

When the system boots you can get more information from the kernel version in the grub screen and RHEL/Centos usually says something like "starting RHEL <version>"

If you get the disks mounted check the logs.
0
 
LVL 61

Expert Comment

by:gheist
ID: 39880464
It looks like your your fstab is messed up
EXT3 - i assume you have RHEL5
Get 1st CD orDVD of RHEL5 or CentOS5 and follow the onscreen menus to mount system from the disk, chroot and you can edit files on your disk (mounting will be problematic with broken fstab, so maybe you end up scanning LVM,  mounting / and checking fstab)
0
 

Author Comment

by:Jason Yu
ID: 39887335
I burned two cds and will use the linux rescue mode to recover.

I will update you guys once I have progress.
0
 

Author Comment

by:Jason Yu
ID: 39887411
I entered into rescue mode and ran "chroot /mnt/sysimage" command. it indeed gave me a shell. However, when I went into etc/ folder and want to use "ls -alt" command, it says couldn't find the command. Please advise.

Which file may be changed incorrectly? /etc/fstab?  

thanks.
0
 

Author Comment

by:Jason Yu
ID: 39887445
I couldn't run lots of standard linux commands, why?

Here are the pictures from the screen outputs.
photo-1.jpg
photo-2.jpg
photo-3.jpg
0
 
LVL 25

Expert Comment

by:Zephyr ICT
ID: 39887942
Did you try with "/sbin/fdisk" ?
0
 
LVL 61

Expert Comment

by:gheist
ID: 39888027
since fstab is damaged it mounts nothing
try mounting lvm devices without chroot looking for signs of /etc...
0
 

Author Comment

by:Jason Yu
ID: 39888124
To spravtek: I will try this command tomorrow, do I need run chroot before this command?

gheist:

You mean I can try to mount lvm without running chroot command?

thanks.
0
 
LVL 25

Expert Comment

by:Zephyr ICT
ID: 39888135
You should be able to Scan and mount volume groups yes ...

> lvm vgscan -v

Activate all available volume groups with something like this ...
> lvm vgchange -a y

List the logical volumes ...
> lvm lvs –all

with the volumes activated, you should be able to mount the volumes now ... If all worked without a problem that is

> mount /dev/volumegroup/logicalvolume  /mountingpoint
0
 

Author Comment

by:Jason Yu
ID: 39890581
I successfully mounted the /dev/VolGroup00/Logvol00 to /mnt/lv00.

And I was able to open /etc/fstab file and review the file, but i didn't see any changes has been made on it.

I commented out the last line for mounting an external drive and tried to reboot, but I got the same error. could you guys please help?

Any comments will be great apprecaited, thank you in advance.
0
 

Author Comment

by:Jason Yu
ID: 39890610
Here are the pictures before it reboots itself.

From picture, I can see a line "no fstab.sys, mounting internal defaults". Was this file lost?

What is the function of this file and could I copy it from another system? Please advise.

thank you.
0
 

Author Comment

by:Jason Yu
ID: 39890622
I searched online for the lost fstab.sys error and found two links, all of them rebuild the init kernel. Is this the solution?

1. http://troysunix.blogspot.com/2011/12/fixing-broken-initrd-in-linux.html
2. https://www.centos.org/forums/viewtopic.php?t=17504

With our filesystems mounted, we need 'chroot' to "/mnt/sysimage", back up
the existing "initrd" file, and finally create a new one with 'mkinitrd':

        sh-3.2# /usr/sbin/chroot /mnt/sysimage
        sh-3.2# /bin/ls /boot/initrd*
        /boot/initrd-2.6.18-238.el5.img
        sh-3.2# /bin/mv /boot/initrd-2.6.18-238.el5.img /boot/old-initrd-2.6.18-238.el5.img-old
        sh-3.2# /sbin/mkinitrd /boot/initrd-2.6.18-238.el5.img 2.6.18-238.el5



If this is the problem you're having and the old system isn't working anymore you'll probably need to boot up a rescue cd, mount/chroot the fs on the disk(s), update etc/modprobe.conf, rebuild initrd, then reboot. If you're not sure what module is needed you can look at the output of lsmod when up on the rescue cd.
0
 

Author Comment

by:Jason Yu
ID: 39891056
Here are the pictures, I forgot to attach them.
pic1.JPG
photo-2.JPG
0
 
LVL 25

Expert Comment

by:Zephyr ICT
ID: 39891106
Yes, that was my thought from the start ...
Though you might need to create a new initrd file ...
...

So I would definitely try that ... The first link you posted seems like a good howto ... Did you try that?
0
 

Author Comment

by:Jason Yu
ID: 39891232
What do you think about the cause of this failure?

Is it a hardware failure? This is an old server.
0
 
LVL 25

Expert Comment

by:Zephyr ICT
ID: 39891445
It can be hardware yes, sometimes this can be caused by a storage controller (RAID card) or something... But for that you'll need to investigate further I think.

Did you find out what it was that your co-worker did on the system!
0
 
LVL 61

Expert Comment

by:gheist
ID: 39891532
where it chokes on "ro" it starts running /etc/rc.sysinit
compare the file you have with healthy system.
0
 

Author Comment

by:Jason Yu
ID: 39892636
Thanks for replies. My coworker doesn't want to tell me the changes he has made on the system.

I will try to check rc.sysinit file first. If it doesn't work, I will go recreat that initrd file.

Thank you guys, I will update you once I make progress.
0
 
LVL 61

Expert Comment

by:gheist
ID: 39892865
recreate initrd is 10s...
0
 

Author Comment

by:Jason Yu
ID: 39893464
what does this mean?

"recreate initrd is 10s..."
0
What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 
LVL 61

Expert Comment

by:gheist
ID: 39894693
suggestion to try quickest thing first. (you can as well choose other boot option that has known good initrd)
0
 

Author Comment

by:Jason Yu
ID: 39896462
It's very strange I couldn't see any initid file under /root. After i ran "chroot /mnt/sysimage" , it didn't let me use "ls" command anymore. I had to quite root mode to use those basic commands.
photo-4.jpg
photo-3.jpg
0
 
LVL 61

Expert Comment

by:gheist
ID: 39896947
initrd's are in /Boot not in /Root
0
 
LVL 25

Expert Comment

by:Zephyr ICT
ID: 39897117
Sorry, was away for the day, keep us posted.
0
 
LVL 61

Expert Comment

by:gheist
ID: 39897128
show us fstab...
0
 

Author Comment

by:Jason Yu
ID: 39899275
Here is hte fstab file, it doesn't look abnormal. I commented out the last line because we didn't connect the external drive now.
photo.jpg
0
 

Author Comment

by:Jason Yu
ID: 39902016
I am sorry, it was a typo. I indeed search the initid file under /boot, please see the attached pictures for detail. If there is no initid files, what should I do? thanks.


"It's very strange I couldn't see any initid file under /root. After i ran "chroot /mnt/sysimage" , it didn't let me use "ls" command anymore. I had to quite root mode to use those basic commands."  /root is a typo.
0
 

Author Comment

by:Jason Yu
ID: 39905808
If I couldn't find any initrd file under /boot, what should I do? do I need copy this file from a healthy system with the same distro version?

Please help, thank you.
0
 
LVL 25

Expert Comment

by:Zephyr ICT
ID: 39905824
So you chroot into sysimage -- $ /usr/sbin/chroot /mnt/sysimage

Can you get any result from following? -- $ /bin/ls /boot/initrd*
0
 
LVL 61

Accepted Solution

by:
gheist earned 50 total points
ID: 39905934
/boot is separate filesystem usually and it is /dev/sda1 (or sda2) if you install without repartitioning
0
 

Author Comment

by:Jason Yu
ID: 39918539
do I need manuly mount this /boot partition by issuing command :

mount /dev/sda1 /boot
0
 

Author Comment

by:Jason Yu
ID: 39919152
I tried to run the command /bin/ls /boot/initrd, it says "no such file or directory". I noticed the /dev/sda1 was mounted on /mnt/sysimage/boot ( it is an CentOS system, that may be different than the standard redhat linux ).

Then I went to the /mnt/sysimage/boot folder and found the original initrd file "initrd-2.6.18-194.17.1.el5.centos.pulsxen.img". I mv this file with XXX.orig, and tried to run mkinitrd command to rebuild the mkinitrd. However, it couldn't find the "mkinitrd" command.

I went to the /mnt/sysimage/sbin folder to search this command, but still couldn't find it.

How could I find the "mkinitrd" command to rebuild my kernel. Please advise, thank you.
photo-1.jpg
photo-2.jpg
photo-3.jpg
photo-4.jpg
0
 
LVL 25

Expert Comment

by:Zephyr ICT
ID: 39919783
It could be in the /usr/bin/ directory ... Can you try use the command "locate mkinitrd" ...

Can maybe also try "rpm -qi mkinitrd"
0
 

Author Comment

by:Jason Yu
ID: 39919796
I couldn't use locate command to find it, in fact, the locate command was not configured. I used "find" command. but still couldn't find it.

I will try to use rpm -qi mkinitrd tomorrow, do I need network to run this rpm command.

Please advise, thank you.
0
 
LVL 25

Expert Comment

by:Zephyr ICT
ID: 39919811
No network access will not be needed ... It will search for the package on the system, you might need to download it and make it available, but in the first place we'll do a query (-q) for it ...

It's strange though, that it's not available ...
0
 

Author Comment

by:Jason Yu
ID: 39946627
I tried to use "rpm -qa mkinird" but it says package is not install, how could i install this package?

thanks.
0
 

Author Comment

by:Jason Yu
ID: 39954135
Any help for this? How could I install mkinird package? Would I install it from the CD-ROM?

thanks.
0
 
LVL 61

Expert Comment

by:gheist
ID: 39954164
initrd is a file containing initial filesystem and mkinitrd is on the cd usually in rescue system.
But if you dont mount all filesystems you cannot mkinitrd.
have you tried booting other kernel versions present on the system?
0
 

Author Comment

by:Jason Yu
ID: 39954236
It looks like I only have one initrd file under the /boot. please review the pictures I uploaded in my former posts.

How could I find the mkinitrd command to make a new initrd file?
0
 
LVL 61

Expert Comment

by:gheist
ID: 39954477
It is close to impossible to repair xen boot chain.
Can you check command line of boot that it uses initrd*.img.orig ?
0
 

Author Comment

by:Jason Yu
ID: 39954632
I renamed that .img file to .img.orig, some experts above suggested me to backup the original kernel file and use mkinitrd command to make a new one. Did I understand the solution correctly?

thanks.
0
 
LVL 61

Expert Comment

by:gheist
ID: 39954741
xen is not really centos. we need to take different approach. btw rhel6 does not use xen anymore...

my /etc/grub.conf has following line:

kernel=/vmlinuz-...... ro root=/dev/sd....

maybe this ro pops up somewhere?
0
 

Author Comment

by:Jason Yu
ID: 39967622
HI, Gheist:

First, thank you for your reply. However, I have some difficulty in understanding your method. Are you asking my to change the kernel file?

By following the above posts, I have already made some progress by finding the problematic kernel file. The next step I need do is to use mkinitrd command to recreate the initrd file. The problem is that I don't have this command installed.
0
 
LVL 61

Expert Comment

by:gheist
ID: 39967669
It just takes too long.
What is the boot command line you use?
Does it boot in single mode?
Without them halw-way-into-xen somehow does not help at all.
0
 

Author Comment

by:Jason Yu
ID: 39990181
I don't think I can boot it into single mode.

I already renamed the initrd*.img to initrd*.img.orig.

How could I check which command line of boot uses this file? Any instructions on how to do it.

This server is down for nearly a month, If i couldn't bring it back, I have to copy all the scripts files out and reinstall, please help.

"It is close to impossible to repair xen boot chain.
Can you check command line of boot that it uses initrd*.img.orig ? "

What is teh "xen boot chain", thanks.
0
 
LVL 61

Expert Comment

by:gheist
ID: 39990292
Boot from install CD (1st DVD 3rd boot option, centos is fine)
Mount system and chroot to /mnt/sysimage, so you are in your system
Copy needed files out and reinstall. (so you use xen or not? if not dont install it)
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Suggested Solutions

SSH (Secure Shell) - Tips and Tricks As you all know SSH(Secure Shell) is a network protocol, which we use to access/transfer files securely between two networked devices. SSH was actually designed as a replacement for insecure protocols that sen…
The purpose of this article is to demonstrate how we can use conditional statements using Python.
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now