Jason Yu
asked on
Redhat system unable to boot
I have a redhat linux server working as a backup server. After my coworker worked on this system, for some reason it couldn't start. Please see the following output for diagnose.
......
Activating Logical volumes
4 logical volume(s) in volume group "VolGroup00" now active.
Creating root device. Mounting root filesystem.
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
Setting up other filesystems.
Setting up new root fs
no fstab.sys, mounting internal defaults
Switching to new root and runing init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
/bin/sh: ro: No such file or directory
Kernel panic - not syncing: Attempted to kill init!
After these steps, the system just keep rebooting. Please advise what caused this failure and how to remedy it. Thank you experts!
pic1.JPG
photo-1.JPG
photo-3.JPG
......
Activating Logical volumes
4 logical volume(s) in volume group "VolGroup00" now active.
Creating root device. Mounting root filesystem.
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
Setting up other filesystems.
Setting up new root fs
no fstab.sys, mounting internal defaults
Switching to new root and runing init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
/bin/sh: ro: No such file or directory
Kernel panic - not syncing: Attempted to kill init!
After these steps, the system just keep rebooting. Please advise what caused this failure and how to remedy it. Thank you experts!
pic1.JPG
photo-1.JPG
photo-3.JPG
ASKER
The problem is I don't know what he did on the system before failure. Also I only know it's a redhat linux, I don't know the version, which rescue disk should I use to recover?
Thanks.
Thanks.
Do you still have the original install media which was used to install the server? That should be enough to use for rescue...
There's more info on rescue from Red Hat here.
Though you might need to create a new initrd file ...
There's more info on rescue from Red Hat here.
Though you might need to create a new initrd file ...
ASKER
thanks for reply, may I ask what this initrd file for?
could you list some occasions it got lost?
thanks.
could you list some occasions it got lost?
thanks.
initrd is part of the Linux boot process, stand for initial ramdisk, to load a temporary root file system into memory...
It might just be corrupt/broken, a new initrd needs to be created using rescue, this can happen when something went wrong with the disk config or maybe the controller
It might just be corrupt/broken, a new initrd needs to be created using rescue, this can happen when something went wrong with the disk config or maybe the controller
It looks like you have a raid setup, there is a message on the top of the screen with what looks like v0.25941 which at a guess (and I really mean guess) could be 5.x
If you dont know you can use any live distro type disk but you might have to manually start the raid and lvm. Having the rescue disk will do this for you and will save hassle.
When the system boots you can get more information from the kernel version in the grub screen and RHEL/Centos usually says something like "starting RHEL <version>"
If you get the disks mounted check the logs.
If you dont know you can use any live distro type disk but you might have to manually start the raid and lvm. Having the rescue disk will do this for you and will save hassle.
When the system boots you can get more information from the kernel version in the grub screen and RHEL/Centos usually says something like "starting RHEL <version>"
If you get the disks mounted check the logs.
It looks like your your fstab is messed up
EXT3 - i assume you have RHEL5
Get 1st CD orDVD of RHEL5 or CentOS5 and follow the onscreen menus to mount system from the disk, chroot and you can edit files on your disk (mounting will be problematic with broken fstab, so maybe you end up scanning LVM, mounting / and checking fstab)
EXT3 - i assume you have RHEL5
Get 1st CD orDVD of RHEL5 or CentOS5 and follow the onscreen menus to mount system from the disk, chroot and you can edit files on your disk (mounting will be problematic with broken fstab, so maybe you end up scanning LVM, mounting / and checking fstab)
ASKER
I burned two cds and will use the linux rescue mode to recover.
I will update you guys once I have progress.
I will update you guys once I have progress.
ASKER
I entered into rescue mode and ran "chroot /mnt/sysimage" command. it indeed gave me a shell. However, when I went into etc/ folder and want to use "ls -alt" command, it says couldn't find the command. Please advise.
Which file may be changed incorrectly? /etc/fstab?
thanks.
Which file may be changed incorrectly? /etc/fstab?
thanks.
ASKER
I couldn't run lots of standard linux commands, why?
Here are the pictures from the screen outputs.
photo-1.jpg
photo-2.jpg
photo-3.jpg
Here are the pictures from the screen outputs.
photo-1.jpg
photo-2.jpg
photo-3.jpg
Did you try with "/sbin/fdisk" ?
since fstab is damaged it mounts nothing
try mounting lvm devices without chroot looking for signs of /etc...
try mounting lvm devices without chroot looking for signs of /etc...
ASKER
To spravtek: I will try this command tomorrow, do I need run chroot before this command?
gheist:
You mean I can try to mount lvm without running chroot command?
thanks.
gheist:
You mean I can try to mount lvm without running chroot command?
thanks.
You should be able to Scan and mount volume groups yes ...
> lvm vgscan -v
Activate all available volume groups with something like this ...
> lvm vgchange -a y
List the logical volumes ...
> lvm lvs –all
with the volumes activated, you should be able to mount the volumes now ... If all worked without a problem that is
> mount /dev/volumegroup/logicalvo lume /mountingpoint
> lvm vgscan -v
Activate all available volume groups with something like this ...
> lvm vgchange -a y
List the logical volumes ...
> lvm lvs –all
with the volumes activated, you should be able to mount the volumes now ... If all worked without a problem that is
> mount /dev/volumegroup/logicalvo
ASKER
I successfully mounted the /dev/VolGroup00/Logvol00 to /mnt/lv00.
And I was able to open /etc/fstab file and review the file, but i didn't see any changes has been made on it.
I commented out the last line for mounting an external drive and tried to reboot, but I got the same error. could you guys please help?
Any comments will be great apprecaited, thank you in advance.
And I was able to open /etc/fstab file and review the file, but i didn't see any changes has been made on it.
I commented out the last line for mounting an external drive and tried to reboot, but I got the same error. could you guys please help?
Any comments will be great apprecaited, thank you in advance.
ASKER
Here are the pictures before it reboots itself.
From picture, I can see a line "no fstab.sys, mounting internal defaults". Was this file lost?
What is the function of this file and could I copy it from another system? Please advise.
thank you.
From picture, I can see a line "no fstab.sys, mounting internal defaults". Was this file lost?
What is the function of this file and could I copy it from another system? Please advise.
thank you.
ASKER
I searched online for the lost fstab.sys error and found two links, all of them rebuild the init kernel. Is this the solution?
1. http://troysunix.blogspot.com/2011/12/fixing-broken-initrd-in-linux.html
2. https://www.centos.org/forums/viewtopic.php?t=17504
With our filesystems mounted, we need 'chroot' to "/mnt/sysimage", back up
the existing "initrd" file, and finally create a new one with 'mkinitrd':
sh-3.2# /usr/sbin/chroot /mnt/sysimage
sh-3.2# /bin/ls /boot/initrd*
/boot/initrd-2.6.18-238.el 5.img
sh-3.2# /bin/mv /boot/initrd-2.6.18-238.el 5.img /boot/old-initrd-2.6.18-23 8.el5.img- old
sh-3.2# /sbin/mkinitrd /boot/initrd-2.6.18-238.el 5.img 2.6.18-238.el5
If this is the problem you're having and the old system isn't working anymore you'll probably need to boot up a rescue cd, mount/chroot the fs on the disk(s), update etc/modprobe.conf, rebuild initrd, then reboot. If you're not sure what module is needed you can look at the output of lsmod when up on the rescue cd.
1. http://troysunix.blogspot.com/2011/12/fixing-broken-initrd-in-linux.html
2. https://www.centos.org/forums/viewtopic.php?t=17504
With our filesystems mounted, we need 'chroot' to "/mnt/sysimage", back up
the existing "initrd" file, and finally create a new one with 'mkinitrd':
sh-3.2# /usr/sbin/chroot /mnt/sysimage
sh-3.2# /bin/ls /boot/initrd*
/boot/initrd-2.6.18-238.el
sh-3.2# /bin/mv /boot/initrd-2.6.18-238.el
sh-3.2# /sbin/mkinitrd /boot/initrd-2.6.18-238.el
If this is the problem you're having and the old system isn't working anymore you'll probably need to boot up a rescue cd, mount/chroot the fs on the disk(s), update etc/modprobe.conf, rebuild initrd, then reboot. If you're not sure what module is needed you can look at the output of lsmod when up on the rescue cd.
ASKER
Yes, that was my thought from the start ...
So I would definitely try that ... The first link you posted seems like a good howto ... Did you try that?
Though you might need to create a new initrd file ......
So I would definitely try that ... The first link you posted seems like a good howto ... Did you try that?
ASKER
What do you think about the cause of this failure?
Is it a hardware failure? This is an old server.
Is it a hardware failure? This is an old server.
It can be hardware yes, sometimes this can be caused by a storage controller (RAID card) or something... But for that you'll need to investigate further I think.
Did you find out what it was that your co-worker did on the system!
Did you find out what it was that your co-worker did on the system!
where it chokes on "ro" it starts running /etc/rc.sysinit
compare the file you have with healthy system.
compare the file you have with healthy system.
ASKER
Thanks for replies. My coworker doesn't want to tell me the changes he has made on the system.
I will try to check rc.sysinit file first. If it doesn't work, I will go recreat that initrd file.
Thank you guys, I will update you once I make progress.
I will try to check rc.sysinit file first. If it doesn't work, I will go recreat that initrd file.
Thank you guys, I will update you once I make progress.
recreate initrd is 10s...
ASKER
what does this mean?
"recreate initrd is 10s..."
"recreate initrd is 10s..."
suggestion to try quickest thing first. (you can as well choose other boot option that has known good initrd)
ASKER
It's very strange I couldn't see any initid file under /root. After i ran "chroot /mnt/sysimage" , it didn't let me use "ls" command anymore. I had to quite root mode to use those basic commands.
photo-4.jpg
photo-3.jpg
photo-4.jpg
photo-3.jpg
initrd's are in /Boot not in /Root
Sorry, was away for the day, keep us posted.
show us fstab...
ASKER
Here is hte fstab file, it doesn't look abnormal. I commented out the last line because we didn't connect the external drive now.
photo.jpg
photo.jpg
ASKER
I am sorry, it was a typo. I indeed search the initid file under /boot, please see the attached pictures for detail. If there is no initid files, what should I do? thanks.
"It's very strange I couldn't see any initid file under /root. After i ran "chroot /mnt/sysimage" , it didn't let me use "ls" command anymore. I had to quite root mode to use those basic commands." /root is a typo.
"It's very strange I couldn't see any initid file under /root. After i ran "chroot /mnt/sysimage" , it didn't let me use "ls" command anymore. I had to quite root mode to use those basic commands." /root is a typo.
ASKER
If I couldn't find any initrd file under /boot, what should I do? do I need copy this file from a healthy system with the same distro version?
Please help, thank you.
Please help, thank you.
So you chroot into sysimage -- $ /usr/sbin/chroot /mnt/sysimage
Can you get any result from following? -- $ /bin/ls /boot/initrd*
Can you get any result from following? -- $ /bin/ls /boot/initrd*
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
do I need manuly mount this /boot partition by issuing command :
mount /dev/sda1 /boot
mount /dev/sda1 /boot
ASKER
I tried to run the command /bin/ls /boot/initrd, it says "no such file or directory". I noticed the /dev/sda1 was mounted on /mnt/sysimage/boot ( it is an CentOS system, that may be different than the standard redhat linux ).
Then I went to the /mnt/sysimage/boot folder and found the original initrd file "initrd-2.6.18-194.17.1.el 5.centos.p ulsxen.img ". I mv this file with XXX.orig, and tried to run mkinitrd command to rebuild the mkinitrd. However, it couldn't find the "mkinitrd" command.
I went to the /mnt/sysimage/sbin folder to search this command, but still couldn't find it.
How could I find the "mkinitrd" command to rebuild my kernel. Please advise, thank you.
photo-1.jpg
photo-2.jpg
photo-3.jpg
photo-4.jpg
Then I went to the /mnt/sysimage/boot folder and found the original initrd file "initrd-2.6.18-194.17.1.el
I went to the /mnt/sysimage/sbin folder to search this command, but still couldn't find it.
How could I find the "mkinitrd" command to rebuild my kernel. Please advise, thank you.
photo-1.jpg
photo-2.jpg
photo-3.jpg
photo-4.jpg
It could be in the /usr/bin/ directory ... Can you try use the command "locate mkinitrd" ...
Can maybe also try "rpm -qi mkinitrd"
Can maybe also try "rpm -qi mkinitrd"
ASKER
I couldn't use locate command to find it, in fact, the locate command was not configured. I used "find" command. but still couldn't find it.
I will try to use rpm -qi mkinitrd tomorrow, do I need network to run this rpm command.
Please advise, thank you.
I will try to use rpm -qi mkinitrd tomorrow, do I need network to run this rpm command.
Please advise, thank you.
No network access will not be needed ... It will search for the package on the system, you might need to download it and make it available, but in the first place we'll do a query (-q) for it ...
It's strange though, that it's not available ...
It's strange though, that it's not available ...
ASKER
I tried to use "rpm -qa mkinird" but it says package is not install, how could i install this package?
thanks.
thanks.
ASKER
Any help for this? How could I install mkinird package? Would I install it from the CD-ROM?
thanks.
thanks.
initrd is a file containing initial filesystem and mkinitrd is on the cd usually in rescue system.
But if you dont mount all filesystems you cannot mkinitrd.
have you tried booting other kernel versions present on the system?
But if you dont mount all filesystems you cannot mkinitrd.
have you tried booting other kernel versions present on the system?
ASKER
It looks like I only have one initrd file under the /boot. please review the pictures I uploaded in my former posts.
How could I find the mkinitrd command to make a new initrd file?
How could I find the mkinitrd command to make a new initrd file?
It is close to impossible to repair xen boot chain.
Can you check command line of boot that it uses initrd*.img.orig ?
Can you check command line of boot that it uses initrd*.img.orig ?
ASKER
I renamed that .img file to .img.orig, some experts above suggested me to backup the original kernel file and use mkinitrd command to make a new one. Did I understand the solution correctly?
thanks.
thanks.
xen is not really centos. we need to take different approach. btw rhel6 does not use xen anymore...
my /etc/grub.conf has following line:
kernel=/vmlinuz-...... ro root=/dev/sd....
maybe this ro pops up somewhere?
my /etc/grub.conf has following line:
kernel=/vmlinuz-...... ro root=/dev/sd....
maybe this ro pops up somewhere?
ASKER
HI, Gheist:
First, thank you for your reply. However, I have some difficulty in understanding your method. Are you asking my to change the kernel file?
By following the above posts, I have already made some progress by finding the problematic kernel file. The next step I need do is to use mkinitrd command to recreate the initrd file. The problem is that I don't have this command installed.
First, thank you for your reply. However, I have some difficulty in understanding your method. Are you asking my to change the kernel file?
By following the above posts, I have already made some progress by finding the problematic kernel file. The next step I need do is to use mkinitrd command to recreate the initrd file. The problem is that I don't have this command installed.
It just takes too long.
What is the boot command line you use?
Does it boot in single mode?
Without them halw-way-into-xen somehow does not help at all.
What is the boot command line you use?
Does it boot in single mode?
Without them halw-way-into-xen somehow does not help at all.
ASKER
I don't think I can boot it into single mode.
I already renamed the initrd*.img to initrd*.img.orig.
How could I check which command line of boot uses this file? Any instructions on how to do it.
This server is down for nearly a month, If i couldn't bring it back, I have to copy all the scripts files out and reinstall, please help.
"It is close to impossible to repair xen boot chain.
Can you check command line of boot that it uses initrd*.img.orig ? "
What is teh "xen boot chain", thanks.
I already renamed the initrd*.img to initrd*.img.orig.
How could I check which command line of boot uses this file? Any instructions on how to do it.
This server is down for nearly a month, If i couldn't bring it back, I have to copy all the scripts files out and reinstall, please help.
"It is close to impossible to repair xen boot chain.
Can you check command line of boot that it uses initrd*.img.orig ? "
What is teh "xen boot chain", thanks.
Boot from install CD (1st DVD 3rd boot option, centos is fine)
Mount system and chroot to /mnt/sysimage, so you are in your system
Copy needed files out and reinstall. (so you use xen or not? if not dont install it)
Mount system and chroot to /mnt/sysimage, so you are in your system
Copy needed files out and reinstall. (so you use xen or not? if not dont install it)
What kind of work was done on this system?