Solved

Kernel Panic after fsck

Posted on 2008-10-11
18
2,889 Views
Last Modified: 2012-05-05
I've got a machine running CentOS 5.  There are 2 hard drives in the machine, they are running on RAID 1 so I'm mirroring everything so that if one fails I have a backup on the other drive.  I'm running Apache, PHP, MySQL and some FTP services on the machine.

I just installed a 1TB disk and I had to move the other 2 disks from sdb to sda because the computer wouldn't start with the 1TB disk on sda even though I tried to change the grub.conf file to make it use hd1,0 instead of hd0,0.  That worked fine.  So then I had 2 disks on sda (sda1 and sda2) and one on sdb.

I wanted to have only one partition on sdb so I used fdisk and deleted sdb1 and sdb2.  I created a new partition set to primary and number 1 with default block settings.  I pressed T and selected 83 (Linux) for ext2.  Then I pressed W to write the changes and fdisk told me I had to restart because sdb was still in use by the system.  I didn't understand that because sdb was not in use by anything, I mean, the 2 hard drives running /boot where on sda.

I restarted and the machine didn't answer ping so I went down to the engine room and saw that there was an error.  It said that there where some bad blocks (or something like that) on /dev/VolGroup00/Log00 and that I should run fsck manually to fix it.  I ran fsck -y and it restarted the computer.

Now, when it finished starting up it had a new error with "Kernel Panic" :-/

It said:
no fstab.sys, mounting internal defaults
Switching to new root and running init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
audit(1223665793.629:2) enforcing=1 old_enforcing=0 ...
libsepol.policydb_read: policydb magic number ********* does not match expected magic number ********* or *********
libsepol.policydb_from_image: policy image is invalid
security: policydb macig number ******** does not match expected magic number **********
Unable to load SELinux Policy. Machine is in enforcing mode. Halting now.
Kernel panic - not syncing: Attempted to kill init!


So my question is; What is wrong?  How can I fix it without destroying anything on the machine?  Can I somehow run the Live CD and fix this there?
0
Comment
Question by:trymbill
  • 5
  • 4
  • 3
  • +2
18 Comments
 
LVL 19

Expert Comment

by:http:// thevpn.guru
ID: 22694351
Well at it's simplest your linux server is searching for your partitions at sda and they are at sdb
0
 
LVL 23

Expert Comment

by:Mysidia
ID: 22695014
When you are booting append the following option:

selinux=0  

To temporarily disable selinux.


You may need to get in single user mode to change /etc/fstab

to do so  append

selinux=0  init=/bin/sh


to the kernel boot options.
0
 
LVL 23

Expert Comment

by:Mysidia
ID: 22695035
to append kernel options when booting with lilo, you can normally at the boot prompt type:

boot: linux  selinux=0  init=/bin/sh
pressing tab at 'boot:'    sometimes gives a list of options.

With grub it is easier, use the menu keys to edit the  kernel command line

Or choose your kernel and hit 'a'  for append,  add the additional options
press enter...  'b' to boot



0
 
LVL 34

Accepted Solution

by:
Duncan Roe earned 43 total points
ID: 22695853
Unplug the 1TB disk and ensure you can boot again. I don't like the sound of what you've done so far - why should there have been 2 partitions on the new disk? Sounds to me like you just deleted the old ones
0
 
LVL 19

Expert Comment

by:jools
ID: 22697666
The last time I had this error the /selinux directory didnt exist.

do;
   mkdir /selinux

Then reboot, worth a try.
0
 

Author Comment

by:trymbill
ID: 22697863
Thanks MySidia, I'll try that later today.

What should I do if I can boot into the server without SELinux?

Duncan_roe; I agree ... I think it was a stupid mistake deleting those partitions cuz I wasn't sure what they were doing there.

Even so; I know that there was nothing running on sdb.  I checked 'mount' before doing anything to see a list of everything mounted and there was no mention of sdb there.  Why it showed 2 partitions on sdb, I don't know.  But I'm sure that the 2 RAID 1 disks are running on sda ... think I remember that /boot was mounted on sda.

But my question is now; How can I fix this?  Can I some how just fix the kernel or can I update the system to fix it or do I just have to run selinux=0 always? :-/

Thanks for the help! =)
0
 
LVL 19

Expert Comment

by:jools
ID: 22697883
Did the selinux directory exist?
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 

Author Comment

by:trymbill
ID: 22697888
I just read my own comment and I think I figured out what I did ...

There are 4 SATA slots on the computer.  The two disks on RAID 1 are connected to one joint slot and of course they are on SDA and SDB !!  There was also a SDC and that is the 1 TB disk ... omg!  I've been to messed up in the head to make sence of anything so I deleted everything on one of the disks.

O.K., can some one please tell me ... if everything is deleted from a disk on RAID 1, does the other disk wipe it self?! :S
0
 
LVL 19

Expert Comment

by:jools
ID: 22697893
Yes, you have an exact mirror of nothing.
0
 

Author Comment

by:trymbill
ID: 22697898
How can CentOS then start it self up?  I mean ... the OS is running on the hard drives ... if there is nothing on them, how can an OS .. you see my point? :)
0
 
LVL 23

Expert Comment

by:Mysidia
ID: 22697904
You can edit  /etc/sysconfig/selinux
to change from Enforcing to Permissive  mode after the first boot.
Check that your files are intact in /selinux

See what state the system is in before being concerned about SELinux

From there, it seems like you may need to restore /selinux from backup or rebuild your selinux policy database
& possibly  relabel

touch /.autorelabel
and reboot again  to re-label files on the filesystem  according to the
labels defined inyour SELinux configuration

0
 
LVL 19

Assisted Solution

by:jools
jools earned 41 total points
ID: 22697981
.... you couldnt have removed everything from the disk then...

Boot the system up now, do you still get  the issue above?

Run fdisk -l on the system to see what disks are on what controller?

Hardware or software RAID? if hardware RAID, check the status, if you moved the disks then the raid controller may still look at the proper disk. Software RAID? run mdadm --detail --scan
0
 
LVL 19

Expert Comment

by:jools
ID: 22698044
The directory just needs to exist for selinux to work....
0
 
LVL 23

Assisted Solution

by:Mysidia
Mysidia earned 41 total points
ID: 22698141
If  /etc/selinux and /selinux exist, but some  SELinux policy data files in /selinux and /etc/selinux are present and corrupt, and SELinux is in enforcing mode, the kernel WILL panic with messages like that shown above.  

If you are using a hardware RAID controller and have two logical disks; one with the first two disks in a hardware raid setup,  and a second logical drive with the new 1TB drive  as a JBOD/Volume with just one drive,  then this is a fairly typical setup, and what I would expect to be used in a proper configuration  (Software RAID is essentially no good,  in that it is a major performance hit on disk I/O).


If you are using software RAID,  then I would be concerned that when you used the command  "fdisk /dev/sdb"

You could have accidentally changed the partition table of the mirrored drive and caused corruption of the tables.   That could explain why the selinux database is inaccessible.

Your bootloader may still be looking at a partition that is no longer recorded as existing;   the kernel loads into memory successfully,  but some read operations (like the read for /etc/selinux)  fail.

I would strongly consider booting the system with a Knoppix CD and examining which partitions  and drives the system has.

Is there a /dev/sda   is there a /dev/sdb   is there a /dev/sdc   ??



0
 
LVL 34

Expert Comment

by:Duncan Roe
ID: 22698803
I still think you need to find out just what you still have. I suggest boot a standalone distribution like Knoppix and check what is on each disk using fdisk (p to print table, q to quit). You may find that sda is still good and sdb is trashed - since they are mirrors that might account for the kernel panic. In that case, try putting sdb's partition table back *exactly* as sda's - experiment until they really match (without changeing sda's table of course). They *might* then sync up
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

Daily system administration tasks often require administrators to connect remote systems. But allowing these remote systems to accept passwords makes these systems vulnerable to the risk of brute-force password guessing attacks. Furthermore there ar…
Setting up Secure Ubuntu server on VMware 1.      Insert the Ubuntu Server distribution CD or attach the ISO of the CD which is in the “Datastore”. Note that it is important to install the x64 edition on servers, not the X86 editions. 2.      Power on th…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now