Kernel Panic after fsck

I've got a machine running CentOS 5.  There are 2 hard drives in the machine, they are running on RAID 1 so I'm mirroring everything so that if one fails I have a backup on the other drive.  I'm running Apache, PHP, MySQL and some FTP services on the machine.

I just installed a 1TB disk and I had to move the other 2 disks from sdb to sda because the computer wouldn't start with the 1TB disk on sda even though I tried to change the grub.conf file to make it use hd1,0 instead of hd0,0.  That worked fine.  So then I had 2 disks on sda (sda1 and sda2) and one on sdb.

I wanted to have only one partition on sdb so I used fdisk and deleted sdb1 and sdb2.  I created a new partition set to primary and number 1 with default block settings.  I pressed T and selected 83 (Linux) for ext2.  Then I pressed W to write the changes and fdisk told me I had to restart because sdb was still in use by the system.  I didn't understand that because sdb was not in use by anything, I mean, the 2 hard drives running /boot where on sda.

I restarted and the machine didn't answer ping so I went down to the engine room and saw that there was an error.  It said that there where some bad blocks (or something like that) on /dev/VolGroup00/Log00 and that I should run fsck manually to fix it.  I ran fsck -y and it restarted the computer.

Now, when it finished starting up it had a new error with "Kernel Panic" :-/

It said:
no fstab.sys, mounting internal defaults
Switching to new root and running init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
audit(1223665793.629:2) enforcing=1 old_enforcing=0 ...
libsepol.policydb_read: policydb magic number ********* does not match expected magic number ********* or *********
libsepol.policydb_from_image: policy image is invalid
security: policydb macig number ******** does not match expected magic number **********
Unable to load SELinux Policy. Machine is in enforcing mode. Halting now.
Kernel panic - not syncing: Attempted to kill init!

So my question is; What is wrong?  How can I fix it without destroying anything on the machine?  Can I somehow run the Live CD and fix this there?
Who is Participating?
Duncan RoeConnect With a Mentor Software DeveloperCommented:
Unplug the 1TB disk and ensure you can boot again. I don't like the sound of what you've done so far - why should there have been 2 partitions on the new disk? Sounds to me like you just deleted the old ones
http:// thevpn.guruCommented:
Well at it's simplest your linux server is searching for your partitions at sda and they are at sdb
When you are booting append the following option:


To temporarily disable selinux.

You may need to get in single user mode to change /etc/fstab

to do so  append

selinux=0  init=/bin/sh

to the kernel boot options.
Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

to append kernel options when booting with lilo, you can normally at the boot prompt type:

boot: linux  selinux=0  init=/bin/sh
pressing tab at 'boot:'    sometimes gives a list of options.

With grub it is easier, use the menu keys to edit the  kernel command line

Or choose your kernel and hit 'a'  for append,  add the additional options
press enter...  'b' to boot

The last time I had this error the /selinux directory didnt exist.

   mkdir /selinux

Then reboot, worth a try.
trymbillAuthor Commented:
Thanks MySidia, I'll try that later today.

What should I do if I can boot into the server without SELinux?

Duncan_roe; I agree ... I think it was a stupid mistake deleting those partitions cuz I wasn't sure what they were doing there.

Even so; I know that there was nothing running on sdb.  I checked 'mount' before doing anything to see a list of everything mounted and there was no mention of sdb there.  Why it showed 2 partitions on sdb, I don't know.  But I'm sure that the 2 RAID 1 disks are running on sda ... think I remember that /boot was mounted on sda.

But my question is now; How can I fix this?  Can I some how just fix the kernel or can I update the system to fix it or do I just have to run selinux=0 always? :-/

Thanks for the help! =)
Did the selinux directory exist?
trymbillAuthor Commented:
I just read my own comment and I think I figured out what I did ...

There are 4 SATA slots on the computer.  The two disks on RAID 1 are connected to one joint slot and of course they are on SDA and SDB !!  There was also a SDC and that is the 1 TB disk ... omg!  I've been to messed up in the head to make sence of anything so I deleted everything on one of the disks.

O.K., can some one please tell me ... if everything is deleted from a disk on RAID 1, does the other disk wipe it self?! :S
Yes, you have an exact mirror of nothing.
trymbillAuthor Commented:
How can CentOS then start it self up?  I mean ... the OS is running on the hard drives ... if there is nothing on them, how can an OS .. you see my point? :)
You can edit  /etc/sysconfig/selinux
to change from Enforcing to Permissive  mode after the first boot.
Check that your files are intact in /selinux

See what state the system is in before being concerned about SELinux

From there, it seems like you may need to restore /selinux from backup or rebuild your selinux policy database
& possibly  relabel

touch /.autorelabel
and reboot again  to re-label files on the filesystem  according to the
labels defined inyour SELinux configuration

joolsConnect With a Mentor Commented:
.... you couldnt have removed everything from the disk then...

Boot the system up now, do you still get  the issue above?

Run fdisk -l on the system to see what disks are on what controller?

Hardware or software RAID? if hardware RAID, check the status, if you moved the disks then the raid controller may still look at the proper disk. Software RAID? run mdadm --detail --scan
The directory just needs to exist for selinux to work....
MysidiaConnect With a Mentor Commented:
If  /etc/selinux and /selinux exist, but some  SELinux policy data files in /selinux and /etc/selinux are present and corrupt, and SELinux is in enforcing mode, the kernel WILL panic with messages like that shown above.  

If you are using a hardware RAID controller and have two logical disks; one with the first two disks in a hardware raid setup,  and a second logical drive with the new 1TB drive  as a JBOD/Volume with just one drive,  then this is a fairly typical setup, and what I would expect to be used in a proper configuration  (Software RAID is essentially no good,  in that it is a major performance hit on disk I/O).

If you are using software RAID,  then I would be concerned that when you used the command  "fdisk /dev/sdb"

You could have accidentally changed the partition table of the mirrored drive and caused corruption of the tables.   That could explain why the selinux database is inaccessible.

Your bootloader may still be looking at a partition that is no longer recorded as existing;   the kernel loads into memory successfully,  but some read operations (like the read for /etc/selinux)  fail.

I would strongly consider booting the system with a Knoppix CD and examining which partitions  and drives the system has.

Is there a /dev/sda   is there a /dev/sdb   is there a /dev/sdc   ??

Duncan RoeSoftware DeveloperCommented:
I still think you need to find out just what you still have. I suggest boot a standalone distribution like Knoppix and check what is on each disk using fdisk (p to print table, q to quit). You may find that sda is still good and sdb is trashed - since they are mirrors that might account for the kernel panic. In that case, try putting sdb's partition table back *exactly* as sda's - experiment until they really match (without changeing sda's table of course). They *might* then sync up
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.