Solved

Kernel Panic after fsck

Posted on 2008-10-11
18
2,903 Views
Last Modified: 2012-05-05
I've got a machine running CentOS 5.  There are 2 hard drives in the machine, they are running on RAID 1 so I'm mirroring everything so that if one fails I have a backup on the other drive.  I'm running Apache, PHP, MySQL and some FTP services on the machine.

I just installed a 1TB disk and I had to move the other 2 disks from sdb to sda because the computer wouldn't start with the 1TB disk on sda even though I tried to change the grub.conf file to make it use hd1,0 instead of hd0,0.  That worked fine.  So then I had 2 disks on sda (sda1 and sda2) and one on sdb.

I wanted to have only one partition on sdb so I used fdisk and deleted sdb1 and sdb2.  I created a new partition set to primary and number 1 with default block settings.  I pressed T and selected 83 (Linux) for ext2.  Then I pressed W to write the changes and fdisk told me I had to restart because sdb was still in use by the system.  I didn't understand that because sdb was not in use by anything, I mean, the 2 hard drives running /boot where on sda.

I restarted and the machine didn't answer ping so I went down to the engine room and saw that there was an error.  It said that there where some bad blocks (or something like that) on /dev/VolGroup00/Log00 and that I should run fsck manually to fix it.  I ran fsck -y and it restarted the computer.

Now, when it finished starting up it had a new error with "Kernel Panic" :-/

It said:
no fstab.sys, mounting internal defaults
Switching to new root and running init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
audit(1223665793.629:2) enforcing=1 old_enforcing=0 ...
libsepol.policydb_read: policydb magic number ********* does not match expected magic number ********* or *********
libsepol.policydb_from_image: policy image is invalid
security: policydb macig number ******** does not match expected magic number **********
Unable to load SELinux Policy. Machine is in enforcing mode. Halting now.
Kernel panic - not syncing: Attempted to kill init!


So my question is; What is wrong?  How can I fix it without destroying anything on the machine?  Can I somehow run the Live CD and fix this there?
0
Comment
Question by:trymbill
  • 5
  • 4
  • 3
  • +2
18 Comments
 
LVL 19

Expert Comment

by:http:// thevpn.guru
ID: 22694351
Well at it's simplest your linux server is searching for your partitions at sda and they are at sdb
0
 
LVL 23

Expert Comment

by:Mysidia
ID: 22695014
When you are booting append the following option:

selinux=0  

To temporarily disable selinux.


You may need to get in single user mode to change /etc/fstab

to do so  append

selinux=0  init=/bin/sh


to the kernel boot options.
0
 
LVL 23

Expert Comment

by:Mysidia
ID: 22695035
to append kernel options when booting with lilo, you can normally at the boot prompt type:

boot: linux  selinux=0  init=/bin/sh
pressing tab at 'boot:'    sometimes gives a list of options.

With grub it is easier, use the menu keys to edit the  kernel command line

Or choose your kernel and hit 'a'  for append,  add the additional options
press enter...  'b' to boot



0
Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.

 
LVL 34

Accepted Solution

by:
Duncan Roe earned 43 total points
ID: 22695853
Unplug the 1TB disk and ensure you can boot again. I don't like the sound of what you've done so far - why should there have been 2 partitions on the new disk? Sounds to me like you just deleted the old ones
0
 
LVL 19

Expert Comment

by:jools
ID: 22697666
The last time I had this error the /selinux directory didnt exist.

do;
   mkdir /selinux

Then reboot, worth a try.
0
 

Author Comment

by:trymbill
ID: 22697863
Thanks MySidia, I'll try that later today.

What should I do if I can boot into the server without SELinux?

Duncan_roe; I agree ... I think it was a stupid mistake deleting those partitions cuz I wasn't sure what they were doing there.

Even so; I know that there was nothing running on sdb.  I checked 'mount' before doing anything to see a list of everything mounted and there was no mention of sdb there.  Why it showed 2 partitions on sdb, I don't know.  But I'm sure that the 2 RAID 1 disks are running on sda ... think I remember that /boot was mounted on sda.

But my question is now; How can I fix this?  Can I some how just fix the kernel or can I update the system to fix it or do I just have to run selinux=0 always? :-/

Thanks for the help! =)
0
 
LVL 19

Expert Comment

by:jools
ID: 22697883
Did the selinux directory exist?
0
 

Author Comment

by:trymbill
ID: 22697888
I just read my own comment and I think I figured out what I did ...

There are 4 SATA slots on the computer.  The two disks on RAID 1 are connected to one joint slot and of course they are on SDA and SDB !!  There was also a SDC and that is the 1 TB disk ... omg!  I've been to messed up in the head to make sence of anything so I deleted everything on one of the disks.

O.K., can some one please tell me ... if everything is deleted from a disk on RAID 1, does the other disk wipe it self?! :S
0
 
LVL 19

Expert Comment

by:jools
ID: 22697893
Yes, you have an exact mirror of nothing.
0
 

Author Comment

by:trymbill
ID: 22697898
How can CentOS then start it self up?  I mean ... the OS is running on the hard drives ... if there is nothing on them, how can an OS .. you see my point? :)
0
 
LVL 23

Expert Comment

by:Mysidia
ID: 22697904
You can edit  /etc/sysconfig/selinux
to change from Enforcing to Permissive  mode after the first boot.
Check that your files are intact in /selinux

See what state the system is in before being concerned about SELinux

From there, it seems like you may need to restore /selinux from backup or rebuild your selinux policy database
& possibly  relabel

touch /.autorelabel
and reboot again  to re-label files on the filesystem  according to the
labels defined inyour SELinux configuration

0
 
LVL 19

Assisted Solution

by:jools
jools earned 41 total points
ID: 22697981
.... you couldnt have removed everything from the disk then...

Boot the system up now, do you still get  the issue above?

Run fdisk -l on the system to see what disks are on what controller?

Hardware or software RAID? if hardware RAID, check the status, if you moved the disks then the raid controller may still look at the proper disk. Software RAID? run mdadm --detail --scan
0
 
LVL 19

Expert Comment

by:jools
ID: 22698044
The directory just needs to exist for selinux to work....
0
 
LVL 23

Assisted Solution

by:Mysidia
Mysidia earned 41 total points
ID: 22698141
If  /etc/selinux and /selinux exist, but some  SELinux policy data files in /selinux and /etc/selinux are present and corrupt, and SELinux is in enforcing mode, the kernel WILL panic with messages like that shown above.  

If you are using a hardware RAID controller and have two logical disks; one with the first two disks in a hardware raid setup,  and a second logical drive with the new 1TB drive  as a JBOD/Volume with just one drive,  then this is a fairly typical setup, and what I would expect to be used in a proper configuration  (Software RAID is essentially no good,  in that it is a major performance hit on disk I/O).


If you are using software RAID,  then I would be concerned that when you used the command  "fdisk /dev/sdb"

You could have accidentally changed the partition table of the mirrored drive and caused corruption of the tables.   That could explain why the selinux database is inaccessible.

Your bootloader may still be looking at a partition that is no longer recorded as existing;   the kernel loads into memory successfully,  but some read operations (like the read for /etc/selinux)  fail.

I would strongly consider booting the system with a Knoppix CD and examining which partitions  and drives the system has.

Is there a /dev/sda   is there a /dev/sdb   is there a /dev/sdc   ??



0
 
LVL 34

Expert Comment

by:Duncan Roe
ID: 22698803
I still think you need to find out just what you still have. I suggest boot a standalone distribution like Knoppix and check what is on each disk using fdisk (p to print table, q to quit). You may find that sda is still good and sdb is trashed - since they are mirrors that might account for the kernel panic. In that case, try putting sdb's partition table back *exactly* as sda's - experiment until they really match (without changeing sda's table of course). They *might* then sync up
0

Featured Post

Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Daily system administration tasks often require administrators to connect remote systems. But allowing these remote systems to accept passwords makes these systems vulnerable to the risk of brute-force password guessing attacks. Furthermore there ar…
Google Drive is extremely cheap offsite storage, and it's even possible to get extra storage for free for two years.  You can use the free account 15GB, and if you have an Android device..when you install Google Drive for the first time it will give…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

813 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now