Solved

Kernel Panic after fsck

Posted on 2008-10-11
18
2,935 Views
Last Modified: 2012-05-05
I've got a machine running CentOS 5.  There are 2 hard drives in the machine, they are running on RAID 1 so I'm mirroring everything so that if one fails I have a backup on the other drive.  I'm running Apache, PHP, MySQL and some FTP services on the machine.

I just installed a 1TB disk and I had to move the other 2 disks from sdb to sda because the computer wouldn't start with the 1TB disk on sda even though I tried to change the grub.conf file to make it use hd1,0 instead of hd0,0.  That worked fine.  So then I had 2 disks on sda (sda1 and sda2) and one on sdb.

I wanted to have only one partition on sdb so I used fdisk and deleted sdb1 and sdb2.  I created a new partition set to primary and number 1 with default block settings.  I pressed T and selected 83 (Linux) for ext2.  Then I pressed W to write the changes and fdisk told me I had to restart because sdb was still in use by the system.  I didn't understand that because sdb was not in use by anything, I mean, the 2 hard drives running /boot where on sda.

I restarted and the machine didn't answer ping so I went down to the engine room and saw that there was an error.  It said that there where some bad blocks (or something like that) on /dev/VolGroup00/Log00 and that I should run fsck manually to fix it.  I ran fsck -y and it restarted the computer.

Now, when it finished starting up it had a new error with "Kernel Panic" :-/

It said:
no fstab.sys, mounting internal defaults
Switching to new root and running init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
audit(1223665793.629:2) enforcing=1 old_enforcing=0 ...
libsepol.policydb_read: policydb magic number ********* does not match expected magic number ********* or *********
libsepol.policydb_from_image: policy image is invalid
security: policydb macig number ******** does not match expected magic number **********
Unable to load SELinux Policy. Machine is in enforcing mode. Halting now.
Kernel panic - not syncing: Attempted to kill init!


So my question is; What is wrong?  How can I fix it without destroying anything on the machine?  Can I somehow run the Live CD and fix this there?
0
Comment
Question by:trymbill
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 4
  • 3
  • +2
18 Comments
 
LVL 19

Expert Comment

by:http:// thevpn.guru
ID: 22694351
Well at it's simplest your linux server is searching for your partitions at sda and they are at sdb
0
 
LVL 23

Expert Comment

by:Mysidia
ID: 22695014
When you are booting append the following option:

selinux=0  

To temporarily disable selinux.


You may need to get in single user mode to change /etc/fstab

to do so  append

selinux=0  init=/bin/sh


to the kernel boot options.
0
 
LVL 23

Expert Comment

by:Mysidia
ID: 22695035
to append kernel options when booting with lilo, you can normally at the boot prompt type:

boot: linux  selinux=0  init=/bin/sh
pressing tab at 'boot:'    sometimes gives a list of options.

With grub it is easier, use the menu keys to edit the  kernel command line

Or choose your kernel and hit 'a'  for append,  add the additional options
press enter...  'b' to boot



0
Efficient way to get backups off site to Azure

This user guide provides instructions on how to deploy and configure both a StoneFly Scale Out NAS Enterprise Cloud Drive virtual machine and Veeam Cloud Connect in the Microsoft Azure Cloud.

 
LVL 35

Accepted Solution

by:
Duncan Roe earned 43 total points
ID: 22695853
Unplug the 1TB disk and ensure you can boot again. I don't like the sound of what you've done so far - why should there have been 2 partitions on the new disk? Sounds to me like you just deleted the old ones
0
 
LVL 19

Expert Comment

by:jools
ID: 22697666
The last time I had this error the /selinux directory didnt exist.

do;
   mkdir /selinux

Then reboot, worth a try.
0
 

Author Comment

by:trymbill
ID: 22697863
Thanks MySidia, I'll try that later today.

What should I do if I can boot into the server without SELinux?

Duncan_roe; I agree ... I think it was a stupid mistake deleting those partitions cuz I wasn't sure what they were doing there.

Even so; I know that there was nothing running on sdb.  I checked 'mount' before doing anything to see a list of everything mounted and there was no mention of sdb there.  Why it showed 2 partitions on sdb, I don't know.  But I'm sure that the 2 RAID 1 disks are running on sda ... think I remember that /boot was mounted on sda.

But my question is now; How can I fix this?  Can I some how just fix the kernel or can I update the system to fix it or do I just have to run selinux=0 always? :-/

Thanks for the help! =)
0
 
LVL 19

Expert Comment

by:jools
ID: 22697883
Did the selinux directory exist?
0
 

Author Comment

by:trymbill
ID: 22697888
I just read my own comment and I think I figured out what I did ...

There are 4 SATA slots on the computer.  The two disks on RAID 1 are connected to one joint slot and of course they are on SDA and SDB !!  There was also a SDC and that is the 1 TB disk ... omg!  I've been to messed up in the head to make sence of anything so I deleted everything on one of the disks.

O.K., can some one please tell me ... if everything is deleted from a disk on RAID 1, does the other disk wipe it self?! :S
0
 
LVL 19

Expert Comment

by:jools
ID: 22697893
Yes, you have an exact mirror of nothing.
0
 

Author Comment

by:trymbill
ID: 22697898
How can CentOS then start it self up?  I mean ... the OS is running on the hard drives ... if there is nothing on them, how can an OS .. you see my point? :)
0
 
LVL 23

Expert Comment

by:Mysidia
ID: 22697904
You can edit  /etc/sysconfig/selinux
to change from Enforcing to Permissive  mode after the first boot.
Check that your files are intact in /selinux

See what state the system is in before being concerned about SELinux

From there, it seems like you may need to restore /selinux from backup or rebuild your selinux policy database
& possibly  relabel

touch /.autorelabel
and reboot again  to re-label files on the filesystem  according to the
labels defined inyour SELinux configuration

0
 
LVL 19

Assisted Solution

by:jools
jools earned 41 total points
ID: 22697981
.... you couldnt have removed everything from the disk then...

Boot the system up now, do you still get  the issue above?

Run fdisk -l on the system to see what disks are on what controller?

Hardware or software RAID? if hardware RAID, check the status, if you moved the disks then the raid controller may still look at the proper disk. Software RAID? run mdadm --detail --scan
0
 
LVL 19

Expert Comment

by:jools
ID: 22698044
The directory just needs to exist for selinux to work....
0
 
LVL 23

Assisted Solution

by:Mysidia
Mysidia earned 41 total points
ID: 22698141
If  /etc/selinux and /selinux exist, but some  SELinux policy data files in /selinux and /etc/selinux are present and corrupt, and SELinux is in enforcing mode, the kernel WILL panic with messages like that shown above.  

If you are using a hardware RAID controller and have two logical disks; one with the first two disks in a hardware raid setup,  and a second logical drive with the new 1TB drive  as a JBOD/Volume with just one drive,  then this is a fairly typical setup, and what I would expect to be used in a proper configuration  (Software RAID is essentially no good,  in that it is a major performance hit on disk I/O).


If you are using software RAID,  then I would be concerned that when you used the command  "fdisk /dev/sdb"

You could have accidentally changed the partition table of the mirrored drive and caused corruption of the tables.   That could explain why the selinux database is inaccessible.

Your bootloader may still be looking at a partition that is no longer recorded as existing;   the kernel loads into memory successfully,  but some read operations (like the read for /etc/selinux)  fail.

I would strongly consider booting the system with a Knoppix CD and examining which partitions  and drives the system has.

Is there a /dev/sda   is there a /dev/sdb   is there a /dev/sdc   ??



0
 
LVL 35

Expert Comment

by:Duncan Roe
ID: 22698803
I still think you need to find out just what you still have. I suggest boot a standalone distribution like Knoppix and check what is on each disk using fdisk (p to print table, q to quit). You may find that sda is still good and sdb is trashed - since they are mirrors that might account for the kernel panic. In that case, try putting sdb's partition table back *exactly* as sda's - experiment until they really match (without changeing sda's table of course). They *might* then sync up
0

Featured Post

Comprehensive Backup Solutions for Microsoft

Acronis protects the complete Microsoft technology stack: Windows Server, Windows PC, laptop and Surface data; Microsoft business applications; Microsoft Hyper-V; Azure VMs; Microsoft Windows Server 2016; Microsoft Exchange 2016 and SQL Server 2016.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Over the last ten+ years I have seen Linux configuration tools come and go. In the early days there was the tried-and-true, all-powerful linuxconf that many thought would remain the one and only Linux configuration tool until the end of times. Well,…
If you have a server on collocation with the super-fast CPU, that doesn't mean that you get it running at full power. Here is a preamble. When doing inventory of Linux servers, that I'm administering, I've found that some of them are running on l…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
Suggested Courses

632 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question