Patrick Fox
asked on
Linux kernel panic ext3-fs error
Hello,
I'm having trouble finding a solution for this problem. The problem occurs on a server Red Hat 5.9 with a raid controller. On restarting the machine a kernel panic occurs as on the picture below :
I've tried to boot with a cd of Systemrescuecd to check if partitions are full which is not the case. I've also checked the SMART status of the disks, they are all healthy. And I did a "fsck" of all partitions (logical volumes) but it didn't solve the problem. The server uses a multipath to connect to a storage bay. Here is a picture of the fstab configuration and multipath.conf :
I am looking for new possible solutions.
Thanks for helping
I'm having trouble finding a solution for this problem. The problem occurs on a server Red Hat 5.9 with a raid controller. On restarting the machine a kernel panic occurs as on the picture below :
I've tried to boot with a cd of Systemrescuecd to check if partitions are full which is not the case. I've also checked the SMART status of the disks, they are all healthy. And I did a "fsck" of all partitions (logical volumes) but it didn't solve the problem. The server uses a multipath to connect to a storage bay. Here is a picture of the fstab configuration and multipath.conf :
I am looking for new possible solutions.
Thanks for helping
ASKER
Thanks Arnold, I have tried running fsck.ext3 on the different logical volumes, I didn't get any errors and restarting the server just gave the same error.
I'm not familiar with multipath, how can I check consistency ?
On the Red Hat server 5.9 I didn't find the folder /var/log/ maybe it is mounted on a different logical volume.
I can almost only take screenshots using teamviewer because the server is distant and I have to access it threw a windows machine with putty.
Is it expensive to have some more technical advice ?
Thanks
Patrick
I'm not familiar with multipath, how can I check consistency ?
On the Red Hat server 5.9 I didn't find the folder /var/log/ maybe it is mounted on a different logical volume.
I can almost only take screenshots using teamviewer because the server is distant and I have to access it threw a windows machine with putty.
Is it expensive to have some more technical advice ?
Thanks
Patrick
What is the hardware on which it is running?
Does the hardware have raid HW?
It is impossible not to have /var/log it looks as though the / partition..
fdisk -l to see the disks
You might have a failed disk.
Does the hardware have raid HW?
It is impossible not to have /var/log it looks as though the / partition..
fdisk -l to see the disks
You might have a failed disk.
ASKER
So the hardware is 1 server IBM x3650 M4 with disk bay IBM V3700
It has a hardware raid controller (dm-0)
I confirm there is no /var/log/ maybe a partition isn't mounted I'm checking for that
I've checked hard disks with "smartcl -H" everything seems fine
It has a hardware raid controller (dm-0)
I confirm there is no /var/log/ maybe a partition isn't mounted I'm checking for that
I've checked hard disks with "smartcl -H" everything seems fine
From what I saw in your fstab /var/log is part of your single Volgroup00/log-vol00 which is /
fdisk -l
Lvmdiskscan
Pvdisplay
Vgdisplay
Lvdisplay
Unless custom/etc/fstab is from another source...
Do you have IBM director installed in the Linux environment, or is it a VM environment over a Windows host?
fdisk -l
Lvmdiskscan
Pvdisplay
Vgdisplay
Lvdisplay
Unless custom/etc/fstab is from another source...
Do you have IBM director installed in the Linux environment, or is it a VM environment over a Windows host?
ASKER
I have recreated the /var/log/ folder and "touched" the different syslog files, I have also commented most of the logical volumes in fstab to have the minimum configuration that might work.
it is a system RH 5.9 working directly on the server
The computer has restarted, I'm going to check if the log files contain something
it is a system RH 5.9 working directly on the server
The computer has restarted, I'm going to check if the log files contain something
this seems to be an oracle server of some sorts, db, application server?
ASKER
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
it seems only /var is affected as if something wiped it out
ASKER
I have "rsynced" the "/var" folder with a similar server to restore to the closest file structure, but unfortunately I have the same error on boot.
Anymore ideas ?
Anymore ideas ?
Init (system initialization process) fails right before because
something wiped /var look in /sbin /bin /usr/bin /usr/sbin
The system is likely missing commands since the system does not drop out into maintenance mode which would follow if a disk or partition can not be mounted, but to continue the init process see if you have /etc/inittab on this system
Do you have an OS level backup.
something wiped /var look in /sbin /bin /usr/bin /usr/sbin
The system is likely missing commands since the system does not drop out into maintenance mode which would follow if a disk or partition can not be mounted, but to continue the init process see if you have /etc/inittab on this system
Do you have an OS level backup.
ASKER
The server had all kinds of problems, suppressed files in /etc and folders in /var it also needed a repair with fsck.ext3.
Finally another fresh installation was used.
Thanks for your help.
Finally another fresh installation was used.
Thanks for your help.
ASKER
We finally changed our server, too much trouble with the one I tried to repair
Your issue might be inconsistency in the multipath.
Posting images as you have does not convey he issue at hand.
Look in /var/log/messages
Check the bay storage for disk issues..