Link to home
Start Free TrialLog in
Avatar of alantone
alantone

asked on

ext3-fs error (device dm-) in start-transition: journal has aborted

I am having trouble with my Linux Server. The server seems to boot initially just fine, then after a random amount of time the box starts acting up. MySql quits, and among other things my users can't log in an place orders, At this poing I can no longer log into the server and when I force a reboot everything seems to work OK again...for a while. If I'm on the machine and click Reboot from the Linux Log in screen I get the following error over and over in a loop on the screen:
ext3-fs error (device dm-0) in start-transition: journal has aborted

Im running RH Entterprise 4: Linux 2.6.9-55.0.2ELsmp
The Server is an Intel P(D) 3.2
It has 2GB Ram and 3 250GB HD's
Avatar of giltjr
giltjr
Flag of United States of America image

If you have not, I would suggest backing up everything NOW.  My first guess is that one of the hard drives is failing.  Are they any other messages dealing with the drives?
Avatar of alantone
alantone

ASKER

I do have everything backed up, how do I know which drive is failing, and is there any way to check it's status, and/or remove the drive from the group without reinstalling everything. The 3 250GB SATAII drives are grouped into 1 750GB group. When I look at the drive(s) in the System Monitor they are labeled as /dev/mapper/VolGroup00-LogVol00.
Do you have smartd enabled?  Do you see any errors in /var/log/messages from smartd?

    grep smartd /var/log/messages
Here's what grep smartd /var/log/messages returns:

[root@host ~]# grep smartd /var/log/messages
Jul 22 10:17:44 host smartd[3098]: smartd version 5.33 [i686-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Jul 22 10:17:44 host smartd[3098]: Home page is http://smartmontools.sourceforge.net/ 
Jul 22 10:17:44 host smartd[3098]: Opened configuration file /etc/smartd.conf
Jul 22 10:17:44 host smartd[3098]: Configuration file /etc/smartd.conf parsed.
Jul 22 10:17:44 host smartd[3098]: Device: /dev/sda, opened
Jul 22 10:17:44 host smartd[3098]: Device: /dev/sda, Bad IEC (SMART) mode page, err=-5, skip device
Jul 22 10:17:44 host smartd[3098]: Unable to register SCSI device /dev/sda at line 1 of file /etc/smartd.conf
Jul 22 10:17:44 host smartd[3098]: Unable to register device /dev/sda (no Directive -d removable). Exiting.
Jul 22 10:17:44 host smartd: smartd startup failed
Jul 22 13:23:58 host smartd[3109]: smartd version 5.33 [i686-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Jul 22 13:23:58 host smartd[3109]: Home page is http://smartmontools.sourceforge.net/ 
Jul 22 13:23:58 host smartd[3109]: Opened configuration file /etc/smartd.conf
Jul 22 13:23:58 host smartd[3109]: Configuration file /etc/smartd.conf parsed.
Jul 22 13:23:58 host smartd[3109]: Device: /dev/sda, opened
Jul 22 13:23:58 host smartd[3109]: Device: /dev/sda, Bad IEC (SMART) mode page, err=-5, skip device
Jul 22 13:23:58 host smartd[3109]: Unable to register SCSI device /dev/sda at line 1 of file /etc/smartd.conf
Jul 22 13:23:58 host smartd[3109]: Unable to register device /dev/sda (no Directive -d removable). Exiting.
Jul 22 13:23:58 host smartd: smartd startup failed
Jul 22 15:22:52 host smartd[3051]: smartd version 5.33 [i686-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Jul 22 15:22:52 host smartd[3051]: Home page is http://smartmontools.sourceforge.net/ 
Jul 22 15:22:52 host smartd[3051]: Opened configuration file /etc/smartd.conf
Jul 22 15:22:52 host smartd[3051]: Configuration file /etc/smartd.conf parsed.
Jul 22 15:22:52 host smartd[3051]: Device: /dev/sda, opened
Jul 22 15:22:52 host smartd[3051]: Device: /dev/sda, Bad IEC (SMART) mode page, err=-5, skip device
Jul 22 15:22:52 host smartd[3051]: Unable to register SCSI device /dev/sda at line 1 of file /etc/smartd.conf
Jul 22 15:22:52 host smartd[3051]: Unable to register device /dev/sda (no Directive -d removable). Exiting.
Jul 22 15:22:52 host smartd: smartd startup failed
Jul 22 17:49:41 host smartd[3098]: smartd version 5.33 [i686-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Jul 22 17:49:41 host smartd[3098]: Home page is http://smartmontools.sourceforge.net/ 
Jul 22 17:49:41 host smartd[3098]: Opened configuration file /etc/smartd.conf
Jul 22 17:49:41 host smartd[3098]: Configuration file /etc/smartd.conf parsed.
Jul 22 17:49:41 host smartd[3098]: Device: /dev/sda, opened
Jul 22 17:49:41 host smartd[3098]: Device: /dev/sda, Bad IEC (SMART) mode page, err=-5, skip device
Jul 22 17:49:41 host smartd[3098]: Unable to register SCSI device /dev/sda at line 1 of file /etc/smartd.conf
Jul 22 17:49:41 host smartd[3098]: Unable to register device /dev/sda (no Directive -d removable). Exiting.
Jul 22 17:49:41 host smartd: smartd startup failed
Jul 22 18:13:04 host smartd[3047]: smartd version 5.33 [i686-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Jul 22 18:13:04 host smartd[3047]: Home page is http://smartmontools.sourceforge.net/ 
Jul 22 18:13:04 host smartd[3047]: Opened configuration file /etc/smartd.conf
Jul 22 18:13:04 host smartd[3047]: Configuration file /etc/smartd.conf parsed.
Jul 22 18:13:04 host smartd[3047]: Device: /dev/sda, opened
Jul 22 18:13:04 host smartd[3047]: Device: /dev/sda, Bad IEC (SMART) mode page, err=-5, skip device
Jul 22 18:13:04 host smartd[3047]: Unable to register SCSI device /dev/sda at line 1 of file /etc/smartd.conf
Jul 22 18:13:04 host smartd[3047]: Unable to register device /dev/sda (no Directive -d removable). Exiting.
Jul 22 18:13:04 host smartd: smartd startup failed
Jul 22 18:17:58 host smartd: smartd shutdown failed
Jul 22 18:20:37 host smartd[2930]: smartd version 5.33 [i686-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Jul 22 18:20:37 host smartd[2930]: Home page is http://smartmontools.sourceforge.net/ 
Jul 22 18:20:37 host smartd[2930]: Opened configuration file /etc/smartd.conf
Jul 22 18:20:37 host smartd[2930]: Configuration file /etc/smartd.conf parsed.
Jul 22 18:20:37 host smartd[2930]: Device: /dev/sda, opened
Jul 22 18:20:37 host smartd[2930]: Device: /dev/sda, Bad IEC (SMART) mode page, err=-5, skip device
Jul 22 18:20:37 host smartd[2930]: Unable to register SCSI device /dev/sda at line 1 of file /etc/smartd.conf
Jul 22 18:20:37 host smartd[2930]: Unable to register device /dev/sda (no Directive -d removable). Exiting.
Jul 22 18:20:37 host smartd: smartd startup failed
Jul 22 18:38:24 host smartd: smartd shutdown failed
Jul 22 18:40:14 host smartd[3046]: smartd version 5.33 [i686-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Jul 22 18:40:14 host smartd[3046]: Home page is http://smartmontools.sourceforge.net/ 
Jul 22 18:40:14 host smartd[3046]: Opened configuration file /etc/smartd.conf
Jul 22 18:40:14 host smartd[3046]: Configuration file /etc/smartd.conf parsed.
Jul 22 18:40:14 host smartd[3046]: Device: /dev/sda, opened
Jul 22 18:40:14 host smartd[3046]: Device: /dev/sda, Bad IEC (SMART) mode page, err=-5, skip device
Jul 22 18:40:14 host smartd[3046]: Unable to register SCSI device /dev/sda at line 1 of file /etc/smartd.conf
Jul 22 18:40:14 host smartd[3046]: Unable to register device /dev/sda (no Directive -d removable). Exiting.
Jul 22 18:40:14 host smartd: smartd startup failed
Jul 22 20:20:28 host smartd[3098]: smartd version 5.33 [i686-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Jul 22 20:20:28 host smartd[3098]: Home page is http://smartmontools.sourceforge.net/ 
Jul 22 20:20:28 host smartd[3098]: Opened configuration file /etc/smartd.conf
Jul 22 20:20:28 host smartd[3098]: Configuration file /etc/smartd.conf parsed.
Jul 22 20:20:28 host smartd[3098]: Device: /dev/sda, opened
Jul 22 20:20:28 host smartd[3098]: Device: /dev/sda, Bad IEC (SMART) mode page, err=-5, skip device
Jul 22 20:20:28 host smartd[3098]: Unable to register SCSI device /dev/sda at line 1 of file /etc/smartd.conf
Jul 22 20:20:28 host smartd[3098]: Unable to register device /dev/sda (no Directive -d removable). Exiting.
Jul 22 20:20:28 host smartd: smartd startup failed
Jul 22 20:56:02 host smartd[3098]: smartd version 5.33 [i686-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Jul 22 20:56:02 host smartd[3098]: Home page is http://smartmontools.sourceforge.net/ 
Jul 22 20:56:02 host smartd[3098]: Opened configuration file /etc/smartd.conf
Jul 22 20:56:02 host smartd[3098]: Configuration file /etc/smartd.conf parsed.
Jul 22 20:56:02 host smartd[3098]: Device: /dev/sda, opened
Jul 22 20:56:02 host smartd[3098]: Device: /dev/sda, Bad IEC (SMART) mode page, err=-5, skip device
Jul 22 20:56:02 host smartd[3098]: Unable to register SCSI device /dev/sda at line 1 of file /etc/smartd.conf
Jul 22 20:56:02 host smartd[3098]: Unable to register device /dev/sda (no Directive -d removable). Exiting.
Jul 22 20:56:02 host smartd: smartd startup failed
Jul 22 21:39:05 host smartd[3072]: smartd version 5.33 [i686-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Jul 22 21:39:05 host smartd[3072]: Home page is http://smartmontools.sourceforge.net/ 
Jul 22 21:39:05 host smartd[3072]: Opened configuration file /etc/smartd.conf
Jul 22 21:39:05 host smartd[3072]: Configuration file /etc/smartd.conf parsed.
Jul 22 21:39:05 host smartd[3072]: Device: /dev/sda, opened
Jul 22 21:39:05 host smartd[3072]: Device: /dev/sda, Bad IEC (SMART) mode page, err=-5, skip device
Jul 22 21:39:05 host smartd[3072]: Unable to register SCSI device /dev/sda at line 1 of file /etc/smartd.conf
Jul 22 21:39:05 host smartd[3072]: Unable to register device /dev/sda (no Directive -d removable). Exiting.
Jul 22 21:39:05 host smartd: smartd startup failed
It appears that smartd is having problems starting.  Double check your drives documentation to see if they support SMART.

I would suggest resloving:

  --> Device: /dev/sda, Bad IEC (SMART) mode page, err=-5, skip device

error and then run the selftest utilties using smartctl from smarttools.
With respect to smartd, do you have any disk related errors when running 'dmesg'? It seems to be related to physical disk problems, which smartd doesn't always detect. It could be that you have a faulty disk (bad sectors are a common problem).
You could also attempt to boot into single-user mode (actually, /sbin/sh as your init) and run fsck on your entire device.
You can do it like this:
from your Grub menu, append to the boot command line the text "init=/bin/sh" (omit the quotes). You will boot into a system where your disk is read-only (which is good). Run "fsck -f /dev/VolGroup00/LogVol00" (omit the quotes, and I assume you used a default installation of RH type system). This should run a full scan on your disk (without any bad sector test, which would take a long while).
Notice - Backup all data you can back up BEFORE doing it. It can happen that during the "repair" process your existing data might be destroyed when hardware problems are involved.
Here is the result from dmesg. I see some Hard Drive info, but I don't see any glaring error messages. Do you see anything?

[root@host ~]# dmesg
Linux version 2.6.9-55.0.2.ELsmp (brewbuilder@hs20-bc2-3.build.redhat.com) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-8)) #1 SMP Tue Jun 12 17:59:08 EDT 2007
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000e7000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000007ffd0000 (usable)
 BIOS-e820: 000000007ffd0000 - 000000007ffde000 (ACPI data)
 BIOS-e820: 000000007ffde000 - 0000000080000000 (ACPI NVS)
 BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fef00000 (reserved)
 BIOS-e820: 00000000ff780000 - 0000000100000000 (reserved)
1151MB HIGHMEM available.
896MB LOWMEM available.
found SMP MP-table at 000ff780
Using x86 segment limits to approximate NX protection
On node 0 totalpages: 524240
  DMA zone: 4096 pages, LIFO batch:1
  Normal zone: 225280 pages, LIFO batch:16
  HighMem zone: 294864 pages, LIFO batch:16
DMI 2.3 present.
Using APIC driver default
ACPI: RSDP (v000 ACPIAM                                ) @ 0x000f9fa0
ACPI: RSDT (v001 A M I  OEMRSDT  0x04000606 MSFT 0x00000097) @ 0x7ffd0000
ACPI: FADT (v002 A M I  OEMFACP  0x04000606 MSFT 0x00000097) @ 0x7ffd0200
ACPI: MADT (v001 A M I  OEMAPIC  0x04000606 MSFT 0x00000097) @ 0x7ffd0390
ACPI: MCFG (v001 A M I  OEMMCFG  0x04000606 MSFT 0x00000097) @ 0x7ffd0410
ACPI: OEMB (v001 A M I  AMI_OEM  0x04000606 MSFT 0x00000097) @ 0x7ffde040
ACPI: DSDT (v001  1ADAE 1ADAE004 0x00000004 INTL 0x02002026) @ 0x00000000
Nvidia board detected. Ignoring ACPI timer override.
ACPI: PM-Timer IO Port: 0x4008
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Processor #0 15:4 APIC version 20
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
Processor #1 15:4 APIC version 20
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x82] disabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x83] disabled)
Enabling APIC mode:  Flat.  Using 0 I/O APICs
ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 2, version 17, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: BIOS IRQ0 pin2 override ignored.
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 14 global_irq 14 high edge)
ACPI: INT_SRC_OVR (bus 0 bus_irq 15 global_irq 15 high edge)
ACPI: IRQ9 used by override.
ACPI: IRQ14 used by override.
ACPI: IRQ15 used by override.
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at 88000000 (gap: 80000000:7ec00000)
Built 1 zonelists
Kernel command line: ro root=/dev/VolGroup00/LogVol00 rhgb quiet
mapped APIC to ffffd000 (fee00000)
Initializing CPU#0
CPU 0 irqstacks, hard=c03f1000 soft=c03d1000
PID hash table entries: 4096 (order: 12, 65536 bytes)
Detected 3214.276 MHz processor.
Using pmtmr for high-res timesource
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 2072964k/2096960k available (1883k kernel code, 23060k reserved, 761k data, 188k init, 1179456k highmem)
Calibrating delay using timer specific routine.. 6433.67 BogoMIPS (lpj=3216835)
Security Scaffold v1.0.0 initialized
SELinux:  Initializing.
SELinux:  Starting in permissive mode
There is already a security framework initialized, register_security failed.
selinux_register_security:  Registering secondary module capability
Capability LSM initialized as secondary
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
CPU: After generic identify, caps: bfebfbff 20000000 00000000 00000000
CPU: After vendor identify, caps:  bfebfbff 20000000 00000000 00000000
monitor/mwait feature present.
using mwait in idle threads.
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
CPU0: Physical Processor ID: 0
CPU0: Processor Core ID: 0
CPU0: Initial APIC ID: 0
CPU: After all inits, caps:        bfebf3ff 20000000 00000000 00000180
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU0: Intel P4/Xeon Extended MCE MSRs (24) available
CPU0: Thermal monitoring enabled
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... OK.
CPU0: Intel(R) Pentium(R) D CPU 3.20GHz stepping 04
per-CPU timeslice cutoff: 2924.17 usecs.
task migration cache decay timeout: 2 msecs.
Booting processor 1/1 eip 3000
CPU 1 irqstacks, hard=c03f2000 soft=c03d2000
Initializing CPU#1
Calibrating delay using timer specific routine.. 6427.02 BogoMIPS (lpj=3213513)
CPU: After generic identify, caps: bfebfbff 20000000 00000000 00000000
CPU: After vendor identify, caps:  bfebfbff 20000000 00000000 00000000
monitor/mwait feature present.
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
CPU1: Physical Processor ID: 0
CPU1: Processor Core ID: 1
CPU1: Initial APIC ID: 1
CPU: After all inits, caps:        bfebf3ff 20000000 00000000 00000180
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#1.
CPU1: Intel P4/Xeon Extended MCE MSRs (24) available
CPU1: Thermal monitoring enabled
CPU1: Intel(R) Pentium(R) D CPU 3.20GHz stepping 04
Total of 2 processors activated (12860.69 BogoMIPS).
ENABLING IO-APIC IRQs
..TIMER: vector=0x31 pin1=0 pin2=-1
checking TSC synchronization across 2 CPUs:
CPU#0 had -1003234 usecs TSC skew, fixed it up.
CPU#1 had 0 usecs TSC skew, fixed it up.
Brought up 2 CPUs
zapping low mappings.
checking if image is initramfs... it is
Freeing initrd memory: 1076k freed
NET: Registered protocol family 16
PCI: PCI BIOS revision 3.00 entry at 0xf0031, last bus=6
PCI: Using MMCONFIG
mtrr: v2.0 (20020519)
ACPI: Subsystem revision 20040816
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (00:00)
PCI: Probing PCI hardware (bus 00)
PCI: Transparent bridge - 0000:00:12.0
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P1._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs 16 17 18 19) *10
ACPI: PCI Interrupt Link [LNKB] (IRQs 16 17 18 19) *5
ACPI: PCI Interrupt Link [LNKC] (IRQs 16 17 18 19) *5
ACPI: PCI Interrupt Link [LNKD] (IRQs 16 17 18 19) *5
ACPI: PCI Interrupt Link [LNKE] (IRQs 16 17 18 19) *5
ACPI: PCI Interrupt Link [LUB0] (IRQs 20 21 22 23) *0, disabled.
ACPI: PCI Interrupt Link [LUB1] (IRQs 20 21 22 23) *0, disabled.
ACPI: PCI Interrupt Link [LUB2] (IRQs 20 21 22 23) *0, disabled.
ACPI: PCI Interrupt Link [LMAC] (IRQs 20 21 22 23) *5
ACPI: PCI Interrupt Link [LACI] (IRQs 20 21 22 23) *0, disabled.
ACPI: PCI Interrupt Link [LMC9] (IRQs 20 21 22 23) *0, disabled.
ACPI: PCI Interrupt Link [LSMB] (IRQs 20 21 22 23) *11
ACPI: PCI Interrupt Link [LSA0] (IRQs 20 21 22 23) *5
ACPI: PCI Interrupt Link [LSA1] (IRQs 20 21 22 23) *5
ACPI: PCI Interrupt Link [LATA] (IRQs 20 21 22 23) *0, disabled.
Linux Plug and Play Support v0.97 (c) Adam Belay
usbcore: registered new driver usbfs
usbcore: registered new driver hub
PCI: Using ACPI for IRQ routing
ACPI: PCI Interrupt Link [LSMB] enabled at IRQ 23
ACPI: PCI Interrupt 0000:00:0a.1[A] -> GSI 23 (level, low) -> IRQ 177
ACPI: PCI Interrupt Link [LMAC] enabled at IRQ 22
ACPI: PCI Interrupt 0000:00:0e.0[A] -> GSI 22 (level, low) -> IRQ 185
ACPI: PCI Interrupt Link [LSA0] enabled at IRQ 21
ACPI: PCI Interrupt 0000:00:10.0[A] -> GSI 21 (level, low) -> IRQ 193
ACPI: PCI Interrupt Link [LSA1] enabled at IRQ 20
ACPI: PCI Interrupt 0000:00:11.0[A] -> GSI 20 (level, low) -> IRQ 201
ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 19
ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 19 (level, low) -> IRQ 209
ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 19 (level, low) -> IRQ 209
ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 18
ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 18 (level, low) -> IRQ 217
ACPI: PCI Interrupt Link [LNKE] enabled at IRQ 17
ACPI: PCI Interrupt 0000:06:04.0[A] -> GSI 17 (level, low) -> IRQ 225
ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 16
ACPI: PCI Interrupt 0000:06:05.0[A] -> GSI 16 (level, low) -> IRQ 233
ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 19
ACPI: PCI Interrupt 0000:06:08.0[A] -> GSI 19 (level, low) -> IRQ 209
PCI: Device 00:70 not found by BIOS
apm: BIOS version 1.2 Flags 0x03 (Driver version 1.16ac)
apm: disabled - APM is not SMP safe.
audit: initializing netlink socket (disabled)
audit(1185189933.297:1): initialized
highmem bounce pool size: 64 pages
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
SELinux:  Registering netfilter hooks
Initializing Cryptographic API
ksign: Installing public key data
Loading keyring
- Added public key 4255E7776794CBF5
- User ID: Red Hat, Inc. (Kernel Module GPG key)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
ACPI: Processor [CPU1] (supports C1, 8 throttling states)
ACPI: Processor [CPU2] (supports C1, 8 throttling states)
Real Time Clock Driver v1.12
Linux agpgart interface v0.100 (c) Dave Jones
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
Serial: 8250/16550 driver $Revision: 1.90 $ 68 ports, IRQ sharing enabled
ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
RAMDISK driver initialized: 16 RAM disks of 16384K size 1024 blocksize
divert: not allocating divert_blk for non-ethernet device lo
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
NFORCE-MCP04: IDE controller at PCI slot 0000:00:0f.0
NFORCE-MCP04: chipset revision 242
NFORCE-MCP04: not 100% native mode: will probe irqs later
NFORCE-MCP04: 0000:00:0f.0 (rev f2) UDMA133 controller
    ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:DMA
    ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:pio, hdd:pio
Probing IDE interface ide0...
hda: Memorex DVD-ROM 16X v2, ATAPI CD/DVD-ROM drive
hdb: SONY CD-ROM CDU5225, ATAPI CD/DVD-ROM drive
Using cfq io scheduler
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Probing IDE interface ide1...
Probing IDE interface ide1...
Probing IDE interface ide2...
Probing IDE interface ide3...
Probing IDE interface ide4...
Probing IDE interface ide5...
hda: ATAPI 48X DVD-ROM drive, 1727kB Cache, UDMA(33)
Uniform CD-ROM driver Revision: 3.20
hdb: ATAPI 52X CD-ROM drive, 96kB Cache, UDMA(33)
ide-floppy driver 0.99.newide
usbcore: registered new driver hiddev
usbcore: registered new driver usbhid
drivers/usb/input/hid-core.c: v2.0:USB HID core driver
mice: PS/2 mouse device common for all mice
input: AT Translated Set 2 keyboard on isa0060/serio0
input: ImPS/2 Generic Wheel Mouse on isa0060/serio1
md: md driver 0.90.0 MAX_MD_DEVS=256, MD_SB_DISKS=27
NET: Registered protocol family 2
IP route cache hash table entries: 131072 (order: 7, 524288 bytes)
TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
TCP bind hash table entries: 262144 (order: 9, 3145728 bytes)
TCP: Hash tables configured (established 262144 bind 262144)
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
    ACPI-1134: *** Error: Method execution failed [\_PR_.CPU1._PDC] (Node f7ffe1c0), AE_BAD_HEADER
    ACPI-1134: *** Error: Method execution failed [\_PR_.CPU2._PDC] (Node f7ffef80), AE_BAD_HEADER
    ACPI-1134: *** Error: Method execution failed [\_PR_.CPU1._PDC] (Node f7ffe1c0), AE_BAD_HEADER
    ACPI-1134: *** Error: Method execution failed [\_PR_.CPU2._PDC] (Node f7ffef80), AE_BAD_HEADER
ACPI wakeup devices:
P0P8 P0PB P0PC PS2K PS2M NSMB USB0 USB1 USB2 NMAC P0P1 MC97 P0PA P0PD PWRB
ACPI: (supports S0 S1 S3 S4 S5)
Freeing unused kernel memory: 188k freed
SCSI subsystem initialized
libata version 2.00 loaded.
sata_nv 0000:00:10.0: version 3.2
ACPI: PCI Interrupt 0000:00:10.0[A] -> GSI 21 (level, low) -> IRQ 193
sata_nv 0000:00:10.0: Using ADMA mode
PCI: Setting latency timer of device 0000:00:10.0 to 64
ata1: SATA max UDMA/133 cmd 0xF881E480 ctl 0xF881E4A0 bmdma 0xA880 irq 193
ata2: SATA max UDMA/133 cmd 0xF881E580 ctl 0xF881E5A0 bmdma 0xA888 irq 193
scsi0 : sata_nv
ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata1.00: ATA-7, max UDMA/133, 490234752 sectors: LBA48 NCQ (depth 31/32)
ata1.00: ata1: dev 0 multi count 16
ata1.00: configured for UDMA/133
scsi1 : sata_nv
ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata2.00: ATA-7, max UDMA/133, 490234752 sectors: LBA48 NCQ (depth 31/32)
ata2.00: ata2: dev 0 multi count 16
ata2.00: configured for UDMA/133
  Vendor: ATA       Model: WDC WD2500YS-01S  Rev: 20.0
  Type:   Direct-Access                      ANSI SCSI revision: 05
ata1: bounce limit 0xFFFFFFFFFFFFFFFF, segment boundary 0xFFFFFFFF, hw segs 61
SCSI device sda: 490234752 512-byte hdwr sectors (251000 MB)
SCSI device sda: drive cache: write back
SCSI device sda: 490234752 512-byte hdwr sectors (251000 MB)
SCSI device sda: drive cache: write back
 sda: sda1 sda2
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
  Vendor: ATA       Model: WDC WD2500YS-01S  Rev: 20.0
  Type:   Direct-Access                      ANSI SCSI revision: 05
ata2: bounce limit 0xFFFFFFFFFFFFFFFF, segment boundary 0xFFFFFFFF, hw segs 61
SCSI device sdb: 490234752 512-byte hdwr sectors (251000 MB)
SCSI device sdb: drive cache: write back
SCSI device sdb: 490234752 512-byte hdwr sectors (251000 MB)
SCSI device sdb: drive cache: write back
 sdb: sdb1
Attached scsi disk sdb at scsi1, channel 0, id 0, lun 0
ACPI: PCI Interrupt 0000:00:11.0[A] -> GSI 20 (level, low) -> IRQ 201
sata_nv 0000:00:11.0: Using ADMA mode
PCI: Setting latency timer of device 0000:00:11.0 to 64
ata3: SATA max UDMA/133 cmd 0xF8820480 ctl 0xF88204A0 bmdma 0xA000 irq 201
ata4: SATA max UDMA/133 cmd 0xF8820580 ctl 0xF88205A0 bmdma 0xA008 irq 201
scsi2 : sata_nv
ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata3.00: ATA-7, max UDMA/133, 490234752 sectors: LBA48 NCQ (depth 31/32)
ata3.00: ata3: dev 0 multi count 16
ata3.00: configured for UDMA/133
scsi3 : sata_nv
ata4: SATA link down (SStatus 0 SControl 300)
  Vendor: ATA       Model: WDC WD2500YS-01S  Rev: 20.0
  Type:   Direct-Access                      ANSI SCSI revision: 05
ata3: bounce limit 0xFFFFFFFFFFFFFFFF, segment boundary 0xFFFFFFFF, hw segs 61
SCSI device sdc: 490234752 512-byte hdwr sectors (251000 MB)
SCSI device sdc: drive cache: write back
SCSI device sdc: 490234752 512-byte hdwr sectors (251000 MB)
SCSI device sdc: drive cache: write back
 sdc: sdc1
Attached scsi disk sdc at scsi2, channel 0, id 0, lun 0
device-mapper: 4.5.5-ioctl (2006-12-01) initialised: dm-devel@redhat.com
cdrom: open failed.
cdrom: open failed.
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: dm-0: orphan cleanup on readonly fs
ext3_orphan_cleanup: deleting unreferenced inode 59441162
ext3_orphan_cleanup: deleting unreferenced inode 59441160
ext3_orphan_cleanup: deleting unreferenced inode 59441159
ext3_orphan_cleanup: deleting unreferenced inode 59441155
EXT3-fs: dm-0: 4 orphan inodes deleted
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
security:  3 users, 4 roles, 356 types, 32 bools
security:  55 classes, 22693 rules
SELinux:  Completing initialization.
SELinux:  Setting up existing superblocks.
SELinux: initialized (dev dm-0, type ext3), uses xattr
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev selinuxfs, type selinuxfs), uses genfs_contexts
SELinux: initialized (dev mqueue, type mqueue), not configured for labeling
SELinux: initialized (dev hugetlbfs, type hugetlbfs), uses genfs_contexts
SELinux: initialized (dev devpts, type devpts), uses transition SIDs
SELinux: initialized (dev eventpollfs, type eventpollfs), uses genfs_contexts
SELinux: initialized (dev pipefs, type pipefs), uses task SIDs
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev futexfs, type futexfs), uses genfs_contexts
SELinux: initialized (dev sockfs, type sockfs), uses task SIDs
SELinux: initialized (dev proc, type proc), uses genfs_contexts
SELinux: initialized (dev bdev, type bdev), uses genfs_contexts
SELinux: initialized (dev rootfs, type rootfs), uses genfs_contexts
SELinux: initialized (dev sysfs, type sysfs), uses genfs_contexts
SELinux: initialized (dev usbfs, type usbfs), uses genfs_contexts
inserting floppy driver for 2.6.9-55.0.2.ELsmp
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
forcedeth.c: Reverse Engineered nForce ethernet driver. Version 0.42-rh.
ACPI: PCI Interrupt 0000:00:0e.0[A] -> GSI 22 (level, low) -> IRQ 185
PCI: Setting latency timer of device 0000:00:0e.0 to 64
divert: allocating divert_blk for eth0
eth0: forcedeth.c: subsystem: 01462:7160 bound to 0000:00:0e.0
8139too Fast Ethernet driver 0.9.27
ACPI: PCI Interrupt 0000:06:08.0[A] -> GSI 19 (level, low) -> IRQ 209
divert: allocating divert_blk for eth1
eth1: RealTek RTL8139 at 0xe400, 00:50:ba:c8:40:d4, IRQ 209
eth1:  Identified 8139 chip type 'RTL-8139C'
ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 19 (level, low) -> IRQ 209
PCI: Setting latency timer of device 0000:02:00.0 to 64
sky2 v1.6 addr 0xfe9fc000 irq 209 Yukon-EC (0xb6) rev 1
divert: allocating divert_blk for eth2
sky2 eth2: addr 00:13:d3:13:a9:22
MSI INIT SUCCESS
sata_sil24 0000:05:00.0: version 0.3
ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 18 (level, low) -> IRQ 217
PCI: Setting latency timer of device 0000:05:00.0 to 64
ata5: SATA max UDMA/100 cmd 0xF8908000 ctl 0x0 bmdma 0x0 irq 217
ata6: SATA max UDMA/100 cmd 0xF890A000 ctl 0x0 bmdma 0x0 irq 217
scsi4 : sata_sil24
ata5: SATA link down (SStatus 0 SControl 300)
scsi5 : sata_sil24
ata6: SATA link down (SStatus 0 SControl 300)
Evaluate _OSC Set fails. Status = 0x0005
pciehp: Both _OSC and OSHP methods do not exist
Evaluate _OSC Set fails. Status = 0x0005
pciehp: Both _OSC and OSHP methods do not exist
Evaluate _OSC Set fails. Status = 0x0005
pciehp: Both _OSC and OSHP methods do not exist
Evaluate _OSC Set fails. Status = 0x0005
pciehp: Both _OSC and OSHP methods do not exist
Evaluate _OSC Set fails. Status = 0x0005
pciehp: Both _OSC and OSHP methods do not exist
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
SELinux: initialized (dev ramfs, type ramfs), uses genfs_contexts
NET: Registered protocol family 10
Disabled Privacy Extensions on device c0346240(lo)
IPv6 over IPv4 tunneling driver
divert: not allocating divert_blk for non-ethernet device sit0
ACPI: Power Button (FF) [PWRF]
EXT3 FS on dm-0, internal journal
device-mapper: unknown target type
device-mapper: error adding target to table
cdrom: open failed.
cdrom: open failed.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
SELinux: initialized (dev sda1, type ext3), uses xattr
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
Adding 2031608k swap on /dev/VolGroup00/LogVol01.  Priority:-1 extents:1
SELinux: initialized (dev binfmt_misc, type binfmt_misc), uses genfs_contexts
cdrom: open failed.
cdrom: open failed.
cdrom: open failed.
cdrom: open failed.
parport0: PC-style at 0x378 (0x778) [PCSPP,TRISTATE,EPP]
parport0: irq 7 detected
ip_tables: (C) 2000-2002 Netfilter core team
ip_tables: (C) 2000-2002 Netfilter core team
ip_tables: (C) 2000-2002 Netfilter core team
eth1: link up, 100Mbps, full-duplex, lpa 0x41E1
SELinux: initialized (dev rpc_pipefs, type rpc_pipefs), uses genfs_contexts
i2c /dev entries driver
parport0: PC-style at 0x378 (0x778) [PCSPP,TRISTATE,EPP]
parport0: irq 7 detected
lp0: using parport0 (polling).
lp0: console ready
eth0: no IPv6 routers present
eth1: no IPv6 routers present
This is the output shortly after a reboot, right? What is the output (only the last few lines - 10-20 lines) of dmesg after the announcement of the filesystem being remounted read-only?
Thanks for all of your help. The one thing it made me realize is that I need to get my Red Hat RCHE Certification. I've had 2 web hosting companies and a web design company, but I've always been the boss who has just enough knowledge to buy the right stuff, but not enough knowledge to fix any of it. LOL.

With that in mind I bought 3 new drives this afternoon, I reinstalled and updated RH, installed Plesk and my 850 or so real estate Virtual Tours and sent the drives back to Tiger Direct. Of course it took all day, but everything is running great!

Thanks again!
As an RHCE, I can tell you that being one doesn't guarantee you know everything. You know some, and you know hoe much you still don't know...
ASKER CERTIFIED SOLUTION
Avatar of giltjr
giltjr
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
The Hard drive is supposed to be a Server Grade RAID drive from Western Digital WD2500YS, it's only about 6 months old. I'll attach them to a windows box and run some tests later to see which one failed.

I'm really sensitive about the server temp so I have multiple temp sensors on the case. Right now the server is 30.1C at the bottom, 36.0 next to the processor, 31.5 at the hard drives, and 31.2 at the top.

Power is +12 Volts=12.137V, +5 Volts=5.093V, +3.3 Volts=3.333V, +1.5Volts=1.26V, and Processor Vccp=1.164V.

After reading some reviews about these drives, everyone says they either fail in 6 months or run forever, I think I just got a bad one.

Thanks again!
Alan