RH9 keeps crashing.

My Linux server crashed every few minutes. it runs RH9.
when the crash occurs, no service on it works and the screen is blank but I can still ping it.

here are the most recent logs on it. It crashes after the last log entry.

here is my df -h:

Filesystem            Size  Used Avail Use% Mounted on
/dev/hdc1              37G  9.5G   26G  28% /
/dev/hda1              99M   14M   80M  15% /boot
none                  251M     0  251M   0% /dev/shm

May  2 10:46:52 LOSPALAVROS last message repeated 5 times
May  2 10:46:54 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:46:54 LOSPALAVROS last message repeated 2 times
May  2 10:46:57 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:47:22 LOSPALAVROS last message repeated 5 times
May  2 10:47:24 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:47:25 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:47:27 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:47:52 LOSPALAVROS last message repeated 5 times
May  2 10:47:55 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:47:57 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:48:22 LOSPALAVROS last message repeated 5 times
May  2 10:48:25 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:48:27 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:48:52 LOSPALAVROS last message repeated 5 times
May  2 10:48:55 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:48:55 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:48:57 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:49:22 LOSPALAVROS last message repeated 5 times
May  2 10:49:25 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:49:26 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:49:27 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:50:02 LOSPALAVROS last message repeated 7 times
May  2 10:50:22 LOSPALAVROS last message repeated 4 times
May  2 10:50:26 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:50:27 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:50:52 LOSPALAVROS last message repeated 5 times
May  2 10:50:55 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:50:56 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:50:57 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:50:59 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:51:02 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:51:22 LOSPALAVROS last message repeated 4 times
May  2 10:51:26 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:51:26 LOSPALAVROS last message repeated 3 times
May  2 10:51:27 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:51:32 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:51:33 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:51:37 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:51:52 LOSPALAVROS last message repeated 3 times
May  2 10:51:57 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:51:57 LOSPALAVROS last message repeated 2 times
May  2 10:51:57 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:51:58 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:52:02 LOSPALAVROS last message repeated 10 times
May  2 10:52:02 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:52:03 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:52:07 LOSPALAVROS last message repeated 3 times
May  2 10:52:07 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:52:08 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:52:11 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:52:12 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:52:22 LOSPALAVROS last message repeated 2 times
May  2 10:52:27 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:52:27 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:52:27 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:52:32 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:52:57 LOSPALAVROS last message repeated 5 times
May  2 10:52:57 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:53:02 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:53:27 LOSPALAVROS last message repeated 5 times
May  2 10:53:28 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:53:32 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:53:57 LOSPALAVROS last message repeated 5 times
May  2 10:53:58 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:53:58 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:54:02 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:54:27 LOSPALAVROS last message repeated 5 times
May  2 10:54:28 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:54:28 LOSPALAVROS last message repeated 2 times
May  2 10:54:32 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:55:07 LOSPALAVROS last message repeated 7 times
May  2 10:55:27 LOSPALAVROS last message repeated 4 times
May  2 10:55:28 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:55:32 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:55:57 LOSPALAVROS last message repeated 5 times
May  2 10:55:59 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:56:02 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:56:27 LOSPALAVROS last message repeated 5 times
May  2 10:56:29 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:56:29 LOSPALAVROS last message repeated 3 times
May  2 10:56:32 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:56:57 LOSPALAVROS last message repeated 5 times
May  2 10:56:59 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:57:00 LOSPALAVROS last message repeated 2 times
May  2 10:57:02 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:57:27 LOSPALAVROS last message repeated 5 times
May  2 10:57:30 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:57:30 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:57:32 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:58:02 LOSPALAVROS last message repeated 6 times
May  2 10:58:07 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:58:07 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:58:27 LOSPALAVROS last message repeated 4 times
May  2 10:58:30 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:58:32 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:58:57 LOSPALAVROS last message repeated 5 times
May  2 10:59:00 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:59:00 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:59:02 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 10:59:27 LOSPALAVROS last message repeated 5 times
May  2 10:59:30 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:59:31 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 10:59:32 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 11:00:07 LOSPALAVROS last message repeated 7 times
May  2 11:00:27 LOSPALAVROS last message repeated 4 times
May  2 11:00:31 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 11:00:32 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 11:00:57 LOSPALAVROS last message repeated 5 times
May  2 11:01:01 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 11:01:02 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 11:01:27 LOSPALAVROS last message repeated 5 times
May  2 11:01:31 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 11:01:32 LOSPALAVROS last message repeated 3 times
May  2 11:01:32 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 11:01:57 LOSPALAVROS last message repeated 5 times
May  2 11:02:02 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 11:02:02 LOSPALAVROS last message repeated 2 times
May  2 11:02:02 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 11:02:27 LOSPALAVROS last message repeated 5 times
May  2 11:02:32 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 11:02:32 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 11:02:32 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 11:03:07 LOSPALAVROS last message repeated 7 times
May  2 11:03:17 LOSPALAVROS last message repeated 2 times
May  2 11:03:20 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 11:03:22 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 11:03:24 LOSPALAVROS kernel: ------------[ cut here ]------------
May  2 11:03:24 LOSPALAVROS kernel: kernel BUG at vmscan.c:573!
May  2 11:03:24 LOSPALAVROS kernel: invalid operand: 0000
May  2 11:03:24 LOSPALAVROS kernel: ide-cd cdrom parport_pc lp parport autofs tulip keybdev mousedev hid input usb-ohci usbcore ext3 jbd  
May  2 11:03:24 LOSPALAVROS kernel: CPU:    0
May  2 11:03:24 LOSPALAVROS kernel: EIP:    0060:[<c013a6ec>]    Not tainted
May  2 11:03:24 LOSPALAVROS kernel: EFLAGS: 00010206
May  2 11:03:24 LOSPALAVROS kernel:
May  2 11:03:24 LOSPALAVROS kernel: EIP is at refill_inactive_zone [kernel] 0x42c (2.4.20-31.9)
May  2 11:03:24 LOSPALAVROS kernel: eax: ec090c00   ebx: c144e14d   ecx: c0309d00   edx: 00000000
May  2 11:03:24 LOSPALAVROS kernel: esi: 00009e02   edi: c144e131   ebp: 00009e25   esp: c170ff38
May  2 11:03:24 LOSPALAVROS kernel: ds: 0068   es: 0068   ss: 0068
May  2 11:03:24 LOSPALAVROS kernel: Process kswapd (pid: 5, stackpage=c170f000)
May  2 11:03:26 LOSPALAVROS kernel: Stack: 000001d0 00000001 c0309df4 c0309e74 00000008 fffffffe 00000004 000000fb
May  2 11:03:26 LOSPALAVROS kernel:        00013aeb 00000001 00003c80 c0309d00 00000000 00000100 c013b1bb c0309d00
May  2 11:03:26 LOSPALAVROS kernel:        00000000 00000045 c0309d00 00000008 00000100 0000000f c013b20f c0309d00
May  2 11:03:26 LOSPALAVROS kernel: Call Trace:   [<c013b1bb>] rebalance_inactive_zone [kernel] 0xcb (0xc170ff70))
May  2 11:03:26 LOSPALAVROS kernel: [<c013b20f>] rebalance_inactive [kernel] 0x3f (0xc170ff90))
May  2 11:03:26 LOSPALAVROS kernel: [<c013b351>] do_try_to_free_pages_kswapd [kernel] 0x51 (0xc170ffb4))
May  2 11:03:26 LOSPALAVROS kernel: [<c013b5c3>] kswapd [kernel] 0x83 (0xc170ffd4))
May  2 11:03:26 LOSPALAVROS kernel: [<c013b540>] kswapd [kernel] 0x0 (0xc170ffe4))
May  2 11:03:26 LOSPALAVROS kernel: [<c010727d>] kernel_thread_helper [kernel] 0x5 (0xc170fff0))
May  2 11:03:26 LOSPALAVROS kernel:
May  2 11:03:26 LOSPALAVROS kernel:
May  2 11:03:26 LOSPALAVROS kernel: Code: 0f 0b 3d 02 4e b7 25 c0 e9 67 fe ff ff 8d b4 26 00 00 00 00
May  2 11:03:27 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 11:03:30 LOSPALAVROS kernel:  <1>Unable to handle kernel paging request at virtual address 309e74ec
May  2 11:03:30 LOSPALAVROS kernel:  printing eip:
May  2 11:03:30 LOSPALAVROS kernel: c013a992
May  2 11:03:30 LOSPALAVROS kernel: *pde = 00000000
May  2 11:03:30 LOSPALAVROS kernel: Oops: 0000
May  2 11:03:30 LOSPALAVROS kernel: ide-cd cdrom parport_pc lp parport autofs tulip keybdev mousedev hid input usb-ohci usbcore ext3 jbd  
May  2 11:03:30 LOSPALAVROS kernel: CPU:    0
May  2 11:03:30 LOSPALAVROS kernel: EIP:    0060:[<c013a992>]    Not tainted
May  2 11:03:30 LOSPALAVROS kernel: EFLAGS: 00010246
May  2 11:03:30 LOSPALAVROS kernel:
May  2 11:03:30 LOSPALAVROS kernel: EIP is at scan_active_list [kernel] 0x42 (2.4.20-31.9)
May  2 11:03:30 LOSPALAVROS kernel: eax: 309e74ec   ebx: c144e130   ecx: ca369bc0   edx: dffb7fb4
May  2 11:03:30 LOSPALAVROS kernel: esi: c0309e74   edi: 309e74ec   ebp: 00000000   esp: dffb7fac
May  2 11:03:31 LOSPALAVROS kernel: ds: 0068   es: 0068   ss: 0068
May  2 11:03:31 LOSPALAVROS kernel: Process kscand/Normal (pid: 7, stackpage=dffb7000)
May  2 11:03:31 LOSPALAVROS kernel: Stack: dffb6000 c01254e0 00000001 00000000 dffb6000 c0309d00 dffb6000 c013bb34
May  2 11:03:31 LOSPALAVROS kernel:        c0309d00 00000000 00000000 c025b799 000009c4 c013ba40 00000000 00000000
May  2 11:03:31 LOSPALAVROS kernel:        00000000 c010727d c0309d00 00000000 00000000
May  2 11:03:31 LOSPALAVROS kernel: Call Trace:   [<c01254e0>] process_timeout [kernel] 0x0 (0xdffb7fb0))
May  2 11:03:31 LOSPALAVROS kernel: [<c013bb34>] kscand [kernel] 0xf4 (0xdffb7fc8))
May  2 11:03:31 LOSPALAVROS kernel: [<c013ba40>] kscand [kernel] 0x0 (0xdffb7fe0))
May  2 11:03:31 LOSPALAVROS kernel: [<c010727d>] kernel_thread_helper [kernel] 0x5 (0xdffb7ff0))
May  2 11:03:31 LOSPALAVROS kernel:
May  2 11:03:31 LOSPALAVROS kernel:
May  2 11:03:31 LOSPALAVROS kernel: Code: 8b 3f 39 f0 75 de 83 c4 0c 31 c0 5b 5e 5f 5d c3 89 6c 24 04
May  2 11:03:32 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 11:03:33 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 11:03:33 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 11:03:37 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
May  2 11:04:02 LOSPALAVROS last message repeated 5 times
May  2 11:04:03 LOSPALAVROS vsftpd: warning: can't get client address: Bad file descriptor
May  2 11:04:07 LOSPALAVROS pam_timestamp_check: pam_timestamp: `/' owner UID != 0
eggster34Asked:
Who is Participating?

[Webinar] Streamline your web hosting managementRegister Today

x
 
m1tk4Connect With a Mentor Commented:
>> Swap: 1044216k av,   54000k used,  990216k free                  234304k cached

So it's not full, most likely it's either HDD or memory failure.

>> should I not see the swap partition in the df -h results?

No, since df shows the usage of mounted volumes (including NFS/SMB ones for example), not your HDD partitions.
0
 
ravenplCommented:
> May  2 11:03:24 LOSPALAVROS kernel: kernel BUG at vmscan.c:573!
It's very propable, that server's memory is broken
check with http://www.memtest.org/
0
 
m1tk4Commented:
Another possibility is that your hard drive is failing and these kernel errors are a result of swap failures.

If your memory test passes, try checking the smartd/smartctl information and then fsck.
0
Never miss a deadline with monday.com

The revolutionary project management tool is here!   Plan visually with a single glance and make sure your projects get done.

 
ppfoongCommented:

Process kswapd is causing the core dump, so it is related to the swapping.

Process kswapd (pid: 5, stackpage=c170f000)


The error message is:

<1>Unable to handle kernel paging request at virtual address 309e74ec


Possibilities:
1. There is bad sector in your swap partition.
2. Your swap space has full.

0
 
Dragon_KromeCommented:
also, there seems root doesn't own /  -  you should do:

chown root:root /
chmod 755 /

and check for intruders on the system with chkrootkit and/or rkhunter.
0
 
eggster34Author Commented:
mmm
how can I check my swap space if it's full or not? how can I extend it?
0
 
Dragon_KromeCommented:
IMHO the linux kernel is not a toy program that will crash at a mere condition like swap full. Try the "free" command to see how much you've got left anyway. Most likely you've got a hardware problem or some device driver causing trouble.
0
 
m1tk4Commented:
>> IMHO the linux kernel is not a toy program that will crash at a mere condition like swap full.

Easily. You can just do what I did last week - let Apache maximum # of child processes reach 50 with each taking 128M of memory;)) -> kernel panic ;)
0
 
Dragon_KromeCommented:
Sorry, never happened to me. But i'll take your word for it.  :)
0
 
m1tk4Commented:
>>how can I check my swap space if it's full or not?

run "top" - it shows it in 3rd or 4th line

>>how can I extend it?

resize swap partition on your hard drive.
0
 
eggster34Author Commented:
here's my top:

 09:32:06  up 1 day, 11 min,  2 users,  load average: 0.02, 0.04, 0.00
83 processes: 81 sleeping, 2 running, 0 zombie, 0 stopped
CPU states:  19.5% user   1.1% system   0.0% nice   0.0% iowait  79.2% idle
Mem:   513828k av,  495044k used,   18784k free,       0k shrd,   99064k buff
                    342464k actv,       0k in_d,    8648k in_c
Swap: 1044216k av,   54000k used,  990216k free                  234304k cached

and my df -h:

Filesystem            Size  Used Avail Use% Mounted on
/dev/hdc1              37G   14G   21G  40% /
/dev/hda1              99M   14M   80M  15% /boot
none                  251M     0  251M   0% /dev/shm

should I not see the swap partition in the df -h results?
0
 
ravenplCommented:
Why will not You start from: check with http://www.memtest.org/
It's common that memory is broken, so it would be nice to exclude this case for sure before looking else places...
0
 
nls73mCommented:
seems like a hardware problem, maybe redhat 9.0 is telling you to upgrade your box? when hardware fails, that is the best time to upgrade hardware and software, maybe centos?
0
 
eggster34Author Commented:
I actually replaced my memory and my network card.
the only thing I have not replaced is my hdd.
0
All Courses

From novice to tech pro — start learning today.