• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 892
  • Last Modified:

Need help understanding var/log/messages

Hello,

My server has gone down twice this week, the cpu loads have been higher than normal (in spikes) when it is running, and I'm trying to figure out why.

Below are the messages in var/log/messages just before it crashed this morning. Can you take a look and tell me what this means? Does this indicate what the problem may be?

Thank you,    Chris

=================

Jan  7 05:07:00 jessica named[1393]: unexpected RCODE (REFUSED) resolving 'branto.in/MX/IN': 213.180.193.1#53
Jan  7 05:07:00 jessica named[1393]: unexpected RCODE (REFUSED) resolving 'branto.in/MX/IN': 213.180.199.34#53
Jan  7 05:07:00 jessica named[1393]: network unreachable resolving 'branto.in/MX/IN': 2a02:6b8::1#53
Jan  7 05:07:02 jessica named[1393]: lame server resolving '119.121.241.180.in-addr.arpa' (in '121.241.180.in-addr.arpa'?): 222.124.18.62#53
Jan  7 05:07:02 jessica named[1393]: lame server resolving '119.121.241.180.in-addr.arpa' (in '121.241.180.in-addr.arpa'?): 202.134.0.62#53
Jan  7 05:11:00 jessica named[1393]: lame server resolving '153-122.webdrivesystems.com' (in 'webdrivesystems.com'?): 65.23.154.110#53
Jan  7 05:11:00 jessica named[1393]: lame server resolving '153-122.webdrivesystems.com' (in 'webdrivesystems.com'?): 65.23.128.7#53
Jan  7 05:11:00 jessica named[1393]: lame server resolving '153-122.webdrivesystems.com' (in 'webdrivesystems.com'?): 65.23.128.6#53
Jan  7 05:11:02 jessica named[1393]: lame server resolving '153-122.webdrivesystems.com' (in 'webdrivesystems.com'?): 65.23.128.6#53
Jan  7 05:11:02 jessica named[1393]: lame server resolving '153-122.webdrivesystems.com' (in 'webdrivesystems.com'?): 65.23.128.7#53
Jan  7 05:11:02 jessica named[1393]: lame server resolving '153-122.webdrivesystems.com' (in 'webdrivesystems.com'?): 65.23.154.110#53
Jan  7 05:12:06 jessica clamd[3111]: SelfCheck: Database status OK.
Jan  7 05:22:07 jessica clamd[3111]: SelfCheck: Database status OK.
Jan  7 05:37:05 jessica clamd[3111]: SelfCheck: Database status OK.
Jan  7 05:37:56 jessica named[1393]: lame server resolving '111.69.205.82.in-addr.arpa' (in '69.205.82.in-addr.arpa'?): 192.116.16.26#53
Jan  7 05:37:56 jessica named[1393]: lame server resolving '111.69.205.82.in-addr.arpa' (in '69.205.82.in-addr.arpa'?): 217.66.226.8#53
Jan  7 05:37:57 jessica named[1393]: lame server resolving '111.69.205.82.in-addr.arpa' (in '69.205.82.in-addr.arpa'?): 217.66.226.8#53
Jan  7 05:37:57 jessica named[1393]: lame server resolving '111.69.205.82.in-addr.arpa' (in '69.205.82.in-addr.arpa'?): 192.116.16.26#53
Jan  7 05:43:23 jessica named[1393]: network unreachable resolving 'apache.org/NS/IN': 2001:500:f::1#53
Jan  7 05:43:23 jessica named[1393]: network unreachable resolving 'apache.org/NS/IN': 2001:500:e::1#53
Jan  7 05:43:23 jessica named[1393]: network unreachable resolving 'apache.org/NS/IN': 2001:500:48::1#53
Jan  7 05:43:23 jessica named[1393]: network unreachable resolving 'apache.org/NS/IN': 2001:500:c::1#53
Jan  7 05:43:23 jessica named[1393]: network unreachable resolving 'apache.org/NS/IN': 2001:610:3:200a:192:87:36:2#53
Jan  7 05:43:23 jessica named[1393]: network unreachable resolving 'apache.org/NS/IN': 2620:0:2e60::33#53
Jan  7 05:43:50 jessica named[1393]: network unreachable resolving 'comcast.net/NS/IN': 2001:558:1004:7:68:87:85:132#53
Jan  7 05:43:50 jessica named[1393]: network unreachable resolving 'comcast.net/NS/IN': 2001:558:1002:a:68:87:29:164#53
Jan  7 05:47:07 jessica clamd[3111]: SelfCheck: Database status OK.
Jan  7 05:56:59 jessica named[1393]: unexpected RCODE (REFUSED) resolving 'fronomi.in/MX/IN': 213.180.193.1#53
Jan  7 05:56:59 jessica named[1393]: unexpected RCODE (REFUSED) resolving 'fronomi.in/MX/IN': 213.180.199.34#53
Jan  7 05:56:59 jessica named[1393]: network unreachable resolving 'fronomi.in/MX/IN': 2a02:6b8::1#53
Jan  7 05:56:59 jessica named[1393]: unexpected RCODE (REFUSED) resolving 'fronomi.in/MX/IN': 213.180.193.1#53
Jan  7 05:56:59 jessica named[1393]: unexpected RCODE (REFUSED) resolving 'fronomi.in/MX/IN': 213.180.199.34#53
Jan  7 05:56:59 jessica named[1393]: network unreachable resolving 'fronomi.in/MX/IN': 2a02:6b8::1#53
Jan  7 05:57:00 jessica named[1393]: unexpected RCODE (REFUSED) resolving 'fronomi.in/MX/IN': 213.180.193.1#53
Jan  7 05:57:00 jessica named[1393]: unexpected RCODE (REFUSED) resolving 'fronomi.in/MX/IN': 213.180.199.34#53
Jan  7 05:57:00 jessica named[1393]: network unreachable resolving 'fronomi.in/MX/IN': 2a02:6b8::1#53
Jan  7 05:57:00 jessica named[1393]: unexpected RCODE (REFUSED) resolving 'adesbi.in/MX/IN': 213.180.193.1#53
Jan  7 05:57:00 jessica named[1393]: unexpected RCODE (REFUSED) resolving 'fronomi.in/MX/IN': 213.180.193.1#53
Jan  7 05:57:01 jessica named[1393]: unexpected RCODE (REFUSED) resolving 'adesbi.in/MX/IN': 213.180.199.34#53
Jan  7 05:57:02 jessica named[1393]: unexpected RCODE (REFUSED) resolving 'fronomi.in/MX/IN': 213.180.199.34#53
Jan  7 05:57:03 jessica named[1393]: network unreachable resolving 'adesbi.in/MX/IN': 2a02:6b8::1#53
Jan  7 05:57:03 jessica named[1393]: network unreachable resolving 'fronomi.in/MX/IN': 2a02:6b8::1#53
Jan  7 05:57:04 jessica named[1393]: unexpected RCODE (REFUSED) resolving 'adesbi.in/MX/IN': 213.180.199.34#53
Jan  7 05:57:04 jessica named[1393]: unexpected RCODE (REFUSED) resolving 'adesbi.in/MX/IN': 213.180.193.1#53
Jan  7 05:57:04 jessica named[1393]: network unreachable resolving 'adesbi.in/MX/IN': 2a02:6b8::1#53
Jan  7 05:57:04 jessica named[1393]: unexpected RCODE (REFUSED) resolving 'adesbi.in/MX/IN': 213.180.193.1#53
Jan  7 05:57:05 jessica named[1393]: unexpected RCODE (REFUSED) resolving 'adesbi.in/MX/IN': 213.180.199.34#53
Jan  7 05:57:05 jessica named[1393]: network unreachable resolving 'adesbi.in/MX/IN': 2a02:6b8::1#53
Jan  7 05:57:06 jessica named[1393]: unexpected RCODE (REFUSED) resolving 'adesbi.in/MX/IN': 213.180.199.34#53
Jan  7 05:57:06 jessica named[1393]: unexpected RCODE (REFUSED) resolving 'adesbi.in/MX/IN': 213.180.193.1#53
Jan  7 05:57:06 jessica named[1393]: network unreachable resolving 'adesbi.in/MX/IN': 2a02:6b8::1#53
Jan  7 05:57:08 jessica clamd[3111]: SelfCheck: Database status OK.
Jan  7 06:06:16 jessica named[1393]: network unreachable resolving '8.119.78.203.in-addr.arpa/PTR/IN': 2001:500:13::c7d4:35#53
Jan  7 06:12:18 jessica kernel: INFO: task auditd:1327 blocked for more than 120 seconds.
Jan  7 06:13:20 jessica kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  7 06:13:20 jessica kernel: auditd        D 00095efd7d396367     0  1327      1          1347  1326 (NOTLB)
Jan  7 06:13:22 jessica kernel:  ffff8800a0d4dd98  0000000000000286  ffff8800a0d4dda8  ffffffff80261dd5
Jan  7 06:13:23 jessica kernel:  0000000000000009  ffff8800a0fb3080  ffffffff804ffb80  00000000000019bb
Jan  7 06:13:30 jessica kernel:  ffff8800a0fb3268  ffff8800a3822860
Jan  7 06:13:30 jessica kernel: Call Trace:
Jan  7 06:13:31 jessica kernel:  [<ffffffff80261dd5>] thread_return+0x6c/0x113
Jan  7 06:13:32 jessica kernel:  [<ffffffff88036de4>] :jbd:log_wait_commit+0xa3/0xf5
Jan  7 06:13:32 jessica kernel:  [<ffffffff8029d71a>] autoremove_wake_function+0x0/0x2e
Jan  7 06:13:32 jessica kernel:  [<ffffffff8029437b>] process_timeout+0x0/0x5
Jan  7 06:13:32 jessica kernel:  [<ffffffff8803179a>] :jbd:journal_stop+0x1d3/0x203
Jan  7 06:13:32 jessica kernel:  [<ffffffff80230eb0>] __writeback_single_inode+0x1dd/0x31c
Jan  7 06:13:32 jessica kernel:  [<ffffffff802e7c0d>] sync_inode+0x24/0x33
Jan  7 06:13:32 jessica kernel:  [<ffffffff8804c37e>] :ext3:ext3_sync_file+0xce/0xf8
Jan  7 06:13:32 jessica kernel:  [<ffffffff802519dd>] do_fsync+0x52/0xa4
Jan  7 06:13:32 jessica kernel:  [<ffffffff802d5752>] __do_fsync+0x23/0x36
Jan  7 06:13:32 jessica kernel:  [<ffffffff8025f106>] system_call+0x86/0x8b
Jan  7 06:13:32 jessica kernel:  [<ffffffff8025f080>] system_call+0x0/0x8b
Jan  7 06:13:32 jessica named[1393]: network unreachable resolving 'ns3.apnic.net/A/IN': 2001:dc0:1:0:4777::131#53
Jan  7 06:13:32 jessica kernel:
Jan  7 06:13:36 jessica kernel: INFO: task syslogd:1347 blocked for more than 120 seconds.
Jan  7 06:13:36 jessica named[1393]: network unreachable resolving 'ns4.apnic.net/A/IN': 2001:dc0:1:0:4777::131#53
Jan  7 06:13:36 jessica kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  7 06:13:38 jessica kernel: syslogd       D 00095efd7d339583     0  1347      1          1351  1327 (NOTLB)
Jan  7 06:13:38 jessica kernel:  ffff8800a113fd98  0000000000000286  ffff8800a113fda8  ffffffff80261dd5
Jan  7 06:13:38 jessica kernel:  0000000000000009  ffff8800a211f820  ffff88000675c040  00000000000019b8
Jan  7 06:13:38 jessica kernel:  ffff8800a211fa08  ffff88000675c040
Jan  7 06:13:38 jessica kernel: Call Trace:
Jan  7 06:13:40 jessica kernel:  [<ffffffff80261dd5>] thread_return+0x6c/0x113
Jan  7 06:13:42 jessica kernel:  [<ffffffff88036de4>] :jbd:log_wait_commit+0xa3/0xf5
Jan  7 06:13:42 jessica kernel:  [<ffffffff8029d71a>] autoremove_wake_function+0x0/0x2e
Jan  7 06:13:42 jessica kernel:  [<ffffffff8803179a>] :jbd:journal_stop+0x1d3/0x203
Jan  7 06:13:42 jessica kernel:  [<ffffffff80230eb0>] __writeback_single_inode+0x1dd/0x31c
Jan  7 06:13:45 jessica kernel:  [<ffffffff802d4f45>] do_readv_writev+0x26e/0x291
Jan  7 06:13:46 jessica kernel:  [<ffffffff802e7c0d>] sync_inode+0x24/0x33
Jan  7 06:13:46 jessica kernel:  [<ffffffff8804c37e>] :ext3:ext3_sync_file+0xce/0xf8
Jan  7 06:13:46 jessica kernel:  [<ffffffff802519dd>] do_fsync+0x52/0xa4
Jan  7 06:13:46 jessica kernel:  [<ffffffff802d5752>] __do_fsync+0x23/0x36
Jan  7 06:13:48 jessica kernel:  [<ffffffff8025f2f9>] tracesys+0xab/0xb6
Jan  7 06:13:48 jessica kernel:
Jan  7 06:14:52 jessica clamd[3111]: SelfCheck: Database status OK.
Jan  7 06:14:52 jessica named[1393]: socket: Too many open files in system
Jan  7 06:14:57 jessica named[1393]: network unreachable resolving '8.119.78.203.in-addr.arpa/PTR/IN': 2001:dc0:1:0:4777::131#53
Jan  7 06:22:13 jessica kernel: INFO: task httpd:8327 blocked for more than 120 seconds.
Jan  7 06:22:14 jessica kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  7 06:22:14 jessica kernel: httpd         D 0000000000000246     0  8327   2634  8833    8331  8326 (NOTLB)
Jan  7 06:22:14 jessica kernel:  ffff880057399d28  0000000000000286  000000004f082a33  ffff880067c3e5f8
Jan  7 06:22:14 jessica kernel:  0000000000000007  ffff8800925b57a0  ffff880058004080  0000000000016b1f
Jan  7 06:22:14 jessica kernel:  ffff8800925b5988  ffff880007657130
Jan  7 06:22:14 jessica kernel: Call Trace:
Jan  7 06:22:14 jessica kernel:  [<ffffffff8026274e>] __wait_on_bit_lock+0x5b/0x66
Jan  7 06:22:14 jessica kernel:  [<ffffffff80211c68>] do_wp_page+0x3e9/0x8e8
Jan  7 06:22:14 jessica kernel:  [<ffffffff80262a6a>] __mutex_lock_slowpath+0x60/0x9b
Jan  7 06:22:14 jessica kernel:  [<ffffffff80262ab4>] .text.lock.mutex+0xf/0x14
Jan  7 06:22:14 jessica kernel:  [<ffffffff8022235e>] generic_file_aio_write+0x50/0xc3
Jan  7 06:22:15 jessica kernel:  [<ffffffff8804c1c2>] :ext3:ext3_file_write+0x16/0x91
Jan  7 06:30:30 jessica kernel:  [<ffffffff80218936>] do_sync_write+0xc7/0x104
Jan  7 06:30:31 jessica kernel:  [<ffffffff80263909>] _spin_lock_irqsave+0x9/0x14
Jan  7 06:30:31 jessica kernel:  [<ffffffff80266da7>] do_page_fault+0xfa5/0x131b
Jan  7 06:30:31 jessica kernel:  [<ffffffff8029d71a>] autoremove_wake_function+0x0/0x2e
Jan  7 06:30:31 jessica kernel:  [<ffffffff8022edb7>] __wake_up+0x38/0x4f
Jan  7 06:30:31 jessica kernel:  [<ffffffff8021749f>] vfs_write+0xce/0x174
Jan  7 06:30:31 jessica kernel:  [<ffffffff80217ce8>] sys_write+0x45/0x6e
Jan  7 06:30:31 jessica kernel:  [<ffffffff8025f2f9>] tracesys+0xab/0xb6
Jan  7 06:30:31 jessica kernel:
Jan  7 06:30:31 jessica kernel: INFO: task httpd:8703 blocked for more than 120 seconds.
Jan  7 06:30:31 jessica kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  7 06:30:31 jessica kernel: httpd         D 0000000000000246     0  8703   2634  8787    8716  8700 (NOTLB)
Jan  7 06:30:32 jessica kernel:  ffff880096ed5d28  0000000000000282  000000004f082a4e  ffff88004e4813f8
Jan  7 06:30:32 jessica kernel:  0000000000000007  ffff880046d14080  ffff8800701af080  0000000000007955
Jan  7 06:30:32 jessica kernel:  ffff880046d14268  ffffffff0000000f
Jan  7 06:30:32 jessica kernel: Call Trace:
Jan  7 06:30:32 jessica kernel:  [<ffffffff802b7a6d>] delayacct_end+0x5d/0x86
Jan  7 06:30:32 jessica kernel:  [<ffffffff80262a6a>] __mutex_lock_slowpath+0x60/0x9b
Jan  7 06:30:32 jessica kernel:  [<ffffffff80262ab4>] .text.lock.mutex+0xf/0x14
Jan  7 06:30:32 jessica kernel:  [<ffffffff8022235e>] generic_file_aio_write+0x50/0xc3
Jan  7 06:30:32 jessica kernel:  [<ffffffff8804c1c2>] :ext3:ext3_file_write+0x16/0x91
Jan  7 06:30:28 jessica clamd[3111]: SelfCheck: Database status OK.
Jan  7 06:30:32 jessica kernel:  [<ffffffff80218936>] do_sync_write+0xc7/0x104
Jan  7 06:30:32 jessica kernel:  [<ffffffff80263909>] _spin_lock_irqsave+0x9/0x14
Jan  7 06:30:32 jessica kernel:  [<ffffffff80266da7>] do_page_fault+0xfa5/0x131b
Jan  7 06:30:34 jessica kernel:  [<ffffffff8029d71a>] autoremove_wake_function+0x0/0x2e
Jan  7 06:30:35 jessica kernel:  [<ffffffff8022edb7>] __wake_up+0x38/0x4f
Jan  7 06:30:35 jessica kernel:  [<ffffffff8021749f>] vfs_write+0xce/0x174
Jan  7 06:30:35 jessica kernel:  [<ffffffff80217ce8>] sys_write+0x45/0x6e
Jan  7 06:30:36 jessica kernel:  [<ffffffff8025f2f9>] tracesys+0xab/0xb6
Jan  7 06:30:37 jessica kernel:
Jan  7 06:30:41 jessica kernel: INFO: task httpd:8326 blocked for more than 120 seconds.
Jan  7 06:30:41 jessica kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  7 06:30:47 jessica kernel: httpd         D 0000000000000246     0  8326   2634  8895    8327  8325 (NOTLB)
Jan  7 06:30:52 jessica kernel:  ffff88004a64fd28  0000000000000282  ffff88006c40ac80  ffff8800a3797400
Jan  7 06:30:52 jessica kernel:  000000000000000a  ffff88003face100  ffff8800750457a0  000000000001a0c3
Jan  7 06:30:52 jessica kernel:  ffff88003face2e8  0000000000000000
Jan  7 06:30:55 jessica kernel: Call Trace:
Jan  7 06:30:56 jessica kernel:  [<ffffffff80211c68>] do_wp_page+0x3e9/0x8e8
Jan  7 06:30:57 jessica kernel:  [<ffffffff80262a6a>] __mutex_lock_slowpath+0x60/0x9b
Jan  7 06:30:57 jessica kernel:  [<ffffffff80262ab4>] .text.lock.mutex+0xf/0x14
Jan  7 06:31:11 jessica kernel:  [<ffffffff8022235e>] generic_file_aio_write+0x50/0xc3
Jan  7 06:31:12 jessica kernel:  [<ffffffff8804c1c2>] :ext3:ext3_file_write+0x16/0x91
Jan  7 06:31:12 jessica kernel:  [<ffffffff80218936>] do_sync_write+0xc7/0x104
Jan  7 06:31:12 jessica kernel:  [<ffffffff80263909>] _spin_lock_irqsave+0x9/0x14
Jan  7 06:31:13 jessica kernel:  [<ffffffff80266da7>] do_page_fault+0xfa5/0x131b
Jan  7 06:31:13 jessica kernel:  [<ffffffff8029d71a>] autoremove_wake_function+0x0/0x2e
Jan  7 06:31:13 jessica kernel:  [<ffffffff8029437b>] process_timeout+0x0/0x5
Jan  7 06:31:17 jessica kernel:  [<ffffffff802639c8>] _write_lock_irqsave+0x9/0x1c
Jan  7 06:31:18 jessica kernel:  [<ffffffff802ed0c2>] sys_epoll_wait+0x3b8/0x3f9
Jan  7 06:31:19 jessica kernel:  [<ffffffff8021749f>] vfs_write+0xce/0x174
Jan  7 06:31:20 jessica kernel:  [<ffffffff80217ce8>] sys_write+0x45/0x6e
Jan  7 06:31:23 jessica kernel:  [<ffffffff8025f2f9>] tracesys+0xab/0xb6
Jan  7 06:31:23 jessica kernel:
Jan  7 06:31:24 jessica kernel: INFO: task httpd:8340 blocked for more than 120 seconds.
Jan  7 06:31:24 jessica kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  7 06:31:24 jessica kernel: httpd         D 0000000000000246     0  8340   2634  8882    8343  8339 (NOTLB)
Jan  7 06:31:24 jessica kernel:  ffff88006c31dd28  0000000000000286  000000004f082a14  00000000195530a4
Jan  7 06:31:26 jessica kernel:  000000000000000a  ffff880085e25820  ffff8800871aa820  000000000001acab
Jan  7 06:31:39 jessica kernel:  ffff880085e25a08  0000000000000000
Jan  7 06:31:40 jessica kernel: Call Trace:
Jan  7 06:31:40 jessica kernel:  [<ffffffff80211c68>] do_wp_page+0x3e9/0x8e8
Jan  7 06:31:40 jessica kernel:  [<ffffffff80262a6a>] __mutex_lock_slowpath+0x60/0x9b
Jan  7 06:31:40 jessica kernel:  [<ffffffff80262ab4>] .text.lock.mutex+0xf/0x14
Jan  7 06:31:41 jessica kernel:  [<ffffffff8022235e>] generic_file_aio_write+0x50/0xc3
Jan  7 06:31:41 jessica kernel:  [<ffffffff8804c1c2>] :ext3:ext3_file_write+0x16/0x91
Jan  7 06:31:41 jessica kernel:  [<ffffffff80218936>] do_sync_write+0xc7/0x104
Jan  7 06:31:41 jessica kernel:  [<ffffffff80263909>] _spin_lock_irqsave+0x9/0x14
Jan  7 06:31:41 jessica kernel:  [<ffffffff80266da7>] do_page_fault+0xfa5/0x131b
Jan  7 06:31:41 jessica kernel:  [<ffffffff8029d71a>] autoremove_wake_function+0x0/0x2e
Jan  7 06:31:41 jessica kernel:  [<ffffffff8029437b>] process_timeout+0x0/0x5
Jan  7 06:31:41 jessica kernel:  [<ffffffff802639c8>] _write_lock_irqsave+0x9/0x1c
Jan  7 06:31:43 jessica kernel:  [<ffffffff802ed0c2>] sys_epoll_wait+0x3b8/0x3f9
Jan  7 06:31:46 jessica kernel:  [<ffffffff8021749f>] vfs_write+0xce/0x174
Jan  7 06:31:46 jessica kernel:  [<ffffffff80217ce8>] sys_write+0x45/0x6e
Jan  7 06:31:46 jessica kernel:  [<ffffffff8025f2f9>] tracesys+0xab/0xb6
Jan  7 06:31:48 jessica kernel:
Jan  7 06:31:48 jessica kernel: INFO: task httpd:8343 blocked for more than 120 seconds.
Jan  7 06:32:18 jessica kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  7 06:32:18 jessica kernel: httpd         D 0000000000000246     0  8343   2634  8899    8347  8340 (NOTLB)
Jan  7 06:32:19 jessica kernel:  ffff880099a1bd28  0000000000000286  000000004f082aa1  ffff8800a17de778
Jan  7 06:32:34 jessica kernel:  0000000000000009  ffff88004a0c97a0  ffff8800722eb860  0000000000009d8d
Jan  7 06:32:34 jessica kernel:  ffff88004a0c9988  ffff880007a02d68
Jan  7 06:32:35 jessica kernel: Call Trace:
Jan  7 06:32:35 jessica kernel:  [<ffffffff8026274e>] __wait_on_bit_lock+0x5b/0x66
Jan  7 06:32:35 jessica kernel:  [<ffffffff8024164d>] __lock_page+0x5e/0x64
Jan  7 06:32:35 jessica kernel:  [<ffffffff80262a6a>] __mutex_lock_slowpath+0x60/0x9b
Jan  7 06:32:35 jessica kernel:  [<ffffffff80262ab4>] .text.lock.mutex+0xf/0x14
Jan  7 06:32:35 jessica kernel:  [<ffffffff8022235e>] generic_file_aio_write+0x50/0xc3
Jan  7 06:32:35 jessica kernel:  [<ffffffff8804c1c2>] :ext3:ext3_file_write+0x16/0x91
Jan  7 06:32:35 jessica kernel:  [<ffffffff80218936>] do_sync_write+0xc7/0x104
Jan  7 06:32:35 jessica kernel:  [<ffffffff80263909>] _spin_lock_irqsave+0x9/0x14
Jan  7 06:32:36 jessica kernel:  [<ffffffff80266da7>] do_page_fault+0xfa5/0x131b
Jan  7 06:32:36 jessica kernel:  [<ffffffff8029d71a>] autoremove_wake_function+0x0/0x2e
Jan  7 06:32:36 jessica kernel:  [<ffffffff8029437b>] process_timeout+0x0/0x5
Jan  7 06:32:36 jessica kernel:  [<ffffffff802639c8>] _write_lock_irqsave+0x9/0x1c
Jan  7 06:32:36 jessica kernel:  [<ffffffff802ed0c2>] sys_epoll_wait+0x3b8/0x3f9
Jan  7 06:32:36 jessica kernel:  [<ffffffff8021749f>] vfs_write+0xce/0x174
Jan  7 06:32:36 jessica kernel:  [<ffffffff80217ce8>] sys_write+0x45/0x6e
Jan  7 06:32:36 jessica kernel:  [<ffffffff8025f2f9>] tracesys+0xab/0xb6
Jan  7 06:32:36 jessica kernel:
Jan  7 06:32:36 jessica kernel: INFO: task httpd:8370 blocked for more than 120 seconds.
Jan  7 06:32:36 jessica kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan  7 06:32:36 jessica kernel: httpd         D 0000000000000246     0  8370   2634  8883    8373  8369 (NOTLB)
Jan  7 06:32:37 jessica kernel:  ffff880081331d28  0000000000000286  000000004f082a15  000000000b07ce30
Jan  7 06:32:37 jessica kernel:  000000000000000a  ffff880073f3e7a0  ffff88004a342040  000000000001dc3e
Jan  7 06:32:37 jessica kernel:  ffff880073f3e988  0000000000000000
Jan  7 06:32:37 jessica kernel: Call Trace:
Jan  7 06:32:37 jessica kernel:  [<ffffffff80211c68>] do_wp_page+0x3e9/0x8e8

at this point, messages stopped until the server was rebooted.
 
0
St_Aug_Beach_Bum
Asked:
St_Aug_Beach_Bum
  • 2
1 Solution
 
viju2008Commented:
http://www.sibbald.com/unixutil/logalizer.html

Try this it helps in understanding unix system log files . Logalizer is a command line utility

http://www.sawmill.net/formats/unix_syslog.html

Sawmill can parse Open Source Unix Syslog logs, import them into a MySQL, Microsoft SQL Server, or Oracle database (or its own built-in database), aggregate them, and generate dynamically filtered reports, all through a web interface

Here are some other tools which could be of use ..

http://www.mela.de/Unix/log.html
0
 
woolmilkporcCommented:
This seems to be the major issue:

socket: Too many open files in system

Try to identify the responsible process (using "lsof").

If you think that your system should allow for more open files
change the limit by editing /etc/sysctl.conf and set fs.file-max to a higher value.

Then run

sysctl -p

to apply the change to your system.


wmp

0
 
woolmilkporcCommented:
Number of open files per process:

lsof | awk '{print $2}' | sort -n | uniq -c | sort -n

The number of open files is in column 1, the process id is in column 2

Total number of open files:

lsof | wc -l

wmp
0
 
St_Aug_Beach_BumAuthor Commented:
Wmp - thank you very much, that has given me some solid leads I am following up on, running tests and trying to figure it out :)

Appreciate it!

Chris
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Cloud Class® Course: Certified Penetration Testing

This CPTE Certified Penetration Testing Engineer course covers everything you need to know about becoming a Certified Penetration Testing Engineer. Career Path: Professional roles include Ethical Hackers, Security Consultants, System Administrators, and Chief Security Officers.

  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now