Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!
Mysqld Crash Autoposy Help Needed
We are running a LAMP server with CentOS 6 which serves about 50 (low to medium traffic) websites, mostly wordpress. Today we found the majority of WP sites were not working and upon investigation found InnoDB errors for those databases. Restarting mysql fixed the problem. Now I want to figure out exactly why.
In looking at the mysql error log I saw that there was no process running, and mysqld_safe tried to restart but failed on InnoDB due to no memory buffer available.
151125 12:34:45 mysqld_safe Number of processes running now: 0
151125 12:34:45 mysqld_safe mysqld restarted
151125 12:34:46 [Note] /usr/sbin/mysqld (mysqld 5.5.46-cll) starting as process 8159 ...
151125 12:34:46 [Note] Plugin 'FEDERATED' is disabled.
151125 12:34:46 InnoDB: The InnoDB memory heap is disabled
151125 12:34:46 InnoDB: Mutexes and rw_locks use GCC atomic builtins
151125 12:34:46 InnoDB: Compressed tables use zlib 1.2.3
151125 12:34:46 InnoDB: Using Linux native AIO
151125 12:34:46 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
151125 12:34:46 InnoDB: Completed initialization of buffer pool
151125 12:34:46 InnoDB: Fatal error: cannot allocate memory for the buffer pool
151125 12:34:46 Plugin 'InnoDB' init function returned error.
151125 12:34:46 Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
I check the syslog and found that the system killed the process due to lack of memory, as follows:
/var/log/messages
Nov 25 12:33:26 web5 kernel: [2676096.344431] Out of memory: Kill process 15004 (mysqld) score 57 or sacrifice child
Nov 25 12:33:26 web5 kernel: [2676096.344431] Killed process 15004, UID 498, (mysqld) total-vm:2416928kB, anon-rss:230776kB, file-rss:40kB
SO...what I need help with is trying to track down WHY the server ran out of memory.
Below is the syslog for that time period. Please let me know your thoughts, or where else I should be looking to figure this out.
* Note: I am a mid-level techie on LAMP, not a novice but by no means an expert, so please give me verbose answers and/or inquiries. Thanks!
In looking at the mysql error log I saw that there was no process running, and mysqld_safe tried to restart but failed on InnoDB due to no memory buffer available.
151125 12:34:45 mysqld_safe Number of processes running now: 0
151125 12:34:45 mysqld_safe mysqld restarted
151125 12:34:46 [Note] /usr/sbin/mysqld (mysqld 5.5.46-cll) starting as process 8159 ...
151125 12:34:46 [Note] Plugin 'FEDERATED' is disabled.
151125 12:34:46 InnoDB: The InnoDB memory heap is disabled
151125 12:34:46 InnoDB: Mutexes and rw_locks use GCC atomic builtins
151125 12:34:46 InnoDB: Compressed tables use zlib 1.2.3
151125 12:34:46 InnoDB: Using Linux native AIO
151125 12:34:46 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
151125 12:34:46 InnoDB: Completed initialization of buffer pool
151125 12:34:46 InnoDB: Fatal error: cannot allocate memory for the buffer pool
151125 12:34:46 Plugin 'InnoDB' init function returned error.
151125 12:34:46 Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
I check the syslog and found that the system killed the process due to lack of memory, as follows:
/var/log/messages
Nov 25 12:33:26 web5 kernel: [2676096.344431] Out of memory: Kill process 15004 (mysqld) score 57 or sacrifice child
Nov 25 12:33:26 web5 kernel: [2676096.344431] Killed process 15004, UID 498, (mysqld) total-vm:2416928kB, anon-rss:230776kB, file-rss:40kB
SO...what I need help with is trying to track down WHY the server ran out of memory.
Below is the syslog for that time period. Please let me know your thoughts, or where else I should be looking to figure this out.
* Note: I am a mid-level techie on LAMP, not a novice but by no means an expert, so please give me verbose answers and/or inquiries. Thanks!
Nov 25 12:33:26 web5 kernel: [2676096.343927] tailwatchd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Nov 25 12:33:26 web5 kernel: [2676096.343943] tailwatchd cpuset=/ mems_allowed=0
Nov 25 12:33:26 web5 kernel: [2676096.343950] Pid: 1293, comm: tailwatchd Not tainted 2.6.32-504.23.4.el6.x86_64 #1
Nov 25 12:33:26 web5 kernel: [2676096.343956] Call Trace:
Nov 25 12:33:26 web5 kernel: [2676096.343968] [<ffffffff810d4241>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Nov 25 12:33:26 web5 kernel: [2676096.343978] [<ffffffff81127500>] ? dump_header+0x90/0x1b0
Nov 25 12:33:26 web5 kernel: [2676096.343985] [<ffffffff81127982>] ? oom_kill_process+0x82/0x2a0
Nov 25 12:33:26 web5 kernel: [2676096.343991] [<ffffffff811278c1>] ? select_bad_process+0xe1/0x120
Nov 25 12:33:26 web5 kernel: [2676096.343998] [<ffffffff81127dc0>] ? out_of_memory+0x220/0x3c0
Nov 25 12:33:26 web5 kernel: [2676096.344005] [<ffffffff811346ff>] ? __alloc_pages_nodemask+0x89f/0x8d0
Nov 25 12:33:26 web5 kernel: [2676096.344014] [<ffffffff8116c9aa>] ? alloc_pages_current+0xaa/0x110
Nov 25 12:33:26 web5 kernel: [2676096.344020] [<ffffffff811248f7>] ? __page_cache_alloc+0x87/0x90
Nov 25 12:33:26 web5 kernel: [2676096.344027] [<ffffffff811242de>] ? find_get_page+0x1e/0xa0
Nov 25 12:33:26 web5 kernel: [2676096.344033] [<ffffffff81125897>] ? filemap_fault+0x1a7/0x500
Nov 25 12:33:26 web5 kernel: [2676096.344041] [<ffffffff8100636f>] ? xen_set_pte_at+0xaf/0x170
Nov 25 12:33:26 web5 kernel: [2676096.344049] [<ffffffff8114ed04>] ? __do_fault+0x54/0x530
Nov 25 12:33:26 web5 kernel: [2676096.344055] [<ffffffff8114f2d7>] ? handle_pte_fault+0xf7/0xb00
Nov 25 12:33:26 web5 kernel: [2676096.344063] [<ffffffff8100330b>] ? xen_write_msr_safe+0x6b/0x90
Nov 25 12:33:26 web5 kernel: [2676096.344071] [<ffffffff810097cc>] ? __switch_to+0x1ac/0x320
Nov 25 12:33:26 web5 kernel: [2676096.344077] [<ffffffff81007c01>] ? xen_clocksource_read+0x21/0x30
Nov 25 12:33:26 web5 kernel: [2676096.344083] [<ffffffff81004b29>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
Nov 25 12:33:26 web5 kernel: [2676096.344090] [<ffffffff8114ff79>] ? handle_mm_fault+0x299/0x3d0
Nov 25 12:33:26 web5 kernel: [2676096.344096] [<ffffffff810075dd>] ? xen_force_evtchn_callback+0xd/0x10
Nov 25 12:33:26 web5 kernel: [2676096.344103] [<ffffffff8104d096>] ? __do_page_fault+0x146/0x500
Nov 25 12:33:26 web5 kernel: [2676096.344110] [<ffffffff810a3f3d>] ? hrtimer_try_to_cancel+0x3d/0xd0
Nov 25 12:33:26 web5 kernel: [2676096.344116] [<ffffffff810a3ff2>] ? hrtimer_cancel+0x22/0x30
Nov 25 12:33:26 web5 kernel: [2676096.344124] [<ffffffff81041e98>] ? pvclock_clocksource_read+0x58/0xd0
Nov 25 12:33:26 web5 kernel: [2676096.344130] [<ffffffff81007c01>] ? xen_clocksource_read+0x21/0x30
Nov 25 12:33:26 web5 kernel: [2676096.344136] [<ffffffff81007ce9>] ? xen_clocksource_get_cycles+0x9/0x10
Nov 25 12:33:26 web5 kernel: [2676096.344145] [<ffffffff810a9c47>] ? getnstimeofday+0x57/0xe0
Nov 25 12:33:26 web5 kernel: [2676096.344154] [<ffffffff8153001e>] ? do_page_fault+0x3e/0xa0
Nov 25 12:33:26 web5 kernel: [2676096.344160] [<ffffffff8152d3d5>] ? page_fault+0x25/0x30
Nov 25 12:33:26 web5 kernel: [2676096.344166] Mem-Info:
Nov 25 12:33:26 web5 kernel: [2676096.344169] Node 0 DMA per-cpu:
Nov 25 12:33:26 web5 kernel: [2676096.344174] CPU 0: hi: 0, btch: 1 usd: 0
Nov 25 12:33:26 web5 kernel: [2676096.344179] CPU 1: hi: 0, btch: 1 usd: 0
Nov 25 12:33:26 web5 kernel: [2676096.344183] CPU 2: hi: 0, btch: 1 usd: 0
Nov 25 12:33:26 web5 kernel: [2676096.344187] CPU 3: hi: 0, btch: 1 usd: 0
Nov 25 12:33:26 web5 kernel: [2676096.344192] Node 0 DMA32 per-cpu:
Nov 25 12:33:26 web5 kernel: [2676096.344196] CPU 0: hi: 186, btch: 31 usd: 0
Nov 25 12:33:26 web5 kernel: [2676096.344201] CPU 1: hi: 186, btch: 31 usd: 0
Nov 25 12:33:26 web5 kernel: [2676096.344205] CPU 2: hi: 186, btch: 31 usd: 0
Nov 25 12:33:26 web5 kernel: [2676096.344210] CPU 3: hi: 186, btch: 31 usd: 0
Nov 25 12:33:26 web5 kernel: [2676096.344216] active_anon:776462 inactive_anon:155426 isolated_anon:0
Nov 25 12:33:26 web5 kernel: [2676096.344217] active_file:210 inactive_file:198 isolated_file:32
Nov 25 12:33:26 web5 kernel: [2676096.344218] unevictable:0 dirty:0 writeback:0 unstable:0
Nov 25 12:33:26 web5 kernel: [2676096.344219] free:3964 slab_reclaimable:6228 slab_unreclaimable:16717
Nov 25 12:33:26 web5 kernel: [2676096.344220] mapped:100 shmem:24 pagetables:20778 bounce:0
Nov 25 12:33:26 web5 kernel: [2676096.344237] Node 0 DMA free:7768kB min:12kB low:12kB high:16kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:7764kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Nov 25 12:33:26 web5 kernel: [2676096.344261] lowmem_reserve[]: 0 4024 4024 4024
Nov 25 12:33:26 web5 kernel: [2676096.344270] Node 0 DMA32 free:8088kB min:8108kB low:10132kB high:12160kB active_anon:3105848kB inactive_anon:621704kB active_file:840kB inactive_file:792kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:4120800kB mlocked:0kB dirty:0kB writeback:0kB mapped:400kB shmem:96kB slab_reclaimable:24912kB slab_unreclaimable:66868kB kernel_stack:4456kB pagetables:83112kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:480 all_unreclaimable? no
Nov 25 12:33:26 web5 kernel: [2676096.344297] lowmem_reserve[]: 0 0 0 0
Nov 25 12:33:26 web5 kernel: [2676096.344304] Node 0 DMA: 2*4kB 0*8kB 1*16kB 0*32kB 1*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7768kB
Nov 25 12:33:26 web5 kernel: [2676096.344323] Node 0 DMA32: 1421*4kB 17*8kB 1*16kB 1*32kB 1*64kB 2*128kB 4*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 8236kB
Nov 25 12:33:26 web5 kernel: [2676096.344341] 5545 total pagecache pages
Nov 25 12:33:26 web5 kernel: [2676096.344345] 5095 pages in swap cache
Nov 25 12:33:26 web5 kernel: [2676096.344349] Swap cache stats: add 22039814, delete 22034719, find 518462369/520674956
Nov 25 12:33:26 web5 kernel: [2676096.344355] Free swap = 0kB
Nov 25 12:33:26 web5 kernel: [2676096.344358] Total swap = 2096444kB
Nov 25 12:33:26 web5 kernel: [2676096.344431] 1048575 pages RAM
Nov 25 12:33:26 web5 kernel: [2676096.344431] 55058 pages reserved
Nov 25 12:33:26 web5 kernel: [2676096.344431] 118778 pages shared
Nov 25 12:33:26 web5 kernel: [2676096.344431] 980328 pages non-shared
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 344] 0 344 2676 1 3 -17 -1000 udevd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1181] 0 1181 23289 46 1 -17 -1000 auditd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1372] 0 1372 1540 2 0 0 0 portreserve
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1382] 0 1382 62368 830 3 0 0 rsyslogd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1405] 0 1405 4561 29 3 0 0 irqbalance
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1424] 0 1424 3066 265 1 0 0 xe-daemon
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1583] 81 1583 5358 41 3 0 0 dbus-daemon
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1630] 68 1630 9461 10 0 0 0 hald
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1631] 0 1631 5100 2 2 0 0 hald-runner
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1738] 0 1738 16083 17 3 -17 -1000 sshd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1983] 0 1983 44751 2 2 0 0 abrtd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1993] 0 1993 44709 21 3 0 0 abrt-dump-oops
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 2014] 0 2014 34188 17 3 0 0 pure-ftpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 2016] 0 2016 34033 12 0 0 0 pure-authd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 2043] 0 2043 4856 9 3 0 0 atd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 2222] 0 2222 26426 479 3 0 0 cpsrvd (SSL) -
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 2344] 0 2344 1016 2 3 0 0 mingetty
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 2348] 0 2348 1016 2 3 0 0 mingetty
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 2350] 0 2350 1016 2 2 0 0 mingetty
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 2353] 0 2353 1016 2 2 0 0 mingetty
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 2355] 0 2355 1016 2 2 0 0 mingetty
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 2357] 0 2357 1016 2 1 0 0 mingetty
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 8767] 0 8767 28750 26 3 0 0 crond
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 3597] 47 3597 17640 37 3 0 0 exim
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1138] 28 1138 574724 316 0 0 0 nscd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1270] 0 1270 8171 381 3 0 0 queueprocd - wa
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1293] 0 1293 20924 1447 3 0 0 tailwatchd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1308] 0 1308 8659 199 2 0 0 cPhulkd - proce
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1345] 0 1345 30562 289 3 0 0 cpdavd - accept
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1354] 0 1354 5915 131 2 0 0 cpanellogd - sl
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 9251] 0 9251 20868 924 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [16412] 0 16412 49841 799 3 0 0 /usr/local/cpan
Nov 25 12:33:26 web5 kernel: [2676096.344431] [14605] 0 14605 21021 2 3 0 0 login
Nov 25 12:33:26 web5 kernel: [2676096.344431] [14633] 0 14633 26610 2 2 0 0 bash
Nov 25 12:33:26 web5 kernel: [2676096.344431] [14472] 0 14472 2393 4 3 0 0 mysqld_safe
Nov 25 12:33:26 web5 kernel: [2676096.344431] [15004] 498 15004 604232 57704 3 0 0 mysqld
Nov 25 12:33:26 web5 kernel: [2676096.344431] [11806] 0 11806 50207 6961 3 0 0 spamd child
Nov 25 12:33:26 web5 kernel: [2676096.344431] [26807] 0 26807 40823 1827 3 0 0 lfd - sleeping
Nov 25 12:33:26 web5 kernel: [2676096.344431] [22024] 0 22024 19132 4 1 0 0 leechprotect
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 5216] 99 5216 21000 942 2 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 5778] 99 5778 20967 980 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 5840] 99 5840 21059 966 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 5998] 99 5998 21072 958 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6259] 99 6259 20936 883 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6294] 99 6294 20986 965 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6519] 518 6519 49584 10 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6537] 99 6537 20967 942 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6538] 99 6538 20959 924 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6539] 99 6539 20936 930 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6540] 99 6540 20967 884 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6549] 99 6549 20936 884 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6552] 99 6552 20968 884 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6553] 99 6553 20970 884 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6637] 518 6637 38062 10 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6644] 518 6644 38062 10 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6647] 518 6647 38062 10 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6651] 518 6651 38062 10 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6655] 518 6655 38062 10 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6664] 518 6664 38062 10 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6667] 99 6667 20902 889 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6668] 518 6668 38062 10 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6676] 518 6676 38062 10 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6678] 99 6678 20902 885 2 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6679] 518 6679 38062 10 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6683] 518 6683 38062 10 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6685] 99 6685 20936 884 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6686] 99 6686 20935 885 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6689] 518 6689 38062 10 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6691] 99 6691 20902 885 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6692] 99 6692 20902 885 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6694] 99 6694 20936 884 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6699] 518 6699 38062 10 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6701] 518 6701 38062 10 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6703] 99 6703 20938 884 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6704] 99 6704 20936 884 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6705] 99 6705 20902 885 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6706] 99 6706 20936 884 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6707] 99 6707 20940 884 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6708] 99 6708 20934 884 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6709] 99 6709 20902 885 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6710] 99 6710 20936 884 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6716] 518 6716 38062 10 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6728] 518 6728 38062 10 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6730] 518 6730 38062 10 1 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6734] 518 6734 38062 10 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6737] 518 6737 38062 10 1 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6740] 518 6740 38062 10 1 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6742] 518 6742 38062 10 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6771] 99 6771 20902 884 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6772] 518 6772 38062 10 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6785] 518 6785 38062 10 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6788] 518 6788 38062 10 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6790] 518 6790 38062 10 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6791] 518 6791 38062 10 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6796] 518 6796 38062 10 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6798] 99 6798 20936 883 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6802] 99 6802 21004 907 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6805] 99 6805 20934 883 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6806] 99 6806 20970 883 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6807] 99 6807 20970 883 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6808] 99 6808 20934 883 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6810] 99 6810 20934 883 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6813] 99 6813 21038 883 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6814] 99 6814 20934 963 2 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6816] 99 6816 20966 935 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6821] 518 6821 38062 10 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6835] 518 6835 38062 10 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6861] 518 6861 38062 10 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6929] 518 6929 38062 10 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6951] 518 6951 38062 10 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6953] 518 6953 38062 10 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6955] 99 6955 20940 883 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6970] 99 6970 20936 883 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6971] 99 6971 20902 884 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6972] 518 6972 38062 10 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6975] 518 6975 37977 10 1 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6978] 518 6978 38062 10 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6985] 518 6985 38062 10 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6989] 99 6989 20934 883 1 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6990] 99 6990 20936 883 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6991] 99 6991 20932 920 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.808193] Firewall: *TCP_IN Blocked* IN=eth1 OUT= MAC=06:94:92:7e:a2:1a:00:08:e3:ff:fd:90:08:00 SRC=222.160.113.26 DST=50.23.196.42 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=8844 DF PROTO=TCP SPT=45720 DPT=23 WINDOW=5808 RES=0x00 SYN URGP=0
Nov 25 12:33:26 web5 kernel: [2676096.809633] Firewall: *TCP_IN Blocked* IN=eth1 OUT= MAC=06:94:92:7e:a2:1a:00:08:e3:ff:fd:90:08:00 SRC=222.160.113.26 DST=50.23.196.41 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=48839 DF PROTO=TCP SPT=2395 DPT=23 WINDOW=5808 RES=0x00 SYN URGP=0
Nov 25 12:33:26 web5 kernel: [2676096.821583] Firewall: *TCP_IN Blocked* IN=eth1 OUT= MAC=06:94:92:7e:a2:1a:00:08:e3:ff:fd:90:08:00 SRC=222.160.113.26 DST=50.23.196.40 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=53374 DF PROTO=TCP SPT=40594 DPT=23 WINDOW=5808 RES=0x00 SYN URGP=0
Nov 25 12:33:26 web5 kernel: [2676096.854948] Firewall: *TCP_IN Blocked* IN=eth1 OUT= MAC=06:94:92:7e:a2:1a:00:08:e3:ff:fd:90:08:00 SRC=222.160.113.26 DST=50.23.196.43 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=22396 DF PROTO=TCP SPT=38763 DPT=23 WINDOW=5808 RES=0x00 SYN URGP=0
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6992] 99 6992 20932 929 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7003] 518 7003 38062 10 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7006] 518 7006 38062 10 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7008] 99 7008 20932 883 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7009] 99 7009 20964 939 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7010] 99 7010 20934 917 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7011] 99 7011 20936 898 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7012] 99 7012 20934 910 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7013] 99 7013 20932 907 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7047] 518 7047 38062 10 1 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7063] 518 7063 38127 7104 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7081] 518 7081 38062 10895 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7085] 518 7085 38062 10898 1 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7091] 99 7091 20936 963 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7092] 518 7092 38062 10899 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7094] 518 7094 38062 10898 1 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7105] 518 7105 38062 10896 1 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7108] 99 7108 20902 957 2 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7115] 518 7115 38062 10895 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7117] 99 7117 20934 970 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7118] 99 7118 20940 996 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7119] 518 7119 38062 10897 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7123] 518 7123 38062 10898 1 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7127] 99 7127 20940 996 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7128] 99 7128 20940 996 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7129] 99 7129 20936 970 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7130] 99 7130 20936 996 2 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7133] 518 7133 38062 10898 1 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7199] 518 7199 38062 10897 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7202] 518 7202 38062 10895 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7203] 518 7203 38062 10897 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7205] 518 7205 38062 10892 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7209] 518 7209 38062 10898 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7210] 99 7210 20940 993 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7216] 99 7216 20902 957 2 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7217] 99 7217 20934 973 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7218] 518 7218 38062 10897 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7222] 518 7222 38062 10897 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7223] 518 7223 38062 10898 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7228] 99 7228 20966 1006 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7229] 99 7229 20936 997 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7230] 99 7230 20902 958 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7231] 99 7231 20935 982 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7232] 518 7232 38062 10896 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7236] 518 7236 38062 10899 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7240] 518 7240 38062 10898 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7244] 99 7244 20934 971 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7245] 99 7245 20938 991 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7246] 99 7246 20969 1002 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7247] 99 7247 20936 983 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7248] 99 7248 20936 972 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7249] 99 7249 20930 972 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7250] 518 7250 38062 10899 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7314] 518 7314 38062 10897 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7316] 518 7316 38062 10898 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7318] 518 7318 38062 10899 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7320] 518 7320 38062 10896 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7322] 518 7322 38062 10897 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7324] 518 7324 38062 10897 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7326] 99 7326 20902 958 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7329] 518 7329 38062 10899 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7330] 99 7330 20935 983 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7331] 99 7331 20935 990 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7334] 518 7334 38062 10897 1 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7336] 99 7336 20935 983 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7337] 99 7337 20936 972 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7339] 99 7339 20968 1011 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7342] 518 7342 38062 10896 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7393] 518 7393 38062 10899 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7394] 518 7394 38062 10899 1 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7396] 99 7396 20935 996 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7397] 99 7397 20935 984 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7399] 99 7399 20936 969 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7400] 99 7400 20935 983 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7401] 99 7401 20902 958 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7402] 99 7402 20935 982 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7403] 99 7403 20902 958 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7409] 518 7409 38062 10898 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7414] 518 7414 38062 10899 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7415] 518 7415 38062 10898 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7418] 518 7418 38062 10897 1 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7419] 518 7419 38062 10897 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7420] 99 7420 20935 983 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7421] 518 7421 38062 10897 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7422] 518 7422 38062 10899 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7426] 99 7426 20935 983 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7427] 99 7427 20936 986 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7430] 518 7430 38062 10891 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7431] 518 7431 38062 10892 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7432] 99 7432 20965 1028 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7433] 518 7433 38062 10892 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7437] 518 7437 38062 10899 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7440] 99 7440 20968 1013 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7441] 99 7441 20934 970 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7442] 518 7442 38062 10898 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7445] 99 7445 20935 982 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7446] 99 7446 20938 1003 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7447] 99 7447 20933 997 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7448] 99 7448 20902 967 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7452] 518 7452 38062 10897 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7455] 99 7455 20935 982 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7459] 99 7459 20902 958 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7460] 99 7460 20935 983 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7465] 518 7465 38062 10899 1 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7467] 518 7467 38062 10897 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7468] 518 7468 38062 10891 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7471] 518 7471 38062 10895 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7472] 518 7472 38062 10898 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7477] 518 7477 38062 10898 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7481] 99 7481 20902 958 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7483] 518 7483 38062 10899 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7484] 518 7484 38062 10891 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7485] 99 7485 20936 990 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7486] 99 7486 20902 958 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7489] 518 7489 38062 10897 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7491] 99 7491 20936 977 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7493] 99 7493 20936 983 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7496] 518 7496 38062 10898 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7498] 99 7498 20935 982 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7500] 99 7500 20935 982 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7501] 99 7501 20935 982 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7502] 99 7502 20933 1002 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7503] 99 7503 20935 982 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7505] 99 7505 20934 998 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7511] 518 7511 38062 10898 1 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7514] 0 7514 24595 3990 0 0 0 tailwatchd - ch
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7515] 518 7515 38062 10899 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7525] 518 7525 38062 10897 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7531] 518 7531 37977 10808 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7533] 518 7533 38062 10898 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7535] 518 7535 38062 10899 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7542] 518 7542 38062 10896 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7543] 518 7543 37977 10807 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7545] 518 7545 38062 10892 1 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7547] 99 7547 20902 958 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7548] 518 7548 38062 10897 1 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7555] 99 7555 20936 976 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7556] 99 7556 20935 982 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7558] 99 7558 20935 983 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7559] 99 7559 20902 958 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7561] 99 7561 20934 991 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7562] 518 7562 38062 10898 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7563] 518 7563 38062 10898 1 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7564] 99 7564 20935 982 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7565] 99 7565 20934 965 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7566] 99 7566 20936 970 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7567] 99 7567 20940 994 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7568] 99 7568 20936 984 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7569] 99 7569 20936 971 1 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7570] 99 7570 20934 965 0 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7571] 99 7571 20902 959 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7578] 518 7578 38062 10898 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7585] 518 7585 38062 10897 1 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7616] 99 7616 20936 966 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7637] 518 7637 38062 10898 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7707] 0 7707 1018 19 0 0 0 sleep
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7708] 518 7708 38062 10896 1 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7709] 564 7709 40026 12796 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7712] 518 7712 38062 10895 0 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7714] 518 7714 38062 10898 2 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7716] 514 7716 35839 8631 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7718] 0 7718 26426 675 3 0 0 cpsrvd (SSL) -
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7719] 518 7719 33484 6365 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7720] 518 7720 33484 6367 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7721] 518 7721 33484 6366 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7722] 99 7722 20902 924 3 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7723] 583 7723 30452 3280 3 0 0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7727] 99 7727 20902 924 1 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7728] 99 7728 20902 924 1 0 0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] Out of memory: Kill process 15004 (mysqld) score 57 or sacrifice child
Nov 25 12:33:26 web5 kernel: [2676096.344431] Killed process 15004, UID 498, (mysqld) total-vm:2416928kB, anon-rss:230776kB, file-rss:40kB
Do more with
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
you might want to run msqltuner.
What are the machine specs re: memory?
what is the contents of your my.cnf ?
What are the machine specs re: memory?
what is the contents of your my.cnf ?
In short, David pointed to the possible issue that might lead to resource exhaustion.
Phpmyadmin includes some tiunning information.
Check the webserver log to make sure you were not being under a dos attack...
Phpmyadmin includes some tiunning information.
Check the webserver log to make sure you were not being under a dos attack...
Looks like a low RAM server.
total-vm:2416928kB, anon-rss:230776kB, file-rss:40kB
The system allocated 2.3 GB of RAM, of which only 225MB were actually used.
If this is a VM (yours or from Amazon/Rackspace/etc) you should consider giving it more RAM.
total-vm:2416928kB, anon-rss:230776kB, file-rss:40kB
The system allocated 2.3 GB of RAM, of which only 225MB were actually used.
If this is a VM (yours or from Amazon/Rackspace/etc) you should consider giving it more RAM.
It is a VM with 4 CPUs and 4GB RAM.
There is not a ton of load on the server, for example here is the current apache status:
Here is the contents of my.conf
There is not a ton of load on the server, for example here is the current apache status:
Apache server status for web5.ved.com
Server Version: Apache/2.4.12 (Unix) OpenSSL/1.0.1e-fips mod_bwlimited/1.4
Server MPM: prefork
Server Built: Jul 2 2015 17:19:14
Current Time: Thursday, 26-Nov-2015 09:13:39 CST
Restart Time: Wednesday, 18-Nov-2015 15:45:51 CST
Parent Server Config. Generation: 135
Parent Server MPM Generation: 134
Server uptime: 7 days 17 hours 27 minutes 48 seconds
Server load: 1.88 1.70 1.73
Total accesses: 807775 - Total Traffic: 44.7 GB
CPU Usage: u7.56 s12.6 cu766.88 cs263.47 - .157% CPU load
1.21 requests/sec - 70.3 kB/second - 58.1 kB/request
3 requests currently being processed, 7 idle workers
Here is the contents of my.conf
[mysqld]
default-storage-engine=MyISAM
innodb_file_per_table=1
max_allowed_packet=268435456
open_files_limit=47000
datadir="/home2/_MySQL"
Please use the tools referenced. They will crunch your mysql statistics and will point out where you might have defficiencies.
You have many php, httpd, processies as well as spamd running suggesting this is also a mail server.
With all those you may have contention for write to "disk"
How many httpd children did you configure in httpd.conf are you starting 30 of them?
how big is your mysql user DBs? all excluding mysql?
use iostat -xtc 20 5 and see if you have "disk" contention
When there are multiple application identifying which is consuming and causing the issue is the prescription.
You have many php, httpd, processies as well as spamd running suggesting this is also a mail server.
With all those you may have contention for write to "disk"
How many httpd children did you configure in httpd.conf are you starting 30 of them?
how big is your mysql user DBs? all excluding mysql?
use iostat -xtc 20 5 and see if you have "disk" contention
When there are multiple application identifying which is consuming and causing the issue is the prescription.
4 GB or ram is really cutting things close with using 50 websites with say 30 using mysql .. All one needs is for more than a few websites to be active at the same time and you will run out of memory.. plus if you don't have the disk iops available then mysql will start queuing requests (adding more memory pressure) (now we are in a vicious circle) .. I'd double the ram at minimum and start looking at RAID or SSD for the disk subsystem.
Can you please give me an overview of how to use mysql tuner? Specifically, what options should I use to output the info I need?
I've not used it the following page outlines its use: https://github.com/major/MySQLTuner-perl
Run the tool with its suggestion output to a file that you can then post.
Does your system have phpmyadmin (web based) interface to MySQL?
Run the tool with its suggestion output to a file that you can then post.
Does your system have phpmyadmin (web based) interface to MySQL?
Thank you for all that asissted. Here's the final resolution, in the hopes that this will help someone else diagnose a similar issue.
Here's what I found...
/var/log/messages
The server killed the MySQL process due to lack of memory (as mentioned by contributors), which then caused the SQL Server to try and restart but was unable to allocate memory for InnoDB.
/home/MYSQL/web.domain.com
Here's what I found...
/var/log/messages
Nov 25 12:33:26 web5 kernel: [2676096.344431] Out of memory: Kill process 15004 (mysqld) score 57 or sacrifice child
Nov 25 12:33:26 web5 kernel: [2676096.344431] Killed process 15004, UID 498, (mysqld) total-vm:2416928kB, anon-rss:230776kB, file-rss:40kB
The server killed the MySQL process due to lack of memory (as mentioned by contributors), which then caused the SQL Server to try and restart but was unable to allocate memory for InnoDB.
/home/MYSQL/web.domain.com
151125 12:34:45 mysqld_safe Number of processes running now: 0
151125 12:34:45 mysqld_safe mysqld restarted
151125 12:34:46 [Note] /usr/sbin/mysqld (mysqld 5.5.46-cll) starting as process 8159 ...
151125 12:34:46 [Note] Plugin 'FEDERATED' is disabled.
151125 12:34:46 InnoDB: The InnoDB memory heap is disabled
151125 12:34:46 InnoDB: Mutexes and rw_locks use GCC atomic builtins
151125 12:34:46 InnoDB: Compressed tables use zlib 1.2.3
151125 12:34:46 InnoDB: Using Linux native AIO
151125 12:34:46 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
151125 12:34:46 InnoDB: Completed initialization of buffer pool
151125 12:34:46 InnoDB: Fatal error: cannot allocate memory for the buffer pool
151125 12:34:46 [ERROR] Plugin 'InnoDB' init function returned error.
151125 12:34:46 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
Premium Content
You need an Expert Office subscription to comment.Start Free Trial