Mysqld Crash Autoposy Help Needed

worthyking1
worthyking1 used Ask the Experts™
on
We are running a LAMP server with CentOS 6 which serves about 50 (low to medium traffic) websites, mostly wordpress. Today we found the majority of WP sites were not working and upon investigation found InnoDB errors for those databases. Restarting mysql fixed the problem.  Now I want to figure out exactly why.

In looking at the mysql error log I saw that there was no process running, and mysqld_safe tried to restart but failed on InnoDB due to no memory buffer available.

151125 12:34:45 mysqld_safe Number of processes running now: 0
151125 12:34:45 mysqld_safe mysqld restarted
151125 12:34:46 [Note] /usr/sbin/mysqld (mysqld 5.5.46-cll) starting as process 8159 ...
151125 12:34:46 [Note] Plugin 'FEDERATED' is disabled.
151125 12:34:46 InnoDB: The InnoDB memory heap is disabled
151125 12:34:46 InnoDB: Mutexes and rw_locks use GCC atomic builtins
151125 12:34:46 InnoDB: Compressed tables use zlib 1.2.3
151125 12:34:46 InnoDB: Using Linux native AIO
151125 12:34:46 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
151125 12:34:46 InnoDB: Completed initialization of buffer pool
151125 12:34:46 InnoDB: Fatal error: cannot allocate memory for the buffer pool
151125 12:34:46 Plugin 'InnoDB' init function returned error.
151125 12:34:46 Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

I check the syslog and found that the system killed the process due to lack of memory, as follows:

/var/log/messages

Nov 25 12:33:26 web5 kernel: [2676096.344431] Out of memory: Kill process 15004 (mysqld) score 57 or sacrifice child
Nov 25 12:33:26 web5 kernel: [2676096.344431] Killed process 15004, UID 498, (mysqld) total-vm:2416928kB, anon-rss:230776kB, file-rss:40kB

SO...what I need help with is trying to track down WHY the server ran out of memory.

Below is the syslog for that time period. Please let me know your thoughts, or where else I should be looking to figure this out.

* Note: I am a mid-level techie on LAMP, not a novice but by no means an expert, so please give me verbose answers and/or inquiries. Thanks!

Nov 25 12:33:26 web5 kernel: [2676096.343927] tailwatchd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Nov 25 12:33:26 web5 kernel: [2676096.343943] tailwatchd cpuset=/ mems_allowed=0
Nov 25 12:33:26 web5 kernel: [2676096.343950] Pid: 1293, comm: tailwatchd Not tainted 2.6.32-504.23.4.el6.x86_64 #1
Nov 25 12:33:26 web5 kernel: [2676096.343956] Call Trace:
Nov 25 12:33:26 web5 kernel: [2676096.343968]  [<ffffffff810d4241>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Nov 25 12:33:26 web5 kernel: [2676096.343978]  [<ffffffff81127500>] ? dump_header+0x90/0x1b0
Nov 25 12:33:26 web5 kernel: [2676096.343985]  [<ffffffff81127982>] ? oom_kill_process+0x82/0x2a0
Nov 25 12:33:26 web5 kernel: [2676096.343991]  [<ffffffff811278c1>] ? select_bad_process+0xe1/0x120
Nov 25 12:33:26 web5 kernel: [2676096.343998]  [<ffffffff81127dc0>] ? out_of_memory+0x220/0x3c0
Nov 25 12:33:26 web5 kernel: [2676096.344005]  [<ffffffff811346ff>] ? __alloc_pages_nodemask+0x89f/0x8d0
Nov 25 12:33:26 web5 kernel: [2676096.344014]  [<ffffffff8116c9aa>] ? alloc_pages_current+0xaa/0x110
Nov 25 12:33:26 web5 kernel: [2676096.344020]  [<ffffffff811248f7>] ? __page_cache_alloc+0x87/0x90
Nov 25 12:33:26 web5 kernel: [2676096.344027]  [<ffffffff811242de>] ? find_get_page+0x1e/0xa0
Nov 25 12:33:26 web5 kernel: [2676096.344033]  [<ffffffff81125897>] ? filemap_fault+0x1a7/0x500
Nov 25 12:33:26 web5 kernel: [2676096.344041]  [<ffffffff8100636f>] ? xen_set_pte_at+0xaf/0x170
Nov 25 12:33:26 web5 kernel: [2676096.344049]  [<ffffffff8114ed04>] ? __do_fault+0x54/0x530
Nov 25 12:33:26 web5 kernel: [2676096.344055]  [<ffffffff8114f2d7>] ? handle_pte_fault+0xf7/0xb00
Nov 25 12:33:26 web5 kernel: [2676096.344063]  [<ffffffff8100330b>] ? xen_write_msr_safe+0x6b/0x90
Nov 25 12:33:26 web5 kernel: [2676096.344071]  [<ffffffff810097cc>] ? __switch_to+0x1ac/0x320
Nov 25 12:33:26 web5 kernel: [2676096.344077]  [<ffffffff81007c01>] ? xen_clocksource_read+0x21/0x30
Nov 25 12:33:26 web5 kernel: [2676096.344083]  [<ffffffff81004b29>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
Nov 25 12:33:26 web5 kernel: [2676096.344090]  [<ffffffff8114ff79>] ? handle_mm_fault+0x299/0x3d0
Nov 25 12:33:26 web5 kernel: [2676096.344096]  [<ffffffff810075dd>] ? xen_force_evtchn_callback+0xd/0x10
Nov 25 12:33:26 web5 kernel: [2676096.344103]  [<ffffffff8104d096>] ? __do_page_fault+0x146/0x500
Nov 25 12:33:26 web5 kernel: [2676096.344110]  [<ffffffff810a3f3d>] ? hrtimer_try_to_cancel+0x3d/0xd0
Nov 25 12:33:26 web5 kernel: [2676096.344116]  [<ffffffff810a3ff2>] ? hrtimer_cancel+0x22/0x30
Nov 25 12:33:26 web5 kernel: [2676096.344124]  [<ffffffff81041e98>] ? pvclock_clocksource_read+0x58/0xd0
Nov 25 12:33:26 web5 kernel: [2676096.344130]  [<ffffffff81007c01>] ? xen_clocksource_read+0x21/0x30
Nov 25 12:33:26 web5 kernel: [2676096.344136]  [<ffffffff81007ce9>] ? xen_clocksource_get_cycles+0x9/0x10
Nov 25 12:33:26 web5 kernel: [2676096.344145]  [<ffffffff810a9c47>] ? getnstimeofday+0x57/0xe0
Nov 25 12:33:26 web5 kernel: [2676096.344154]  [<ffffffff8153001e>] ? do_page_fault+0x3e/0xa0
Nov 25 12:33:26 web5 kernel: [2676096.344160]  [<ffffffff8152d3d5>] ? page_fault+0x25/0x30
Nov 25 12:33:26 web5 kernel: [2676096.344166] Mem-Info:
Nov 25 12:33:26 web5 kernel: [2676096.344169] Node 0 DMA per-cpu:
Nov 25 12:33:26 web5 kernel: [2676096.344174] CPU    0: hi:    0, btch:   1 usd:   0
Nov 25 12:33:26 web5 kernel: [2676096.344179] CPU    1: hi:    0, btch:   1 usd:   0
Nov 25 12:33:26 web5 kernel: [2676096.344183] CPU    2: hi:    0, btch:   1 usd:   0
Nov 25 12:33:26 web5 kernel: [2676096.344187] CPU    3: hi:    0, btch:   1 usd:   0
Nov 25 12:33:26 web5 kernel: [2676096.344192] Node 0 DMA32 per-cpu:
Nov 25 12:33:26 web5 kernel: [2676096.344196] CPU    0: hi:  186, btch:  31 usd:   0
Nov 25 12:33:26 web5 kernel: [2676096.344201] CPU    1: hi:  186, btch:  31 usd:   0
Nov 25 12:33:26 web5 kernel: [2676096.344205] CPU    2: hi:  186, btch:  31 usd:   0
Nov 25 12:33:26 web5 kernel: [2676096.344210] CPU    3: hi:  186, btch:  31 usd:   0
Nov 25 12:33:26 web5 kernel: [2676096.344216] active_anon:776462 inactive_anon:155426 isolated_anon:0
Nov 25 12:33:26 web5 kernel: [2676096.344217]  active_file:210 inactive_file:198 isolated_file:32
Nov 25 12:33:26 web5 kernel: [2676096.344218]  unevictable:0 dirty:0 writeback:0 unstable:0
Nov 25 12:33:26 web5 kernel: [2676096.344219]  free:3964 slab_reclaimable:6228 slab_unreclaimable:16717
Nov 25 12:33:26 web5 kernel: [2676096.344220]  mapped:100 shmem:24 pagetables:20778 bounce:0
Nov 25 12:33:26 web5 kernel: [2676096.344237] Node 0 DMA free:7768kB min:12kB low:12kB high:16kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:7764kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Nov 25 12:33:26 web5 kernel: [2676096.344261] lowmem_reserve[]: 0 4024 4024 4024
Nov 25 12:33:26 web5 kernel: [2676096.344270] Node 0 DMA32 free:8088kB min:8108kB low:10132kB high:12160kB active_anon:3105848kB inactive_anon:621704kB active_file:840kB inactive_file:792kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:4120800kB mlocked:0kB dirty:0kB writeback:0kB mapped:400kB shmem:96kB slab_reclaimable:24912kB slab_unreclaimable:66868kB kernel_stack:4456kB pagetables:83112kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:480 all_unreclaimable? no
Nov 25 12:33:26 web5 kernel: [2676096.344297] lowmem_reserve[]: 0 0 0 0
Nov 25 12:33:26 web5 kernel: [2676096.344304] Node 0 DMA: 2*4kB 0*8kB 1*16kB 0*32kB 1*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7768kB
Nov 25 12:33:26 web5 kernel: [2676096.344323] Node 0 DMA32: 1421*4kB 17*8kB 1*16kB 1*32kB 1*64kB 2*128kB 4*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 8236kB
Nov 25 12:33:26 web5 kernel: [2676096.344341] 5545 total pagecache pages
Nov 25 12:33:26 web5 kernel: [2676096.344345] 5095 pages in swap cache
Nov 25 12:33:26 web5 kernel: [2676096.344349] Swap cache stats: add 22039814, delete 22034719, find 518462369/520674956
Nov 25 12:33:26 web5 kernel: [2676096.344355] Free swap  = 0kB
Nov 25 12:33:26 web5 kernel: [2676096.344358] Total swap = 2096444kB
Nov 25 12:33:26 web5 kernel: [2676096.344431] 1048575 pages RAM
Nov 25 12:33:26 web5 kernel: [2676096.344431] 55058 pages reserved
Nov 25 12:33:26 web5 kernel: [2676096.344431] 118778 pages shared
Nov 25 12:33:26 web5 kernel: [2676096.344431] 980328 pages non-shared
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Nov 25 12:33:26 web5 kernel: [2676096.344431] [  344]     0   344     2676        1   3     -17         -1000 udevd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1181]     0  1181    23289       46   1     -17         -1000 auditd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1372]     0  1372     1540        2   0       0             0 portreserve
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1382]     0  1382    62368      830   3       0             0 rsyslogd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1405]     0  1405     4561       29   3       0             0 irqbalance
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1424]     0  1424     3066      265   1       0             0 xe-daemon
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1583]    81  1583     5358       41   3       0             0 dbus-daemon
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1630]    68  1630     9461       10   0       0             0 hald
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1631]     0  1631     5100        2   2       0             0 hald-runner
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1738]     0  1738    16083       17   3     -17         -1000 sshd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1983]     0  1983    44751        2   2       0             0 abrtd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1993]     0  1993    44709       21   3       0             0 abrt-dump-oops
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 2014]     0  2014    34188       17   3       0             0 pure-ftpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 2016]     0  2016    34033       12   0       0             0 pure-authd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 2043]     0  2043     4856        9   3       0             0 atd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 2222]     0  2222    26426      479   3       0             0 cpsrvd (SSL) - 
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 2344]     0  2344     1016        2   3       0             0 mingetty
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 2348]     0  2348     1016        2   3       0             0 mingetty
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 2350]     0  2350     1016        2   2       0             0 mingetty
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 2353]     0  2353     1016        2   2       0             0 mingetty
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 2355]     0  2355     1016        2   2       0             0 mingetty
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 2357]     0  2357     1016        2   1       0             0 mingetty
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 8767]     0  8767    28750       26   3       0             0 crond
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 3597]    47  3597    17640       37   3       0             0 exim
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1138]    28  1138   574724      316   0       0             0 nscd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1270]     0  1270     8171      381   3       0             0 queueprocd - wa
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1293]     0  1293    20924     1447   3       0             0 tailwatchd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1308]     0  1308     8659      199   2       0             0 cPhulkd - proce
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1345]     0  1345    30562      289   3       0             0 cpdavd - accept
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 1354]     0  1354     5915      131   2       0             0 cpanellogd - sl
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 9251]     0  9251    20868      924   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [16412]     0 16412    49841      799   3       0             0 /usr/local/cpan
Nov 25 12:33:26 web5 kernel: [2676096.344431] [14605]     0 14605    21021        2   3       0             0 login
Nov 25 12:33:26 web5 kernel: [2676096.344431] [14633]     0 14633    26610        2   2       0             0 bash
Nov 25 12:33:26 web5 kernel: [2676096.344431] [14472]     0 14472     2393        4   3       0             0 mysqld_safe
Nov 25 12:33:26 web5 kernel: [2676096.344431] [15004]   498 15004   604232    57704   3       0             0 mysqld
Nov 25 12:33:26 web5 kernel: [2676096.344431] [11806]     0 11806    50207     6961   3       0             0 spamd child
Nov 25 12:33:26 web5 kernel: [2676096.344431] [26807]     0 26807    40823     1827   3       0             0 lfd - sleeping
Nov 25 12:33:26 web5 kernel: [2676096.344431] [22024]     0 22024    19132        4   1       0             0 leechprotect
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 5216]    99  5216    21000      942   2       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 5778]    99  5778    20967      980   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 5840]    99  5840    21059      966   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 5998]    99  5998    21072      958   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6259]    99  6259    20936      883   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6294]    99  6294    20986      965   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6519]   518  6519    49584       10   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6537]    99  6537    20967      942   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6538]    99  6538    20959      924   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6539]    99  6539    20936      930   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6540]    99  6540    20967      884   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6549]    99  6549    20936      884   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6552]    99  6552    20968      884   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6553]    99  6553    20970      884   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6637]   518  6637    38062       10   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6644]   518  6644    38062       10   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6647]   518  6647    38062       10   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6651]   518  6651    38062       10   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6655]   518  6655    38062       10   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6664]   518  6664    38062       10   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6667]    99  6667    20902      889   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6668]   518  6668    38062       10   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6676]   518  6676    38062       10   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6678]    99  6678    20902      885   2       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6679]   518  6679    38062       10   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6683]   518  6683    38062       10   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6685]    99  6685    20936      884   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6686]    99  6686    20935      885   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6689]   518  6689    38062       10   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6691]    99  6691    20902      885   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6692]    99  6692    20902      885   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6694]    99  6694    20936      884   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6699]   518  6699    38062       10   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6701]   518  6701    38062       10   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6703]    99  6703    20938      884   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6704]    99  6704    20936      884   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6705]    99  6705    20902      885   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6706]    99  6706    20936      884   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6707]    99  6707    20940      884   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6708]    99  6708    20934      884   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6709]    99  6709    20902      885   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6710]    99  6710    20936      884   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6716]   518  6716    38062       10   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6728]   518  6728    38062       10   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6730]   518  6730    38062       10   1       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6734]   518  6734    38062       10   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6737]   518  6737    38062       10   1       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6740]   518  6740    38062       10   1       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6742]   518  6742    38062       10   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6771]    99  6771    20902      884   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6772]   518  6772    38062       10   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6785]   518  6785    38062       10   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6788]   518  6788    38062       10   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6790]   518  6790    38062       10   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6791]   518  6791    38062       10   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6796]   518  6796    38062       10   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6798]    99  6798    20936      883   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6802]    99  6802    21004      907   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6805]    99  6805    20934      883   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6806]    99  6806    20970      883   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6807]    99  6807    20970      883   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6808]    99  6808    20934      883   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6810]    99  6810    20934      883   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6813]    99  6813    21038      883   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6814]    99  6814    20934      963   2       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6816]    99  6816    20966      935   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6821]   518  6821    38062       10   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6835]   518  6835    38062       10   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6861]   518  6861    38062       10   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6929]   518  6929    38062       10   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6951]   518  6951    38062       10   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6953]   518  6953    38062       10   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6955]    99  6955    20940      883   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6970]    99  6970    20936      883   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6971]    99  6971    20902      884   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6972]   518  6972    38062       10   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6975]   518  6975    37977       10   1       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6978]   518  6978    38062       10   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6985]   518  6985    38062       10   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6989]    99  6989    20934      883   1       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6990]    99  6990    20936      883   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6991]    99  6991    20932      920   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.808193] Firewall: *TCP_IN Blocked* IN=eth1 OUT= MAC=06:94:92:7e:a2:1a:00:08:e3:ff:fd:90:08:00 SRC=222.160.113.26 DST=50.23.196.42 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=8844 DF PROTO=TCP SPT=45720 DPT=23 WINDOW=5808 RES=0x00 SYN URGP=0 
Nov 25 12:33:26 web5 kernel: [2676096.809633] Firewall: *TCP_IN Blocked* IN=eth1 OUT= MAC=06:94:92:7e:a2:1a:00:08:e3:ff:fd:90:08:00 SRC=222.160.113.26 DST=50.23.196.41 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=48839 DF PROTO=TCP SPT=2395 DPT=23 WINDOW=5808 RES=0x00 SYN URGP=0 
Nov 25 12:33:26 web5 kernel: [2676096.821583] Firewall: *TCP_IN Blocked* IN=eth1 OUT= MAC=06:94:92:7e:a2:1a:00:08:e3:ff:fd:90:08:00 SRC=222.160.113.26 DST=50.23.196.40 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=53374 DF PROTO=TCP SPT=40594 DPT=23 WINDOW=5808 RES=0x00 SYN URGP=0 
Nov 25 12:33:26 web5 kernel: [2676096.854948] Firewall: *TCP_IN Blocked* IN=eth1 OUT= MAC=06:94:92:7e:a2:1a:00:08:e3:ff:fd:90:08:00 SRC=222.160.113.26 DST=50.23.196.43 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=22396 DF PROTO=TCP SPT=38763 DPT=23 WINDOW=5808 RES=0x00 SYN URGP=0 
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 6992]    99  6992    20932      929   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7003]   518  7003    38062       10   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7006]   518  7006    38062       10   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7008]    99  7008    20932      883   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7009]    99  7009    20964      939   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7010]    99  7010    20934      917   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7011]    99  7011    20936      898   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7012]    99  7012    20934      910   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7013]    99  7013    20932      907   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7047]   518  7047    38062       10   1       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7063]   518  7063    38127     7104   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7081]   518  7081    38062    10895   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7085]   518  7085    38062    10898   1       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7091]    99  7091    20936      963   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7092]   518  7092    38062    10899   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7094]   518  7094    38062    10898   1       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7105]   518  7105    38062    10896   1       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7108]    99  7108    20902      957   2       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7115]   518  7115    38062    10895   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7117]    99  7117    20934      970   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7118]    99  7118    20940      996   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7119]   518  7119    38062    10897   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7123]   518  7123    38062    10898   1       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7127]    99  7127    20940      996   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7128]    99  7128    20940      996   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7129]    99  7129    20936      970   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7130]    99  7130    20936      996   2       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7133]   518  7133    38062    10898   1       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7199]   518  7199    38062    10897   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7202]   518  7202    38062    10895   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7203]   518  7203    38062    10897   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7205]   518  7205    38062    10892   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7209]   518  7209    38062    10898   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7210]    99  7210    20940      993   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7216]    99  7216    20902      957   2       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7217]    99  7217    20934      973   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7218]   518  7218    38062    10897   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7222]   518  7222    38062    10897   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7223]   518  7223    38062    10898   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7228]    99  7228    20966     1006   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7229]    99  7229    20936      997   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7230]    99  7230    20902      958   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7231]    99  7231    20935      982   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7232]   518  7232    38062    10896   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7236]   518  7236    38062    10899   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7240]   518  7240    38062    10898   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7244]    99  7244    20934      971   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7245]    99  7245    20938      991   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7246]    99  7246    20969     1002   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7247]    99  7247    20936      983   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7248]    99  7248    20936      972   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7249]    99  7249    20930      972   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7250]   518  7250    38062    10899   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7314]   518  7314    38062    10897   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7316]   518  7316    38062    10898   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7318]   518  7318    38062    10899   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7320]   518  7320    38062    10896   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7322]   518  7322    38062    10897   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7324]   518  7324    38062    10897   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7326]    99  7326    20902      958   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7329]   518  7329    38062    10899   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7330]    99  7330    20935      983   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7331]    99  7331    20935      990   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7334]   518  7334    38062    10897   1       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7336]    99  7336    20935      983   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7337]    99  7337    20936      972   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7339]    99  7339    20968     1011   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7342]   518  7342    38062    10896   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7393]   518  7393    38062    10899   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7394]   518  7394    38062    10899   1       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7396]    99  7396    20935      996   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7397]    99  7397    20935      984   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7399]    99  7399    20936      969   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7400]    99  7400    20935      983   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7401]    99  7401    20902      958   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7402]    99  7402    20935      982   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7403]    99  7403    20902      958   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7409]   518  7409    38062    10898   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7414]   518  7414    38062    10899   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7415]   518  7415    38062    10898   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7418]   518  7418    38062    10897   1       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7419]   518  7419    38062    10897   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7420]    99  7420    20935      983   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7421]   518  7421    38062    10897   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7422]   518  7422    38062    10899   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7426]    99  7426    20935      983   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7427]    99  7427    20936      986   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7430]   518  7430    38062    10891   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7431]   518  7431    38062    10892   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7432]    99  7432    20965     1028   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7433]   518  7433    38062    10892   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7437]   518  7437    38062    10899   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7440]    99  7440    20968     1013   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7441]    99  7441    20934      970   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7442]   518  7442    38062    10898   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7445]    99  7445    20935      982   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7446]    99  7446    20938     1003   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7447]    99  7447    20933      997   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7448]    99  7448    20902      967   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7452]   518  7452    38062    10897   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7455]    99  7455    20935      982   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7459]    99  7459    20902      958   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7460]    99  7460    20935      983   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7465]   518  7465    38062    10899   1       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7467]   518  7467    38062    10897   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7468]   518  7468    38062    10891   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7471]   518  7471    38062    10895   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7472]   518  7472    38062    10898   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7477]   518  7477    38062    10898   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7481]    99  7481    20902      958   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7483]   518  7483    38062    10899   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7484]   518  7484    38062    10891   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7485]    99  7485    20936      990   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7486]    99  7486    20902      958   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7489]   518  7489    38062    10897   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7491]    99  7491    20936      977   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7493]    99  7493    20936      983   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7496]   518  7496    38062    10898   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7498]    99  7498    20935      982   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7500]    99  7500    20935      982   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7501]    99  7501    20935      982   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7502]    99  7502    20933     1002   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7503]    99  7503    20935      982   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7505]    99  7505    20934      998   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7511]   518  7511    38062    10898   1       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7514]     0  7514    24595     3990   0       0             0 tailwatchd - ch
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7515]   518  7515    38062    10899   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7525]   518  7525    38062    10897   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7531]   518  7531    37977    10808   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7533]   518  7533    38062    10898   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7535]   518  7535    38062    10899   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7542]   518  7542    38062    10896   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7543]   518  7543    37977    10807   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7545]   518  7545    38062    10892   1       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7547]    99  7547    20902      958   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7548]   518  7548    38062    10897   1       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7555]    99  7555    20936      976   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7556]    99  7556    20935      982   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7558]    99  7558    20935      983   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7559]    99  7559    20902      958   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7561]    99  7561    20934      991   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7562]   518  7562    38062    10898   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7563]   518  7563    38062    10898   1       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7564]    99  7564    20935      982   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7565]    99  7565    20934      965   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7566]    99  7566    20936      970   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7567]    99  7567    20940      994   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7568]    99  7568    20936      984   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7569]    99  7569    20936      971   1       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7570]    99  7570    20934      965   0       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7571]    99  7571    20902      959   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7578]   518  7578    38062    10898   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7585]   518  7585    38062    10897   1       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7616]    99  7616    20936      966   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7637]   518  7637    38062    10898   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7707]     0  7707     1018       19   0       0             0 sleep
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7708]   518  7708    38062    10896   1       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7709]   564  7709    40026    12796   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7712]   518  7712    38062    10895   0       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7714]   518  7714    38062    10898   2       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7716]   514  7716    35839     8631   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7718]     0  7718    26426      675   3       0             0 cpsrvd (SSL) - 
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7719]   518  7719    33484     6365   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7720]   518  7720    33484     6367   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7721]   518  7721    33484     6366   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7722]    99  7722    20902      924   3       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7723]   583  7723    30452     3280   3       0             0 php
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7727]    99  7727    20902      924   1       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] [ 7728]    99  7728    20902      924   1       0             0 httpd
Nov 25 12:33:26 web5 kernel: [2676096.344431] Out of memory: Kill process 15004 (mysqld) score 57 or sacrifice child
Nov 25 12:33:26 web5 kernel: [2676096.344431] Killed process 15004, UID 498, (mysqld) total-vm:2416928kB, anon-rss:230776kB, file-rss:40kB

Open in new window

Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Top Expert 2016

Commented:
you might want to run msqltuner.
What are the machine specs re: memory?
what is the contents of your my.cnf ?
Distinguished Expert 2017

Commented:
In short, David pointed to the possible issue that might lead to resource exhaustion.

Phpmyadmin includes some tiunning information.

Check the webserver log to make sure you were not being under a dos attack...
Commented:
Looks like a low RAM server.

total-vm:2416928kB, anon-rss:230776kB, file-rss:40kB
The system allocated 2.3 GB of RAM, of which only 225MB were actually used.

If this is a VM (yours or from Amazon/Rackspace/etc) you should consider giving it more RAM.
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

It is a VM with 4 CPUs and 4GB RAM.

There is not a ton of load on the server, for example here is the current apache status:

Apache server status for web5.ved.com

Server Version: Apache/2.4.12 (Unix) OpenSSL/1.0.1e-fips mod_bwlimited/1.4
Server MPM: prefork
Server Built: Jul 2 2015 17:19:14
Current Time: Thursday, 26-Nov-2015 09:13:39 CST
Restart Time: Wednesday, 18-Nov-2015 15:45:51 CST
Parent Server Config. Generation: 135
Parent Server MPM Generation: 134
Server uptime: 7 days 17 hours 27 minutes 48 seconds
Server load: 1.88 1.70 1.73
Total accesses: 807775 - Total Traffic: 44.7 GB
CPU Usage: u7.56 s12.6 cu766.88 cs263.47 - .157% CPU load
1.21 requests/sec - 70.3 kB/second - 58.1 kB/request
3 requests currently being processed, 7 idle workers

Open in new window


Here is the contents of my.conf

[mysqld]
default-storage-engine=MyISAM
innodb_file_per_table=1
max_allowed_packet=268435456
open_files_limit=47000
datadir="/home2/_MySQL"

Open in new window

Distinguished Expert 2017

Commented:
Please use the tools referenced.  They will crunch your mysql statistics and will point out where you might have defficiencies.

You have many php, httpd, processies as well as spamd running suggesting this is also a mail server.
With all those you may have contention for write to "disk"

How many httpd children did you configure in httpd.conf are you starting 30 of them?
how big is your mysql user DBs? all excluding mysql?

use iostat -xtc 20 5 and see if you have "disk" contention

When there are multiple application identifying which is consuming and causing the issue is the prescription.
Top Expert 2016
Commented:
4 GB or ram is really cutting things close with using 50 websites with say 30 using mysql .. All one needs is for more than a few websites to be active at the same time and you will run out of memory.. plus if you don't have the disk iops available then mysql will start queuing requests (adding more memory pressure) (now we are in a vicious circle) .. I'd double the ram at minimum and start looking at RAID or SSD for the disk subsystem.
Can you please give me an overview of how to use mysql tuner? Specifically, what options should I use to output the info I need?
Distinguished Expert 2017

Commented:
I've not used it the following page outlines its use: https://github.com/major/MySQLTuner-perl

Run the tool with its suggestion output to a file that you can then post.

Does your system have phpmyadmin (web based) interface to MySQL?
Thank you for all that asissted. Here's the final resolution, in the hopes that this will help someone else diagnose a similar issue.

Here's what I found...

/var/log/messages

Nov 25 12:33:26 web5 kernel: [2676096.344431] Out of memory: Kill process 15004 (mysqld) score 57 or sacrifice child
Nov 25 12:33:26 web5 kernel: [2676096.344431] Killed process 15004, UID 498, (mysqld) total-vm:2416928kB, anon-rss:230776kB, file-rss:40kB

Open in new window


The server killed the MySQL process due to lack of memory (as mentioned by contributors), which then caused the SQL Server to try and restart but was unable to allocate memory for InnoDB.

/home/MYSQL/web.domain.com.err

151125 12:34:45 mysqld_safe Number of processes running now: 0
151125 12:34:45 mysqld_safe mysqld restarted
151125 12:34:46 [Note] /usr/sbin/mysqld (mysqld 5.5.46-cll) starting as process 8159 ...
151125 12:34:46 [Note] Plugin 'FEDERATED' is disabled.
151125 12:34:46 InnoDB: The InnoDB memory heap is disabled
151125 12:34:46 InnoDB: Mutexes and rw_locks use GCC atomic builtins
151125 12:34:46 InnoDB: Compressed tables use zlib 1.2.3
151125 12:34:46 InnoDB: Using Linux native AIO
151125 12:34:46 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
151125 12:34:46 InnoDB: Completed initialization of buffer pool
151125 12:34:46 InnoDB: Fatal error: cannot allocate memory for the buffer pool
151125 12:34:46 [ERROR] Plugin 'InnoDB' init function returned error.
151125 12:34:46 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

Open in new window

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial