Why Does Mysql Keep Increasing Memory Usage?

My site went down due to Maria DB increasing in memory usage.  I am running atop and went through the logs and saw it increase in size enough to take the server down.

I ran mysqltuner on the DB and got these results

-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    Control warning line(s) into /var/log/mariadb/mariadb.log file
    Control error line(s) into /var/log/mariadb/mariadb.log file
    Run OPTIMIZE TABLE to defragment tables for better performance
      OPTIMIZE TABLE `thefruga_smf_test`.`smf_log_errors`; -- can free 47.8366432189941 MB
    Total freed space after theses OPTIMIZE TABLE : 47.8366432189941 Mb
    Restrict Host for user@% to user@SpecificDNSorIp
    MySQL was started within the last 24 hours - recommendations may be inaccurate
    Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
    Adjust your join queries to always utilize indexes
    When making adjustments, make tmp_table_size/max_heap_table_size equal
    Reduce your SELECT DISTINCT queries which have no LIMIT clause
    Performance schema should be activated for better diagnostics
    Consider installing Sys schema from https://github.com/mysql/mysql-sys
    Before changing innodb_log_file_size and/or innodb_log_files_in_group read this: http://bit.ly/2wgkDvS
Variables to adjust:
    query_cache_size (=0)
    query_cache_type (=0)
    query_cache_limit (> 1M, or use smaller result sets)
    join_buffer_size (> 256.0K, or always use indexes with JOINs)
    tmp_table_size (> 16M)
    max_heap_table_size (> 16M)
    performance_schema = ON enable PFS
    innodb_buffer_pool_size (>= 5.6G) if possible.
    innodb_log_file_size should be (=16M) if possible, so InnoDB total log files size equals to 25% of buffer pool size.

Open in new window


I am on an Amazon VPS with 3 GB of memory.

Are any of these setting so important that it would cause the memory amount to cause the server crash?  If so, please give me the steps to make the appropriate changes.

These are the version numbers - https://gyazo.com/fb1106d8fb4efa5766484c1d9981699a

Also, I keep getting this error in the logs - 2018-08-25 15:07:04 140097338496768 [Warning] Access denied for user 'root'@'localhost' (using password: YES)

Thanks,
sharingsunshineAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

David FavorLinux/LXD/WordPress/Hosting SavantCommented:
You said...

I am on an Amazon VPS with 3 GB of memory.

And mysqltuner suggests...

innodb_buffer_pool_size (>= 5.6G) if possible.

Get a Kimsufi 32G dedicated server for $30/month + see if this fixes your problem.

While mysqltuner suggestions aren't written in stone, they tend to be right the majority of the time...

This also means, once you bump up to 5.6G + run for a while, likely mysqltuner will have additional tunings which will require more memory.

Based on the mysqltuner suggestions + normal Linux memory requirements, your minimal viable memory footprint will be 16G + more likely 32G... so...

Your Amazon 3G VPS is just to low.
0
sharingsunshineAuthor Commented:
David,  from now on I would appreciate you limiting your answers to providing answers to fixing my problems.  Not to replacing or changing hosts.  I won't be switching from Amazon just to solve a problem ever.

If there were continuous problems certainly I would consider it.

If you have a specific answer, I certainly want to know about it.
0
arnoldCommented:
The point being your resource use versus available is the issue. Point being made is that your under-spec'd your server.
Raising the Available memory to 6gb may address/resolve..

Look at suggestion on line 13.

Your queries might be fine tuned to reduce the resource demand.
0
Newly released Acronis True Image 2019

In announcing the release of the 15th Anniversary Edition of Acronis True Image 2019, the company revealed that its artificial intelligence-based anti-ransomware technology – stopped more than 200,000 ransomware attacks on 150,000 customers last year.

sharingsunshineAuthor Commented:
I doubled the memory and it still crashed.  If you have suggestions on how to fine tune the queries, that would be great.

It seems whatever resources are thrown at it get used up.  As far as an error segfault 4 is coming up.  Does anyone know how to troubleshoot that error?
0
arnoldCommented:
can you post the actual error?


You have to review your common queries to see what resources they consume.
does your setup include myphpadmin
Did you make the suggested adjustments that you included in your posting?

segfault is a different/could be ....

Configuring advanced logging, i.e. slow query log.
The observation of the tunner that you had many large set queries .......
0
sharingsunshineAuthor Commented:
The error only shows when it won't reboot.  On Amazon you can look at the instance screenshot and that is where I saw it.  After 3 reboots it came up and it isn't showing in the logs either.

I have atop installed and mysql jumps from 11% to 24% in one minute.  

                                  --------------                                         1m0s elapsed
PRC | sys    3.02s | user  11.12s  |              | #proc    348 | #trun      3  | #tslpi   225 |              | #tslpu     0  | #zombie    0 | clones   209 |               | #exit    214 |
CPU | sys       5% | user     18%  | irq       0% |              | idle    377%  | wait      0% |              |               | steal     0% | guest     0% | curf 3.41GHz  | curscal   ?% |
cpu | sys       2% | user      6%  | irq       0% |              | idle     92%  | cpu003 w  0% |              |               | steal     0% | guest     0% | curf 3.41GHz  | curscal   ?% |
cpu | sys       1% | user      4%  | irq       0% |              | idle     94%  | cpu001 w  0% |              |               | steal     0% | guest     0% | curf 3.42GHz  | curscal   ?% |
cpu | sys       1% | user      4%  | irq       0% |              | idle     95%  | cpu000 w  0% |              |               | steal     0% | guest     0% | curf 3.40GHz  | curscal   ?% |
cpu | sys       1% | user      4%  | irq       0% |              | idle     95%  | cpu002 w  0% |              |               | steal     0% | guest     0% | curf 3.40GHz  | curscal   ?% |
CPL | avg1    0.15 | avg5    0.03  |              | avg15   0.01 |               |              | csw   300427 | intr  157207  |              |              | numcpu     4  |              |
MEM | tot     7.4G | free    4.9G  | cache   1.3G | dirty   0.5M | buff   53.6M  | slab   74.9M | slrec  50.0M | shmem   1.8M  | shrss   0.0M | vmbal   0.0M | hptot   0.0M  | hpuse   0.0M |
SWP | tot     0.0M | free    0.0M  |              |              |               |              |              |               |              |              | vmcom   2.5G  | vmlim   3.7G |
NET | transport    | tcpi     535  | tcpo     728 | udpi      11 | udpo      11  | tcpao      8 | tcppo     16 | tcprs     13  | tcpie      0 | tcpor      6 | udpnp      0  | udpie      0 |
NET | network      | ipi      553  | ipo      468 |              | ipfrw      0  | deliv    552 |              |               |              |              | icmpi      6  | icmpo      0 |
NET | eth0    ---- | pcki     555  | pcko     754 | sp    0 Mbps | si    6 Kbps  | so  118 Kbps | coll       0 | mlti       0  | erri       0 | erro       0 | drpi       0  | drpo       0 |
Window resized to 189x50...
  PID   TID  PPID CID            VPID  CTID RUID     RGID      EUID     EGID     SUID     SGID      FSUID    FSGID      STDATE     STTIME    ENDATE    ENTIME  ST  EXC S  CPU CMD         1/8
 4574     -     1 host--------      0     0 mysql    mysql     mysql    mysql    mysql    mysql     mysql    mysql    2018/08/25  19:50:01   active    active  --    - S  11% mysqld
 8642     -  4336 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/25  23:11:23   active    active  --    - R   8% php-fpm
17596     -  4336 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  03:01:18   active    active  --    - S   1% php-fpm
15614     -  4336 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  02:49:45   active    active  --    - S   1% php-fpm
15612     -  4336 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  02:49:36   active    active  --    - S   1% php-fpm
16060     -  4336 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  02:52:20   active    active  --    - S   0% php-fpm
15848     -  4336 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  02:51:21   active    active  --    - S   0% php-fpm
15613     -  4336 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  02:49:41   active    active  --    - S   0% php-fpm
17511     -     1 host--------      0     0 root     root      root     root     root     root      root     root     2018/08/26  00:00:01   active    active  --    - R   0% atop
 5641     -     1 host--------      0     0 root     root      root     root     root     root      root     root     2018/08/25  19:54:54   active    active  --    - S   0% fail2ban-serve
15611     -  4336 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  02:49:20   active    active  --    - S   0% php-fpm
13588     -  3989 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  02:37:59   active    active  --    - S   0% httpd
20430     -  3989 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  03:16:45   active    active  --    - S   0% httpd
20862     -  3989 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  03:19:50   active    active  --    - S   0% httpd
21147     -     - ?                 -     - root     root      -        -        -        -         -        -        2018/08/26  03:21:40 2018/08/26 03:21:41 NE    0 E   0% <dhclient-scr>
12064     -  3989 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  02:29:13   active    active  --    - S   0% httpd
 4025     -  3989 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  01:45:15   active    active  --    - S   0% httpd
 5238     -  3989 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  01:50:51   active    active  --    - S   0% httpd
 7468     -  3989 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  02:04:07   active    active  --    - S   0% httpd
 7591     -  3989 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  02:04:10   active    active  --    - S   0% httpd
14425     -  3989 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  02:42:57   active    active  --    - S   0% httpd
20000     -  3989 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  03:15:02   active    active  --    - S   0% httpd
 2830     -     1 host--------      0     0 root     root      root     root     root     root      root     root     2018/08/25  19:44:14   active    active  --    - S   0% crond
 2082     -     1 host--------      0     0 dbus     dbus      dbus     dbus     dbus     dbus      dbus     dbus     2018/08/25  19:44:11   active    active  --    - S   0% dbus-daemon
    8     -     2 host--------      0     0 root     root      root     root     root     root      root     root     2018/08/25  19:44:07   active    active  --    - I   0% rcu_sched
 1403     -     2 host--------      0     0 root     root      root     root     root     root      root     root     2018/08/25  19:44:10   active    active  --    - I   0% kworker/2:1H
21114     -     - ?                 -     - root     root      -        -        -        -         -        -        2018/08/26  03:21:39 2018/08/26 03:21:40 NE    0 E   0% <dhclient-scr>
21197     -     - ?                 -     - root     root      -        -        -        -         -        -        2018/08/26  03:21:41 2018/08/26 03:21:42 NE    0 E   0% <dhclient-scr>
21249     -     - ?                 -     - root     root      -        -        -        -         -        -        2018/08/26  03:21:42 2018/08/26 03:21:43 NE    0 E   0% <dhclient-scr>
 2059     -     1 host--------      0     0 root     root      root     root     root     root      root     root     2018/08/25  19:44:11   active    active  --    - S   0% rsyslogd
 1467     -     1 host--------      0     0 root     root      root     root     root     root      root     root     2018/08/25  19:44:10   active    active  --    - S   0% systemd-journa
 4336     -     1 host--------      0     0 root     root      root     root     root     root      root     root     2018/08/25  19:49:05   active    active  --    - S   0% php-fpm
 3989     -     1 host--------      0     0 root     root      root     root     root     root      root     root     2018/08/25  19:48:40   active    active  --    - S   0% httpd
31736     -  2827 host--------      0     0 root     root      root     root     root     root      root     root     2018/08/26  01:18:02   active    active  --    - S   0% sshd
 7352     -  2827 host--------      0     0 root     root      root     root     root     root      root     root     2018/08/25  20:03:24   active    active  --    - S   0% sshd

Open in new window


Here it is at 24%

             2018/08/26  03:23:03                                        --------------                                         1m0s elapsed
PRC | sys    6.33s | user  21.79s  |              | #proc    344 | #trun      2  | #tslpi   226 |              | #tslpu     0  | #zombie    0 | clones   212 |               | #exit    209 |
CPU | sys      10% | user     35%  | irq       0% |              | idle    355%  | wait      0% |              |               | steal     0% | guest     0% | curf 3.41GHz  | curscal   ?% |
cpu | sys       3% | user     10%  | irq       0% |              | idle     87%  | cpu000 w  0% |              |               | steal     0% | guest     0% | curf 3.41GHz  | curscal   ?% |
cpu | sys       3% | user      9%  | irq       0% |              | idle     88%  | cpu002 w  0% |              |               | steal     0% | guest     0% | curf 3.40GHz  | curscal   ?% |
cpu | sys       3% | user      7%  | irq       0% |              | idle     90%  | cpu003 w  0% |              |               | steal     0% | guest     0% | curf 3.41GHz  | curscal   ?% |
cpu | sys       2% | user      8%  | irq       0% |              | idle     90%  | cpu001 w  0% |              |               | steal     0% | guest     0% | curf 3.40GHz  | curscal   ?% |
CPL | avg1    0.24 | avg5    0.09  |              | avg15   0.03 |               |              | csw   643043 | intr  333707  |              |              | numcpu     4  |              |
MEM | tot     7.4G | free    5.0G  | cache   1.3G | dirty   0.6M | buff   53.6M  | slab   74.9M | slrec  50.0M | shmem   1.8M  | shrss   0.0M | vmbal   0.0M | hptot   0.0M  | hpuse   0.0M |
SWP | tot     0.0M | free    0.0M  |              |              |               |              |              |               |              |              | vmcom   2.5G  | vmlim   3.7G |
NET | transport    | tcpi     252  | tcpo     435 | udpi      20 | udpo      20  | tcpao     10 | tcppo     16 | tcprs      0  | tcpie      0 | tcpor      2 | udpnp      0  | udpie      0 |
NET | network      | ipi      295  | ipo      331 |              | ipfrw      0  | deliv    280 |              |               |              |              | icmpi      8  | icmpo      1 |
NET | eth0    ---- | pcki     297  | pcko     458 | sp    0 Mbps | si    4 Kbps  | so   54 Kbps | coll       0 | mlti       0  | erri       0 | erro       0 | drpi       0  | drpo       0 |

  PID   TID  PPID CID            VPID  CTID RUID     RGID      EUID     EGID     SUID     SGID      FSUID    FSGID      STDATE     STTIME    ENDATE    ENTIME  ST  EXC S  CPU CMD         1/8
 4574     -     1 host--------      0     0 mysql    mysql     mysql    mysql    mysql    mysql     mysql    mysql    2018/08/25  19:50:01   active    active  --    - S  24% mysqld
15848     -  4336 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  02:51:21   active    active  --    - S   9% php-fpm
15612     -  4336 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  02:49:36   active    active  --    - R   9% php-fpm
 8642     -  4336 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/25  23:11:23   active    active  --    - S   1% php-fpm
15614     -  4336 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  02:49:45   active    active  --    - S   1% php-fpm
15613     -  4336 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  02:49:41   active    active  --    - S   1% php-fpm
17596     -  4336 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  03:01:18   active    active  --    - S   1% php-fpm
16060     -  4336 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  02:52:20   active    active  --    - S   1% php-fpm
15611     -  4336 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  02:49:20   active    active  --    - S   1% php-fpm
17511     -     1 host--------      0     0 root     root      root     root     root     root      root     root     2018/08/26  00:00:01   active    active  --    - R   0% atop
 5641     -     1 host--------      0     0 root     root      root     root     root     root      root     root     2018/08/25  19:54:54   active    active  --    - S   0% fail2ban-serve
12064     -  3989 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  02:29:13   active    active  --    - S   0% httpd
 7468     -  3989 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  02:04:07   active    active  --    - S   0% httpd
14425     -  3989 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  02:42:57   active    active  --    - S   0% httpd
13588     -  3989 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  02:37:59   active    active  --    - S   0% httpd
20862     -  3989 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  03:19:50   active    active  --    - S   0% httpd
20000     -  3989 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  03:15:02   active    active  --    - S   0% httpd
    8     -     2 host--------      0     0 root     root      root     root     root     root      root     root     2018/08/25  19:44:07   active    active  --    - I   0% rcu_sched
21359     -     - ?                 -     - root     root      -        -        -        -         -        -        2018/08/26  03:22:55 2018/08/26 03:22:56 NE    0 E   0% <dhclient-scr>
 4025     -  3989 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  01:45:15   active    active  --    - S   0% httpd
 5238     -  3989 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  01:50:51   active    active  --    - S   0% httpd
 7591     -  3989 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  02:04:10   active    active  --    - S   0% httpd
20430     -  3989 host--------      0     0 apache   apache    apache   apache   apache   apache    apache   apache   2018/08/26  03:16:45   active    active  --    - S   0% httpd
    1     -     0 host--------      0     0 root     root      root     root     root     root      root     root     2018/08/25  19:44:07   active    active  --    - S   0% systemd
 2062     -     1 host--------      0     0 root     root      root     root     root     root      root     root     2018/08/25  19:44:11   active    active  --    - S   0% rngd
21326     -     - ?                 -     - root     root      -        -        -        -         -        -        2018/08/26  03:22:54 2018/08/26 03:22:55 NE    0 E   0% <dhclient-scr>
21410     -     - ?                 -     - root     root      -        -        -        -         -        -        2018/08/26  03:22:56 2018/08/26 03:22:57 NE    0 E   0% <dhclient-scr>
21461     -     - ?                 -     - root     root      -        -        -        -         -        -        2018/08/26  03:22:57 2018/08/26 03:22:58 NE    0 E   0% <dhclient-scr>
 2059     -     1 host--------      0     0 root     root      root     root     root     root      root     root     2018/08/25  19:44:11   active    active  --    - S   0% rsyslogd
 1467     -     1 host--------      0     0 root     root      root     root     root     root      root     root     2018/08/25  19:44:10   active    active  --    - S   0% systemd-journa
 4336     -     1 host--------      0     0 root     root      root     root     root     root      root     root     2018/08/25  19:49:05   active    active  --    - S   0% php-fpm
 3989     -     1 host--------      0     0 root     root      root     root     root     root      root     root     2018/08/25  19:48:40   active    active  --    - S   0% httpd
 2741     -     1 host--------      0     0 root     root      root     root     root     root      root     root     2018/08/25  19:44:14   active    active  --    - S   0% master
 2463     -     1 host--------      0     0 root     root      root     root     root     root      root     root     2018/08/25  19:44:13   active    active  --    - S   0% dhclient
 2830     -     1 host--------      0     0 root     root      root     root     root     root      root     root     2018/08/25  19:44:14   active    active  --    - S   0% crond

Open in new window


I haven't made any changes regarding the tuner recommendations because of my inexperience with mysql and not really understanding what changes might be causing the error.

I do have phpmyadmin installed.

Is there some tool that can tell me what jobs are running at that time and see if I can find the problem that way?
0
arnoldCommented:
what is the function of your server?
phpmyadmin as is the tunner offer recommendations based on statistics data

how big is the database is?

segmentation fault is a separate..

in short, make the recommended suggestions and see .

DB server gets hit ..and commonly is at the center of ... much depends on what your server does.

top is a snapshot and does not shed much ligh though I notice you have many apache processes....
0
arnoldCommented:
check with AWS to have them diagnose/determine the cause for the crash of the server/VPS.

issues related to mysql, apache etc. will trigger the crash of the application not the entire server.
Or the issue he server becomes unresponsive at which point you "try a hard reboot"....
0
David FavorLinux/LXD/WordPress/Hosting SavantCommented:
"David,  from now on I would appreciate you limiting your answers to providing answers to fixing my problems.  Not to replacing or changing hosts.  I won't be switching from Amazon just to solve a problem ever."

I'll rephrase.

You require 16G-32G+ of memory for your database to work.

You can acquire extra memory from AWS, you'll pay a truckload of money for this + you'll still be running inside the massively slow AWS VM system. If slow + expensive works for you, use AWS.

Or you can use KimSufi or some other inexpensive dedicated hosting where your memory cost will be less + your database will run at bare metal speed, rather than AWS VM speed.

There is no one solution to anything.

Simple solution is just pay whatever AWS requires to bump your memory up to 32G.

"If there were continuous problems certainly I would consider it."

There is a continuous problem. Read over my comment again. The mysqltuner output shows the exact problem as memory starvation.
0
Bernard S.CTOCommented:
where are the connections by "root@localhost" coming from? if tyou run a script which runs lots off connection from root, then this will somehow load the memory until the unworking attempt is considered ended and the flushed, freeing some resources for other uses.
(remember that the "root" mysql user is something different from unix "root" user.

Also to avod the refused connection, consider creating a file .my.cnf into the appropriate directory, preseumably here /root/
0
sharingsunshineAuthor Commented:
I am not sure where those connections are coming from.  Do you have any suggestions on how to troubleshoot their origin.  I am not clear what you are suggesting with my.cnf?
0
arnoldCommented:
your coe has to be checked to make sure the connections are closed
0
sharingsunshineAuthor Commented:
What is coe?
0
arnoldCommented:
Code,

You ran the tunner, implement the suggestions.

The issue you noted, your server crashes about every day so the tunner does not have a large enough data set ...

Do you have myphpadmin exposed, accessible from the outside?
0
gr8gonzoConsultantCommented:
I have to disagree with the other experts, at least for now. One thing I try to drill into every developer, DBA, and system admin I meet:

NEVER add hardware or increase limits to solve a resource problem
without knowing the root cause first.


Or the simpler version:

NEVER try to fix a problem without knowing the root cause first.

It's like trying to fix a sinking ship by using a bigger and bigger and bigger bucket to bail water instead of just patching the hole.

Let's say that you add a bunch more memory and it "fixes" the problem for now but then the problem returns after a year. Now what? At some point, it's just going to be way too expensive to keep upgrading physical resources, especially if you don't have the supporting traffic. Proper system administration should never include the phrase, "I don't know exactly why that worked, but it worked." If you don't know why something worked, then you cannot be certain that it truly did work, or if it just masked the symptoms.

While it ultimately may end up being that you indeed do need more memory, I've seen servers run some pretty heavy loads under very limited resources, and usually bad performance (not crashing) is the first sign that you need more physical resources.

Your problem and atop results smell like you have PHP scripts that are running a little wild and either selecting way too much data or not using proper, efficient means to interact with the database (e.g. queries within loops within loops leading to thousands and thousands of queries for every script execution).

While I normally agree with the MySQL tuner results, I'd say that in this case, you should go in the opposite direction and implement some LOWER limits that fall well within your available memory limits. Then debug the issue and after the issue is fixed, implement the suggestions (the query cache, in particular, is almost always a very important piece to have and it will usually take a large load off your database, especially if you have tables that have a lot more reads than writes).

Now, onto specific recommendations:

1. Make sure you have the slow query log enabled. Set it to a threshold of 4 seconds. Pop into the MySQL shell and run these 4 queries:
SET GLOBAL slow_query_log = 'ON';
SET GLOBAL long_query_time = 4;
SET GLOBAL slow_query_log_file = '/var/log/slowqueries.log';
SET GLOBAL log_queries_not_using_indexes = 'ON';

Open in new window


To turn it off later, just run:
SET GLOBAL slow_query_log = 'OFF';

Open in new window


2. Give the server some time to be hit with any slow queries. In the meantime, stay in the MySQL shell and keep issuing this query every 30 seconds or so:
SHOW PROCESSLIST;

That will show you the other queries that are running on the server at that particular moment in time, along with how long they've been running, who was running them, etc... You'll want to pay attention to any active (non-sleeping, non-connecting)  entries that have a query (besides your own SHOW PROCESSLIST one) and see how long they've been running. If you see a query that has been running for more than 10-15 seconds, you should start tracking down that query to figure out why. While it's POSSIBLE to have large queries that run for a long time (because they have so much data), it's in your best interests to minimize these whenever possible.

It's pretty common for people to forget to add indexes to columns they're using in WHERE clauses on large tables, forcing the database to run heavy, long routines to search through the entire table, or for developers to not realize that under certain circumstances, their SELECT query doesn't have any WHERE clause but it does have an ORDER BY, so MySQL has to pretty much copy the entire table, sort it, and then return it (possible tons of memory usage).

Ultimately, if I had to place any bets, I'd say that the problem is a code/database interaction issue like a bad query. You'll notice that in both atop results, you have 1 or 2 PHP-FPM processes that are sitting at about 8%-9% indicating that they're probably the sources of the database activity (they are different child processes in each snapshot).

Also, one last thing - it's pretty unusual to find any code still using persistent connections, but if your SHOW PROCESSLIST shows a ton of empty connections, double-check your PHP code to make sure you're not using persistent connections with the database. Normal database connections using mysqli_connect or new mysqli (you shouldn't be using the old "mysql_" functions anymore) will automatically close at the end of the script.

Finally, after you've combed through some code and SHOW PROCESSLIST results, go check the slow query log and see what goodies are in there. I'd be surprised if it was empty, but you never know. Anyway, assuming there are entries in there, look for queries that take a long time to run, and go track down the code that executes the query and figure out if there's a way to optimize the code and/or the query.

My article has a couple pointers on indexes that might come in handy:
https://www.experts-exchange.com/articles/1250/3-Ways-to-Speed-Up-MySQL.html
1
arnoldCommented:
gr8gonzo no disagreement here, but the VM "crashes" so your dynamic settings would go away once the service/server restarts.
To get meaningful stats over time, adding memory to provide a context to the resource consumption.


The tunner provided some guidance.
1
gr8gonzoConsultantCommented:
Yes, I know the dynamic settings will reset, and that's sort of the point. The slow query log file is recorded on disk, so you'd get data leading up to the point of the crash, which is actually a great thing because you'd be able to see what the last query was right before the crash.

However, my gut is saying that the problem is likely a continued one and will surface some evidence well before the crash happens.

The tuner did provide some guidance, but you need to be careful with trusting its guidance, especially on systems that are crashing frequently. The tuner bases its findings off of long-term performance data, and if the server hasn't been running for very long, those findings could be VERY wrong. It's sort of like the joke:

INTERVIEWER: What would you say are your st---
ME: Strengths? Making inferences from minimal data.
INTERVIEWER: Okay. And your we---
ME: Wheat allergies? None whatsoever.

While MySQL Tuner is a great tool for tweaking performance, it is usually a very bad tool for debugging MySQL crashes. If MySQL is crashing because of bad queries, then the tuner will often not know that they are actually bad queries, so it will try to make recommendations based on the assumption that they're GOOD queries. So you might end up following the advice to try to configure your system to make it accommodate queries that shouldn't be running in the first place. It's like a bank saying, "Bank robbers are negatively impacting our work. We brought in a consultant who said that 3 hours out of every day are being wasted by bank robberies, so if we just bring the cash out of the vault ahead of time, the robberies will only take 1 hour, and we'll be able to get back to business sooner."

That said, there are some good recommendations that came out of the tuner (e.g. turning on query cache), but again, those are all performance tweaks that should be addressed after stability is achieved.

Here is a good article commenting on the innodb_buffer_pool_size setting in particular, since it's potentially related beyond just performance:
https://www.percona.com/blog/2015/06/02/80-ram-tune-innodb_buffer_pool_size/

In case that link ever fails, the very short version states the rule of thumb is to set innodb_buffer_pool_size to 80% of your RAM as a starting point. However, if I'm on a server that is providing multiple services (web, database, mail, etc...), I don't usually follow that rule. I'd start the buffer at maybe 30%-40% of your available RAM (and incidentally, this doesn't have any impact if someone isn't using the InnoDB engine for any of their MySQL databases). You can always increase later as needed, after evaluating the actual usage.

Those other services can spike and fall in memory usage (e.g. virus scanners on mail services, or heavy web processes), so all of that system RAM is not exclusively available for MySQL's use.

My final comment - the OP did not say that the VM itself crashed. I'd be a little surprised if it did. While it's not impossible for the entire VM to crash from a drastic problem, my guess is that he meant that the MySQL service crashed (which subsequently makes the web site unavailable). If the entire VM is crashing, then that's not a resource problem - that's a significantly bad build of MySQL. Any software that crashes should do so without taking the entire OS down with it. If it's taking the entire OS down, then it has some bad interactions with the kernel (e.g. taking up kernel-level resources and not releasing them). However, before I go any further on that topic, the OP can always confirm whether the entire VM is going down (where he/she has to power the entire thing back on again), or if it's just the service crashing.
1
sharingsunshineAuthor Commented:
I have been able to find an old program that needed to be upgraded and that took away the server crashing daily, in addition after a reboot fail2ban failed to work.  So, the bots hitting my server at 1 page a second overwhelmed the site.  Now, I have those issues mitigated.

However, it is losing about .3 GB every day in memory.  So, no matter the size of memory it is still using up those resources.

I have read that my forum SMF isn't able to use mysqli which is what my PHP which is 7 uses.  Could that be the cause of the leak?

I have phpmyadmin locked down to my ip number only.
0
gr8gonzoConsultantCommented:
I'm not familiar with the SMF forum specifically or what it uses. At a glance, it looks like there was an update in February 2018, so it'd be odd that a public application would be that far behind in code updates, so maybe it's using something else besides mysqli (like PDO or something) to connect to the database. Where did you read about it not being able to use mysqli?

I'd still recommend following my steps to try to find large / slow queries as your first step in determining root cause.
0
sharingsunshineAuthor Commented:
I am getting this error on creating the log

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 79138
Server version: 10.2.10-MariaDB MariaDB Server

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> SET GLOBAL slow_query_log = 'ON';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> SET GLOBAL long_query_time = 4;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> SET GLOBAL slow_query_log_file = '/var/log/slowqueries.log';
ERROR 1231 (42000): Variable 'slow_query_log_file' can't be set to the value of '/var/log/slowqueries.log'
MariaDB [(none)]> SET GLOBAL log_queries_not_using_indexes = 'ON';
Query OK, 0 rows affected (0.00 sec)

Open in new window

0
sharingsunshineAuthor Commented:
Initially, I had to restart the server.  That problem has been fixed but still the mysql keeps consuming memory and growing in size.
0
arnoldCommented:
The addition of the information provided by gr8gonzo is to determine the cause first, the adjustments can not resolve the issue.
now look at the /var/log/slowqueries.log and address the common query that exceeds the 4 seconds to respond.
have not recently looked at these logs recently ...

If memory serves,

awk ' { print $3 } ' /var/log/slowqueries.log | uniq -c | sort -rn

the above if the third element is the query, will count number of similar queries, and then will sort based on highest repeated query

as the suggestion earlier, this will let you see the common query that is taking longer than 4 seconds to complete (on the DB, the rendering of the response in the browser does not come into play..)

you can use explain <query> and it will tell you whether as other suggestions to add an index for the column either in a where clause, or if this is a join statement, to add an index on the column used in the join....... if multiple columns, .. date columns if memory serves can not be indexed.......  (the logging guidance gr8gonzo included will record queries that are in need of an index)

then see if the addition reduces the appearance of this query in future reports. work your way through the queries logged..
one of the tunner suggestions deals with "many distinct" queries that do not include a limit ( not quite sure whether this means no where clause, or ........)
1
sharingsunshineAuthor Commented:
I'd be happy to but the log isn't being created.  Here is a screenshot of 2 sessions one with the ls -la and the pwd to show it is located there and the other with the error mysql is throwing.

https://gyazo.com/e6300e90ebd6edc88287f6ea709da373
0
arnoldCommented:
Execute
chown mysql:mysql /var/log/slowqieries.log

This will change file ownership and see if that starts getting updated.
2
sharingsunshineAuthor Commented:
that got me going, thanks.  Best I can tell the log isn't being limited to 4 seconds or greater.  Also, the awk just shows the number "1"

/usr/libexec/mysqld, Version: 10.2.10-MariaDB (MariaDB Server). started with:
Tcp port: 3306  Unix socket: /var/lib/mysql/mysql.sock
Time                 Id Command    Argument
# Time: 180901 15:26:26
# User@Host: root[root] @ localhost []
# Thread_id: 79802  Schema: thefruga_tflnews  QC_hit: No
# Query_time: 0.000665  Lock_time: 0.000034  Rows_sent: 377  Rows_examined: 734
# Rows_affected: 0
use thefruga_tflnews;
SET timestamp=1535829986;
SELECT option_name, option_value FROM wp_options WHERE autoload = 'yes';
# User@Host: root[root] @ localhost []
# Thread_id: 79802  Schema: thefruga_tflnews  QC_hit: No
# Query_time: 0.001223  Lock_time: 0.000022  Rows_sent: 2570  Rows_examined: 2571
# Rows_affected: 0
SET timestamp=1535829986;
SELECT botnickname, id FROM wp_sbb_blacklist WHERE `botstate` LIKE 'Enabled';
# User@Host: root[root] @ localhost []
# Thread_id: 79802  Schema: thefruga_tflnews  QC_hit: No
# Query_time: 0.000537  Lock_time: 0.000049  Rows_sent: 231  Rows_examined: 245
# Rows_affected: 0
SET timestamp=1535829986;
SELECT name, val FROM wp_wfConfig WHERE autoload = 'yes';
# Time: 180901 15:26:27
# User@Host: root[root] @ localhost []
# Thread_id: 79803  Schema: thefruga_tflnews  QC_hit: No
# Query_time: 0.000809  Lock_time: 0.000049  Rows_sent: 377  Rows_examined: 734
# Rows_affected: 0
SET timestamp=1535829987;
SELECT option_name, option_value FROM wp_options WHERE autoload = 'yes';
# User@Host: root[root] @ localhost []
# Thread_id: 79803  Schema: thefruga_tflnews  QC_hit: No
# Query_time: 0.001471  Lock_time: 0.000028  Rows_sent: 2570  Rows_examined: 2571
# Rows_affected: 0
SET timestamp=1535829987;
SELECT botnickname, id FROM wp_sbb_blacklist WHERE `botstate` LIKE 'Enabled';
# User@Host: root[root] @ localhost []
# Thread_id: 79803  Schema: thefruga_tflnews  QC_hit: No
# Query_time: 0.000609  Lock_time: 0.000064  Rows_sent: 231  Rows_examined: 245
# Rows_affected: 0
SET timestamp=1535829987;
SELECT name, val FROM wp_wfConfig WHERE autoload = 'yes';
# User@Host: root[root] @ localhost []
# Thread_id: 79804  Schema: thefruga_tflnews  QC_hit: No
# Query_time: 0.000651  Lock_time: 0.000043  Rows_sent: 377  Rows_examined: 734
# Rows_affected: 0

Open in new window

0
arnoldCommented:
Oh,

This is a wordpress setup

not sure whether you can change the query from using like  for
wp_sbb_blacklist botstate like 'Enabled' to change it to botstate='Enabled' and add an index on the botstate column as well as in wp_options add an index on autoload if missing..


With a Wordpress depending on the size of your site, the issue might be related to resource...
though the initial data is not enough....


but as time goes by, you will see other queries, the point to look is how many rows were scanned to come up with the results.
0
gr8gonzoConsultantCommented:
So the options I mentioned will log 2 different types of queries:
1. Queries that take over X seconds (SET GLOBAL long_query_time = 4)
2. Queries that don't use an index (SET GLOBAL log_queries_not_using_indexes = 'ON')

Normally queries that don't use indexes are bad things, but they are occasionally okay on smaller tables. Given that WP tends to have lots of small tables that often either don't have indexes or MySQL doesn't use them (sometimes it's faster to do a full table scan of a small table than to incur the overhead of using the index), you're probably going to end up with a lot of those kinds of queries. As you can see next to the query time, none of the examples in the log so far are slow. They're all well under 1 second and have pretty small payloads, so none of these are the source of the problem.

So at this point, it might be better to turn off that secondary option:
SET GLOBAL log_queries_not_using_indexes = 'OFF';

...and just let the slow query log accumulate nothing but entries that took 4 or more seconds to run.
0
gr8gonzoConsultantCommented:
On a side note, I checked out your site and did a little performance profiling of the different domains (news, life, the SMF forum). I half-expected there to be more-than-average latency, but everything came back pretty snappy, so that's good (although your news site doesn't seem to have a security certificate - you might want to consider adding one for the little SEO bump). You're probably getting a nice boost in performance from using php-fpm instead of the typical mod_php Apache plugin.

One additional avenue to consider (if you don't do it already) is to check your Apache access logs. Depending on the log format you're using, the log entries MIGHT contain the size of the response body along with how long it took Apache to respond to the request. So if you have that piece of data in your logs, then given that the site is so responsive (speed-wise), if you look for entries that took several seconds, anything that matches that criteria might be something to look at more closely.

Also check the Apache and PHP error logs in case they indicate any unexpected crashes or faults.
1
sharingsunshineAuthor Commented:
I turned off the index switch.  I then decided to make a new file without the indexes.  But now it isn't logging anything.  Please advise.
0
arnoldCommented:
Permissions?

Did you change the log directive in MySQL?

The suggestions are based on your question, the check gr8gonzo provided deals with queries that take longer than 4 seconds to respond. Your issue might not be related to the duration of the query but the number of hits on your Apache server.

MySQL is a memory dependent ...

Let's get to the begining, when your system experiences issues, what do you do to recover? What services are not running? What services need to be started, or is your past just restarted it?
1
sharingsunshineAuthor Commented:
I changed the permission to mysql:mysql

I restart maria

I am considering just setting up a cron to keep restarting maria until I can get this solved.

You mentioned to look at how many rows are read and my forum s/w is reading around 2,500 rows at a time.

On the log file.  I stopped the log and restarted maria.  I then ran through the commands again and then restarted Maria again.  I just checked and still no entries.

Here are the permissions - https://gyazo.com/057136ddc46eeda003c254e4fcb3bd6f

Am I undoing something by restarting Maria each time?
0
arnoldCommented:
Restarting the database server masks what might be going on, check as gr8gonzo suggested the Apache logs to make sure you are not being subjected to a denial of service attack by implementing counter measures, such as limiting number of connections from the same source.

For effectively tunning MySQL, mariadb, the more stats the better, meaning the db has to run for as long as possible.
The other is lower the slow query from 4 to 1

When you restart the service, the log settings are dynamic, meaning once the service restarts it stops loging until the command gr8gonzo provide are rexecuted.

To make these settings permanent, they need to be set in the my.cnf file.

IMHO, memory consumption ...... Is not an explicit indication of an issue. Further assessment and determination has to be made.

Instead of restarting mariadb, restart Apache and see if the issue clears up, I.e. The many connection to MySQL will go away pointing to Apache being inundated with requests it can not fulfill
0
sharingsunshineAuthor Commented:
I have restarted apache and it does nothing for the mysql growth and memory leak.

I do have Sar running if that will give more information.

Once, I restarted Maria I reloaded the mysql commands but still it isn't logging.
0
arnoldCommented:
The threshold is for queries that take longer than 4 seconds.
You are toooooooooo focused on the wrong thing. A memory leak has a specific meaning, mariadb, MySQL expand memory allocation based on queries, hits, they do not release memory as a general practice.

A memory leak is a process that continues to expand in memory consumption without justification. I.e. Few requests are seen, but the process continuously grows in memory use.

Let mariadb run, if and when the issue cones up, confirm the issue is on the SQL side.

Increase the number of connections MySQL Can accept.

The logs shoukd point to where the issue is, you've configured WP to use root as the user, look at migrating to a less privileged user.
I.e. Create a user specific to WP with smoke access to the WP database.
This will help differentiate hits on the server from WP versus other.

Look in variables as was suggested to make sure you are not using persistent connections that keep expanding/increasing. A persistent connection precludes mariadb/MySQL from releasing cached, temporary resources.

Check WP configuration if using/set to persistent to limit how many such connections it ssn open.

Look at Apache how many child/requests it can handle.

Look at how many php processies do you gave running when there are large quantity of open connections in the db side.


Top, atop is only one stat,
How much of your total memory, does MySQL/mariadb consume?

Prioritizing mariadb/MySQL (renice) compared to other could speed up responses......


What are the complete services this server provides, web, db, mail server, etc.
1
gr8gonzoConsultantCommented:
So if you're not getting any logging after changing the slow query log to just handle queries over 4 seconds, then you might not be getting any queries that meet that criteria, which is a good thing if it's truly accurate. There's no harm in keeping that running, so if you do restart the DB server, make sure you re-issue the commands to enable the slow query log.

What was the output of the SHOW PROCESSLIST query after a day or so of letting Maria run?
0
sharingsunshineAuthor Commented:
Thanks for the clarification.  I did restart the DB server thinking that made the command active.  I see now from your clarification that was inaccurate.  

I have now re-issued those commands (I did leave off the index command) and it is filling up again.

MariaDB [(none)]> MariaDB [(none)]> SHOW PROCESSLIST;
+------+-------------+-----------+------------------+---------+------+--------------------------+------------------+----------+
| Id   | User        | Host      | db               | Command | Time | State                    | Info             | Progress |
+------+-------------+-----------+------------------+---------+------+--------------------------+------------------+----------+
|    1 | system user |           | NULL             | Daemon  | NULL | InnoDB purge coordinator | NULL             |    0.000 |
|    2 | system user |           | NULL             | Daemon  | NULL | InnoDB purge worker      | NULL             |    0.000 |
|    3 | system user |           | NULL             | Daemon  | NULL | InnoDB purge worker      | NULL             |    0.000 |
|    4 | system user |           | NULL             | Daemon  | NULL | InnoDB purge worker      | NULL             |    0.000 |
|    5 | system user |           | NULL             | Daemon  | NULL | InnoDB shutdown handler  | NULL             |    0.000 |
| 5135 | root        | localhost | NULL             | Query   |    0 | init                     | SHOW PROCESSLIST |    0.000 |
| 5149 | root        | localhost | thefruga_tflnews | Sleep   |    0 |                          | NULL             |    0.000 |
+------+-------------+-----------+------------------+---------+------+--------------------------+------------------+----------+
7 rows in set (0.00 sec)

Open in new window

0
arnoldCommented:
Look at the queries being added, use explain <query>
And see what it tells you,
If there are a large thousands of rows reviewed while returning a few dozen.
Checking whether there is an index on the columns used in the where clause, and as part of a join

Select * from table1 a inner join table2 b on a.column=b.column
Where a.somecolumn= and b.anothercolumn

Here you will check whether an index on either, some, or all are worth while,

The other on the long query entries are those who return a massive, large dataset that usually will be the queries that put the demand in the MySQL server to get memory allocations.
0
gr8gonzoConsultantCommented:
How long has the server been up before you did the SHOW PROCESSLIST query? Ideally, we want to wait about 24 hours without restarting in case the problem happens overnight.

The goal of this exercise is to see if there are any connections to MySQL that aren't being closed properly. The ones you have so far are normal entries so try checking again tomorrow without restarting and post the results here.
0
gr8gonzoConsultantCommented:
Also, it is rare to have any permanent memory tables but just in case, let's check to make sure you don't have any. Can you post the results from this query tomorrow (again, don't restart before running it):

SELECT * FROM information_schema.tables WHERE engine='memory';
1
gr8gonzoConsultantCommented:
And one more item - let's dump the engine stats now and again tomorrow (no restarting) and compare the memory allocation between the two runs. The query is:
SHOW ENGINE INNODB STATUS

The main section we're interested in is the "BUFFER POOL AND MEMORY" section but paste the whole contents of the command, please.
0
gr8gonzoConsultantCommented:
And for good measure, let's get the output of SHOW STATUS tomorrow, too.
0
sharingsunshineAuthor Commented:
not sure when I restarted httpd so for good measure I will wait till tomorrow evening

MariaDB [(none)]> MariaDB [(none)]> SHOW ENGINE INNODB STATUS;
+--------+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Type   | Name | Status                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
+--------+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| InnoDB |      |
=====================================
2018-09-02 23:25:10 0x7f18ac44b700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 10 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 5743 srv_active, 0 srv_shutdown, 69805 srv_idle
srv_master_thread log flush and writes: 75536
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 34964
OS WAIT ARRAY INFO: signal count 21061
RW-shared spins 0, rounds 17387, OS waits 8440
RW-excl spins 0, rounds 6471, OS waits 183
RW-sx spins 73, rounds 609, OS waits 1
Spin rounds per wait: 17387.00 RW-shared, 6471.00 RW-excl, 8.34 RW-sx
------------
TRANSACTIONS
------------
Trx id counter 4808937
Purge done for trx's n:o < 4808933 undo n:o < 0 state: running but idle
History list length 0
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 421220197575344, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421220197574440, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
 ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 0
37447 OS file reads, 134516 OS file writes, 63635 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 8.60 writes/s, 2.70 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 219, seg size 221, 629 merges
merged operations:
 insert 621, delete mark 9, delete 3
discarded operations:
 insert 0, delete mark 0, delete 0
Hash table size 34673, node heap has 11 buffer(s)
Hash table size 34673, node heap has 21 buffer(s)
Hash table size 34673, node heap has 163 buffer(s)
Hash table size 34673, node heap has 8 buffer(s)
Hash table size 34673, node heap has 5 buffer(s)
Hash table size 34673, node heap has 96 buffer(s)
Hash table size 34673, node heap has 5 buffer(s)
Hash table size 34673, node heap has 10 buffer(s)
375.36 hash searches/s, 150.48 non-hash searches/s
---
LOG
---
Log sequence number 23996541285
Log flushed up to   23996541285
Pages flushed up to 23996541285
Last checkpoint at  23996541276
0 pending log flushes, 0 pending chkp writes
40679 log i/o's done, 1.70 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 137560064
Dictionary memory allocated 847684
Buffer pool size   8191
Free buffers       1024
Database pages     6848
Old database pages 2507
Modified db pages  0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 14891, not young 534196
0.00 youngs/s, 0.00 non-youngs/s
Pages read 35847, created 654, written 87322
0.00 reads/s, 0.00 creates/s, 6.60 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 6848, unzip_LRU len: 0
I/O sum[160]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Process ID=14808, Main thread ID=139744147875584, state: sleeping
Number of rows inserted 12626, updated 6293, deleted 3807, read 40988600
0.40 inserts/s, 0.10 updates/s, 0.30 deletes/s, 1058.49 reads/s
Number of system rows inserted 0, updated 0, deleted 0, read 0
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
 |
+--------+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

Open in new window

0
sharingsunshineAuthor Commented:
I spoke too soon on the log.  It has only created the headers but no entries.
0
sharingsunshineAuthor Commented:
MariaDB [(none)]> MariaDB [(none)]> SHOW PROCESSLIST;
+-------+-------------+-----------+------+---------+------+--------------------------+------------------+----------+
| Id    | User        | Host      | db   | Command | Time | State                    | Info             | Progress |
+-------+-------------+-----------+------+---------+------+--------------------------+------------------+----------+
|     1 | system user |           | NULL | Daemon  | NULL | InnoDB purge coordinator | NULL             |    0.000 |
|     2 | system user |           | NULL | Daemon  | NULL | InnoDB purge worker      | NULL             |    0.000 |
|     3 | system user |           | NULL | Daemon  | NULL | InnoDB purge worker      | NULL             |    0.000 |
|     4 | system user |           | NULL | Daemon  | NULL | InnoDB purge worker      | NULL             |    0.000 |
|     5 | system user |           | NULL | Daemon  | NULL | InnoDB shutdown handler  | NULL             |    0.000 |
| 25289 | root        | localhost | NULL | Query   |    0 | init                     | SHOW PROCESSLIST |    0.000 |
+-------+-------------+-----------+------+---------+------+--------------------------+------------------+----------+
6 rows in set (0.00 sec)

Open in new window

0
sharingsunshineAuthor Commented:
MariaDB [(none)]> SELECT * FROM information_schema.tables WHERE engine='memory';
+---------------+--------------------+---------------------------------------+-------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+------------------+---------------+
| TABLE_CATALOG | TABLE_SCHEMA       | TABLE_NAME                            | TABLE_TYPE  | ENGINE | VERSION | ROW_FORMAT | TABLE_ROWS | AVG_ROW_LENGTH | DATA_LENGTH | MAX_DATA_LENGTH | INDEX_LENGTH | DATA_FREE | AUTO_INCREMENT | CREATE_TIME         | UPDATE_TIME | CHECK_TIME | TABLE_COLLATION | CHECKSUM | CREATE_OPTIONS   | TABLE_COMMENT |
+---------------+--------------------+---------------------------------------+-------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+------------------+---------------+
| def           | information_schema | APPLICABLE_ROLES                      | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |            979 |           0 |        16691950 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=17137   |               |
| def           | information_schema | CHARACTER_SETS                        | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |            384 |           0 |        16434816 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=43690   |               |
| def           | information_schema | COLLATIONS                            | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |            231 |           0 |        16704765 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=72628   |               |
| def           | information_schema | COLLATION_CHARACTER_SET_APPLICABILITY | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |            195 |           0 |        16357770 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=86037   |               |
| def           | information_schema | COLUMN_PRIVILEGES                     | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           2893 |           0 |        16759149 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=5799    |               |
| def           | information_schema | ENABLED_ROLES                         | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |            387 |           0 |        16563213 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=43351   |               |
| def           | information_schema | ENGINES                               | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |            731 |           0 |        16663145 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=22951   |               |
| def           | information_schema | FILES                                 | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           4022 |           0 |        16767718 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=4171    |               |
| def           | information_schema | GLOBAL_STATUS                         | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           6340 |           0 |        16762960 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=2646    |               |
| def           | information_schema | GLOBAL_VARIABLES                      | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           6340 |           0 |        16762960 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=2646    |               |
| def           | information_schema | KEY_CACHES                            | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |            659 |           0 |        16650294 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=25458   |               |
| def           | information_schema | KEY_COLUMN_USAGE                      | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           4637 |           0 |        16762755 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=3618    |               |
| def           | information_schema | PROFILING                             | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |            308 |           0 |        16562084 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=54471   |               |
| def           | information_schema | REFERENTIAL_CONSTRAINTS               | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           4814 |           0 |        16767162 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=3485    |               |
| def           | information_schema | SCHEMATA                              | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           3464 |           0 |        16738048 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=4843    |               |
| def           | information_schema | SCHEMA_PRIVILEGES                     | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           2507 |           0 |        16741746 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=6692    |               |
| def           | information_schema | SESSION_STATUS                        | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           6340 |           0 |        16762960 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=2646    |               |
| def           | information_schema | SESSION_VARIABLES                     | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           6340 |           0 |        16762960 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=2646    |               |
| def           | information_schema | STATISTICS                            | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           5753 |           0 |        16752736 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=2916    |               |
| def           | information_schema | TABLES                                | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |          14829 |           0 |        16771599 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=1131    |               |
| def           | information_schema | TABLESPACES                           | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           6951 |           0 |        16772763 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=2413    |               |
| def           | information_schema | TABLE_CONSTRAINTS                     | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           2504 |           0 |        16721712 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=6700    |               |
| def           | information_schema | TABLE_PRIVILEGES                      | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           2700 |           0 |        16750800 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=6213    |               |
| def           | information_schema | USER_PRIVILEGES                       | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           2314 |           0 |        16732534 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=7250    |               |
| def           | information_schema | GEOMETRY_COLUMNS                      | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           4244 |           0 |        16759556 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=3953    |               |
| def           | information_schema | SPATIAL_REF_SYS                       | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           7691 |           0 |        16758689 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=2181    |               |
| def           | information_schema | CLIENT_STATISTICS                     | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |            386 |           0 |        16520414 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=43464   |               |
| def           | information_schema | INDEX_STATISTICS                      | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           1743 |           0 |        16765917 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=9625    |               |
| def           | information_schema | INNODB_SYS_DATAFILES                  | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |          12007 |           0 |        16773779 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=1397    |               |
| def           | information_schema | USER_STATISTICS                       | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |            567 |           0 |        16747479 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=29589   |               |
| def           | information_schema | INNODB_SYS_TABLESTATS                 | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           1215 |           0 |        16763355 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=13808   |               |
| def           | information_schema | INNODB_LOCKS                          | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |          31244 |           0 |        16746784 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=536     |               |
| def           | information_schema | TokuDB_fractal_tree_block_map         | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           3883 |           0 |        16755145 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=4320    |               |
| def           | information_schema | INNODB_CMPMEM                         | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |             29 |           0 |        15204352 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=578524  |               |
| def           | information_schema | INNODB_CMP_PER_INDEX                  | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           1755 |           0 |        16728660 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=9559    |               |
| def           | information_schema | INNODB_CMP                            | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |             25 |           0 |        13107200 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=671088  |               |
| def           | information_schema | INNODB_TABLESPACES_ENCRYPTION         | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           2012 |           0 |        16743864 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=8338    |               |
| def           | information_schema | INNODB_CMP_RESET                      | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |             25 |           0 |        13107200 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=671088  |               |
| def           | information_schema | TokuDB_lock_waits                     | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           4645 |           0 |        16763805 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=3611    |               |
| def           | information_schema | TABLE_STATISTICS                      | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           1181 |           0 |        16733589 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=14205   |               |
| def           | information_schema | INNODB_SYS_INDEXES                    | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |            618 |           0 |        16615548 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=27147   |               |
| def           | information_schema | INNODB_BUFFER_PAGE_LRU                | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           6669 |           0 |        16765866 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=2515    |               |
| def           | information_schema | ROCKSDB_CFSTATS                       | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           1171 |           0 |        16705486 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=14327   |               |
| def           | information_schema | INNODB_CMPMEM_RESET                   | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |             29 |           0 |        15204352 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=578524  |               |
| def           | information_schema | TokuDB_background_job_status          | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           6276 |           0 |        16763196 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=2673    |               |
| def           | information_schema | INNODB_SYS_COLUMNS                    | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |            610 |           0 |        16613350 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=27503   |               |
| def           | information_schema | ROCKSDB_COMPACTION_STATS              | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           3672 |           0 |        16740648 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=4568    |               |
| def           | information_schema | INNODB_CMP_PER_INDEX_RESET            | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           1755 |           0 |        16728660 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=9559    |               |
| def           | information_schema | ROCKSDB_CF_OPTIONS                    | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           1744 |           0 |        16700544 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=9619    |               |
| def           | information_schema | INNODB_FT_INDEX_CACHE                 | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           1054 |           0 |        16744898 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=15917   |               |
| def           | information_schema | INNODB_SYS_TABLES                     | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           2091 |           0 |        16736364 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=8023    |               |
| def           | information_schema | INNODB_FT_BEING_DELETED               | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |              9 |           0 |         9437184 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=1864135 |               |
| def           | information_schema | TokuDB_locks                          | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           4637 |           0 |        16762755 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=3618    |               |
| def           | information_schema | INNODB_TRX                            | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           4534 |           0 |        16766732 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=3700    |               |
| def           | information_schema | INNODB_SYS_FOREIGN                    | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           1752 |           0 |        16700064 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=9576    |               |
| def           | information_schema | TokuDB_file_map                       | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           3851 |           0 |        16751850 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=4356    |               |
| def           | information_schema | INNODB_FT_DEFAULT_STOPWORD            | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |             56 |           0 |        14680064 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=299593  |               |
| def           | information_schema | INNODB_FT_CONFIG                      | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           1163 |           0 |        16705332 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=14425   |               |
| def           | information_schema | ROCKSDB_DBSTATS                       | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |            590 |           0 |        16720010 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=28435   |               |
| def           | information_schema | INNODB_SYS_TABLESPACES                | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           2133 |           0 |        16752582 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=7865    |               |
| def           | information_schema | TokuDB_fractal_tree_info              | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           3883 |           0 |        16755145 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=4320    |               |
| def           | information_schema | ROCKSDB_PERF_CONTEXT                  | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           2333 |           0 |        16755606 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=7191    |               |
| def           | information_schema | ROCKSDB_INDEX_FILE_MAP                | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           3040 |           0 |        16732160 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=5518    |               |
| def           | information_schema | INNODB_MUTEXES                        | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |          24017 |           0 |        16763866 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=698     |               |
| def           | information_schema | INNODB_SYS_SEMAPHORE_WAITS            | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |          72196 |           0 |        16749472 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=232     |               |
| def           | information_schema | INNODB_SYS_FIELDS                     | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |            594 |           0 |        16609428 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=28244   |               |
| def           | information_schema | ROCKSDB_LOCKS                         | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           1647 |           0 |        16766460 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=10186   |               |
| def           | information_schema | INNODB_METRICS                        | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           3003 |           0 |        16747731 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=5586    |               |
| def           | information_schema | INNODB_FT_DELETED                     | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |              9 |           0 |         9437184 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=1864135 |               |
| def           | information_schema | ROCKSDB_DDL                           | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           2918 |           0 |        16763910 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=5749    |               |
| def           | information_schema | INNODB_BUFFER_POOL_STATS              | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |            257 |           0 |        16332350 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=65280   |               |
| def           | information_schema | INNODB_LOCK_WAITS                     | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |            599 |           0 |        16749238 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=28008   |               |
| def           | information_schema | ROCKSDB_PERF_CONTEXT_GLOBAL           | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |            590 |           0 |        16720010 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=28435   |               |
| def           | information_schema | INNODB_SYS_VIRTUAL                    | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |             17 |           0 |        11883850 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=986895  |               |
| def           | information_schema | INNODB_BUFFER_PAGE                    | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           6852 |           0 |        16766844 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=2448    |               |
| def           | information_schema | ROCKSDB_GLOBAL_INFO                   | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           4624 |           0 |        16748128 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=3628    |               |
| def           | information_schema | TokuDB_trx                            | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |             25 |           0 |        13107200 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=671088  |               |
| def           | information_schema | INNODB_FT_INDEX_TABLE                 | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           1054 |           0 |        16744898 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=15917   |               |
| def           | information_schema | ROCKSDB_TRX                           | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           3341 |           0 |        16761797 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=5021    |               |
| def           | information_schema | user_variables                        | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           6630 |           0 |        16767270 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=2530    |               |
| def           | information_schema | INNODB_TABLESPACES_SCRUBBING          | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           2020 |           0 |        16743780 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=8305    |               |
| def           | information_schema | INNODB_SYS_FOREIGN_COLS               | SYSTEM VIEW | MEMORY |      11 | Fixed      |       NULL |           1748 |           0 |        16738848 |            0 |         0 |           NULL | 2018-09-04 09:40:03 | NULL        | NULL       | utf8_general_ci |     NULL | max_rows=9597    |               |
+---------------+--------------------+---------------------------------------+-------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+------------------+---------------+
82 rows in set (0.18 sec)

Open in new window

0
sharingsunshineAuthor Commented:
MariaDB [(none)]> SHOW ENGINE INNODB STATUS;
+--------+------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Type   | Name | Status                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
+--------+------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| InnoDB |      |
=====================================
2018-09-04 09:41:24 0x7f18fbc5e700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 31 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 15056 srv_active, 0 srv_shutdown, 183832 srv_idle
srv_master_thread log flush and writes: 198854
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 104891
OS WAIT ARRAY INFO: signal count 65899
RW-shared spins 0, rounds 52971, OS waits 25632
RW-excl spins 0, rounds 153807, OS waits 4940
RW-sx spins 982, rounds 5733, OS waits 10
Spin rounds per wait: 52971.00 RW-shared, 153807.00 RW-excl, 5.84 RW-sx
------------
TRANSACTIONS
------------
Trx id counter 4898278
Purge done for trx's n:o < 4898278 undo n:o < 0 state: running
History list length 3
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 421220197575344, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421220197574440, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
 ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 0
294120 OS file reads, 367433 OS file writes, 173693 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.74 writes/s, 0.52 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 219, seg size 221, 1771 merges
merged operations:
 insert 1691, delete mark 41, delete 16
discarded operations:
 insert 0, delete mark 0, delete 0
Hash table size 34673, node heap has 88 buffer(s)
Hash table size 34673, node heap has 24 buffer(s)
Hash table size 34673, node heap has 156 buffer(s)
Hash table size 34673, node heap has 13 buffer(s)
Hash table size 34673, node heap has 10 buffer(s)
Hash table size 34673, node heap has 7 buffer(s)
Hash table size 34673, node heap has 7 buffer(s)
Hash table size 34673, node heap has 10 buffer(s)
0.26 hash searches/s, 3.10 non-hash searches/s
---
LOG
---
Log sequence number 24092570306
Log flushed up to   24092570306
Pages flushed up to 24092570306
Last checkpoint at  24092570297
0 pending log flushes, 0 pending chkp writes
113309 log i/o's done, 0.32 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 137560064
Dictionary memory allocated 3087606
Buffer pool size   8191
Free buffers       1024
Database pages     6852
Old database pages 2509
Modified db pages  0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 42922, not young 26580812
0.00 youngs/s, 0.00 non-youngs/s
Pages read 292520, created 5806, written 236833
0.00 reads/s, 0.00 creates/s, 0.35 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 6852, unzip_LRU len: 0
I/O sum[63]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Process ID=14808, Main thread ID=139744147875584, state: sleeping
Number of rows inserted 247514, updated 23850, deleted 10648, read 131859165
0.03 inserts/s, 0.03 updates/s, 0.00 deletes/s, 338.21 reads/s
Number of system rows inserted 0, updated 0, deleted 0, read 0
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
 |
+--------+------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

Open in new window

0
sharingsunshineAuthor Commented:
MariaDB [(none)]> SHOW STATUS;
+-----------------------------------------------------------------+---------------------------------------------------+
| Variable_name                                                   | Value                                             |
+-----------------------------------------------------------------+---------------------------------------------------+
| Aborted_clients                                                 | 0                                                 |
| Aborted_connects                                                | 179                                               |
| Access_denied_errors                                            | 0                                                 |
| Acl_column_grants                                               | 0                                                 |
| Acl_database_grants                                             | 1                                                 |
| Acl_function_grants                                             | 0                                                 |
| Acl_procedure_grants                                            | 0                                                 |
| Acl_proxy_users                                                 | 2                                                 |
| Acl_role_grants                                                 | 0                                                 |
| Acl_roles                                                       | 0                                                 |
| Acl_table_grants                                                | 0                                                 |
| Acl_users                                                       | 4                                                 |
| Aria_pagecache_blocks_not_flushed                               | 0                                                 |
| Aria_pagecache_blocks_unused                                    | 15706                                             |
| Aria_pagecache_blocks_used                                      | 333                                               |
| Aria_pagecache_read_requests                                    | 2491175                                           |
| Aria_pagecache_reads                                            | 100658                                            |
| Aria_pagecache_write_requests                                   | 893113                                            |
| Aria_pagecache_writes                                           | 0                                                 |
| Aria_transaction_log_syncs                                      | 0                                                 |
| Binlog_commits                                                  | 0                                                 |
| Binlog_group_commits                                            | 0                                                 |
| Binlog_group_commit_trigger_count                               | 0                                                 |
| Binlog_group_commit_trigger_lock_wait                           | 0                                                 |
| Binlog_group_commit_trigger_timeout                             | 0                                                 |
| Binlog_snapshot_file                                            |                                                   |
| Binlog_snapshot_position                                        | 0                                                 |
| Binlog_bytes_written                                            | 0                                                 |
| Binlog_cache_disk_use                                           | 0                                                 |
| Binlog_cache_use                                                | 0                                                 |
| Binlog_stmt_cache_disk_use                                      | 0                                                 |
| Binlog_stmt_cache_use                                           | 0                                                 |
| Busy_time                                                       | 0.000000                                          |
| Bytes_received                                                  | 379                                               |
| Bytes_sent                                                      | 19891                                             |
| Com_admin_commands                                              | 0                                                 |
| Com_alter_db                                                    | 0                                                 |
| Com_alter_db_upgrade                                            | 0                                                 |
| Com_alter_event                                                 | 0                                                 |
| Com_alter_function                                              | 0                                                 |
| Com_alter_procedure                                             | 0                                                 |
| Com_alter_server                                                | 0                                                 |
| Com_alter_table                                                 | 0                                                 |
| Com_alter_tablespace                                            | 0                                                 |
| Com_alter_user                                                  | 0                                                 |
| Com_analyze                                                     | 0                                                 |
| Com_assign_to_keycache                                          | 0                                                 |
| Com_begin                                                       | 0                                                 |
| Com_binlog                                                      | 0                                                 |
| Com_call_procedure                                              | 0                                                 |
| Com_change_db                                                   | 0                                                 |
| Com_change_master                                               | 0                                                 |
| Com_check                                                       | 0                                                 |
| Com_checksum                                                    | 0                                                 |
| Com_commit                                                      | 0                                                 |
| Com_compound_sql                                                | 0                                                 |
| Com_create_db                                                   | 0                                                 |
| Com_create_event                                                | 0                                                 |
| Com_create_function                                             | 0                                                 |
| Com_create_index                                                | 0                                                 |
| Com_create_procedure                                            | 0                                                 |
| Com_create_role                                                 | 0                                                 |
| Com_create_server                                               | 0                                                 |
| Com_create_table                                                | 0                                                 |
| Com_create_temporary_table                                      | 0                                                 |
| Com_create_trigger                                              | 0                                                 |
| Com_create_udf                                                  | 0                                                 |
| Com_create_user                                                 | 0                                                 |
| Com_create_view                                                 | 0                                                 |
| Com_dealloc_sql                                                 | 0                                                 |
| Com_delete                                                      | 0                                                 |
| Com_delete_multi                                                | 0                                                 |
| Com_do                                                          | 0                                                 |
| Com_drop_db                                                     | 0                                                 |
| Com_drop_event                                                  | 0                                                 |
| Com_drop_function                                               | 0                                                 |
| Com_drop_index                                                  | 0                                                 |
| Com_drop_procedure                                              | 0                                                 |
| Com_drop_role                                                   | 0                                                 |
| Com_drop_server                                                 | 0                                                 |
| Com_drop_table                                                  | 0                                                 |
| Com_drop_temporary_table                                        | 0                                                 |
| Com_drop_trigger                                                | 0                                                 |
| Com_drop_user                                                   | 0                                                 |
| Com_drop_view                                                   | 0                                                 |
| Com_empty_query                                                 | 0                                                 |
| Com_execute_immediate                                           | 0                                                 |
| Com_execute_sql                                                 | 0                                                 |
| Com_flush                                                       | 0                                                 |
| Com_get_diagnostics                                             | 0                                                 |
| Com_grant                                                       | 0                                                 |
| Com_grant_role                                                  | 0                                                 |
| Com_ha_close                                                    | 0                                                 |
| Com_ha_open                                                     | 0                                                 |
| Com_ha_read                                                     | 0                                                 |
| Com_help                                                        | 0                                                 |
| Com_insert                                                      | 0                                                 |
| Com_insert_select                                               | 0                                                 |
| Com_install_plugin                                              | 0                                                 |
| Com_kill                                                        | 0                                                 |
| Com_load                                                        | 0                                                 |
| Com_lock_tables                                                 | 0                                                 |
| Com_multi                                                       | 0                                                 |
| Com_optimize                                                    | 0                                                 |
| Com_preload_keys                                                | 0                                                 |
| Com_prepare_sql                                                 | 0                                                 |
| Com_purge                                                       | 0                                                 |
| Com_purge_before_date                                           | 0                                                 |
| Com_release_savepoint                                           | 0                                                 |
| Com_rename_table                                                | 0                                                 |
| Com_rename_user                                                 | 0                                                 |
| Com_repair                                                      | 0                                                 |
| Com_replace                                                     | 0                                                 |
| Com_replace_select                                              | 0                                                 |
| Com_reset                                                       | 0                                                 |
| Com_resignal                                                    | 0                                                 |
| Com_revoke                                                      | 0                                                 |
| Com_revoke_all                                                  | 0                                                 |
| Com_revoke_role                                                 | 0                                                 |
| Com_rollback                                                    | 0                                                 |
| Com_rollback_to_savepoint                                       | 0                                                 |
| Com_savepoint                                                   | 0                                                 |
| Com_select                                                      | 2                                                 |
| Com_set_option                                                  | 0                                                 |
| Com_show_authors                                                | 0                                                 |
| Com_show_binlog_events                                          | 0                                                 |
| Com_show_binlogs                                                | 0                                                 |
| Com_show_charsets                                               | 0                                                 |
| Com_show_collations                                             | 0                                                 |
| Com_show_contributors                                           | 0                                                 |
| Com_show_create_db                                              | 0                                                 |
| Com_show_create_event                                           | 0                                                 |
| Com_show_create_func                                            | 0                                                 |
| Com_show_create_proc                                            | 0                                                 |
| Com_show_create_table                                           | 0                                                 |
| Com_show_create_trigger                                         | 0                                                 |
| Com_show_create_user                                            | 0                                                 |
| Com_show_databases                                              | 0                                                 |
| Com_show_engine_logs                                            | 0                                                 |
| Com_show_engine_mutex                                           | 0                                                 |
| Com_show_engine_status                                          | 1                                                 |
| Com_show_errors                                                 | 0                                                 |
| Com_show_events                                                 | 0                                                 |
| Com_show_explain                                                | 0                                                 |
| Com_show_fields                                                 | 0                                                 |
| Com_show_function_status                                        | 0                                                 |
| Com_show_generic                                                | 0                                                 |
| Com_show_grants                                                 | 0                                                 |
| Com_show_keys                                                   | 0                                                 |
| Com_show_master_status                                          | 0                                                 |
| Com_show_open_tables                                            | 0                                                 |
| Com_show_plugins                                                | 0                                                 |
| Com_show_privileges                                             | 0                                                 |
| Com_show_procedure_status                                       | 0                                                 |
| Com_show_processlist                                            | 2                                                 |
| Com_show_profile                                                | 0                                                 |
| Com_show_profiles                                               | 0                                                 |
| Com_show_relaylog_events                                        | 0                                                 |
| Com_show_slave_hosts                                            | 0                                                 |
| Com_show_slave_status                                           | 0                                                 |
| Com_show_status                                                 | 1                                                 |
| Com_show_storage_engines                                        | 0                                                 |
| Com_show_table_status                                           | 0                                                 |
| Com_show_tables                                                 | 0                                                 |
| Com_show_triggers                                               | 0                                                 |
| Com_show_variables                                              | 0                                                 |
| Com_show_warnings                                               | 0                                                 |
| Com_shutdown                                                    | 0                                                 |
| Com_signal                                                      | 0                                                 |
| Com_start_all_slaves                                            | 0                                                 |
| Com_start_slave                                                 | 0                                                 |
| Com_stmt_close                                                  | 0                                                 |
| Com_stmt_execute                                                | 0                                                 |
| Com_stmt_fetch                                                  | 0                                                 |
| Com_stmt_prepare                                                | 0                                                 |
| Com_stmt_reprepare                                              | 0                                                 |
| Com_stmt_reset                                                  | 0                                                 |
| Com_stmt_send_long_data                                         | 0                                                 |
| Com_stop_all_slaves                                             | 0                                                 |
| Com_stop_slave                                                  | 0                                                 |
| Com_truncate                                                    | 0                                                 |
| Com_uninstall_plugin                                            | 0                                                 |
| Com_unlock_tables                                               | 0                                                 |
| Com_update                                                      | 0                                                 |
| Com_update_multi                                                | 0                                                 |
| Com_xa_commit                                                   | 0                                                 |
| Com_xa_end                                                      | 0                                                 |
| Com_xa_prepare                                                  | 0                                                 |
| Com_xa_recover                                                  | 0                                                 |
| Com_xa_rollback                                                 | 0                                                 |
| Com_xa_start                                                    | 0                                                 |
| Compression                                                     | OFF                                               |
| Connection_errors_accept                                        | 0                                                 |
| Connection_errors_internal                                      | 0                                                 |
| Connection_errors_max_connections                               | 0                                                 |
| Connection_errors_peer_address                                  | 0                                                 |
| Connection_errors_select                                        | 0                                                 |
| Connection_errors_tcpwrap                                       | 0                                                 |
| Connections                                                     | 25324                                             |
| Cpu_time                                                        | 0.000000                                          |
| Created_tmp_disk_tables                                         | 12                                                |
| Created_tmp_files                                               | 698                                               |
| Created_tmp_tables                                              | 95                                                |
| Delayed_errors                                                  | 0                                                 |
| Delayed_insert_threads                                          | 0                                                 |
| Delayed_writes                                                  | 0                                                 |
| Delete_scan                                                     | 0                                                 |
| Empty_queries                                                   | 0                                                 |
| Executed_events                                                 | 0                                                 |
| Executed_triggers                                               | 0                                                 |
| Feature_check_constraint                                        | 0                                                 |
| Feature_delay_key_write                                         | 0                                                 |
| Feature_dynamic_columns                                         | 0                                                 |
| Feature_fulltext                                                | 0                                                 |
| Feature_gis                                                     | 0                                                 |
| Feature_locale                                                  | 0                                                 |
| Feature_subquery                                                | 0                                                 |
| Feature_timezone                                                | 0                                                 |
| Feature_trigger                                                 | 0                                                 |
| Feature_window_functions                                        | 0                                                 |
| Feature_xml                                                     | 0                                                 |
| Flush_commands                                                  | 1                                                 |
| Handler_commit                                                  | 0                                                 |
| Handler_delete                                                  | 0                                                 |
| Handler_discover                                                | 52                                                |
| Handler_external_lock                                           | 0                                                 |
| Handler_icp_attempts                                            | 0                                                 |
| Handler_icp_match                                               | 0                                                 |
| Handler_mrr_init                                                | 0                                                 |
| Handler_mrr_key_refills                                         | 0                                                 |
| Handler_mrr_rowid_refills                                       | 0                                                 |
| Handler_prepare                                                 | 0                                                 |
| Handler_read_first                                              | 0                                                 |
| Handler_read_key                                                | 0                                                 |
| Handler_read_last                                               | 0                                                 |
| Handler_read_next                                               | 0                                                 |
| Handler_read_prev                                               | 0                                                 |
| Handler_read_retry                                              | 0                                                 |
| Handler_read_rnd                                                | 0                                                 |
| Handler_read_rnd_deleted                                        | 0                                                 |
| Handler_read_rnd_next                                           | 2371                                              |
| Handler_rollback                                                | 0                                                 |
| Handler_savepoint                                               | 0                                                 |
| Handler_savepoint_rollback                                      | 0                                                 |
| Handler_tmp_update                                              | 0                                                 |
| Handler_tmp_write                                               | 2370                                              |
| Handler_update                                                  | 0                                                 |
| Handler_write                                                   | 0                                                 |
| Innodb_buffer_pool_dump_status                                  | Dumping of buffer pool not started                |
| Innodb_buffer_pool_load_status                                  | Buffer pool(s) load completed at 180902  2:25:41  |
| Innodb_buffer_pool_resize_status                                |                                                   |
| Innodb_buffer_pool_pages_data                                   | 6924                                              |
| Innodb_buffer_pool_bytes_data                                   | 113442816                                         |
| Innodb_buffer_pool_pages_dirty                                  | 0                                                 |
| Innodb_buffer_pool_bytes_dirty                                  | 0                                                 |
| Innodb_buffer_pool_pages_flushed                                | 10042                                             |
| Innodb_buffer_pool_pages_free                                   | 1024                                              |
| Innodb_buffer_pool_pages_misc                                   | 243                                               |
| Innodb_buffer_pool_pages_total                                  | 8191                                              |
| Innodb_buffer_pool_read_ahead_rnd                               | 0                                                 |
| Innodb_buffer_pool_read_ahead                                   | 6260                                              |
| Innodb_buffer_pool_read_ahead_evicted                           | 98                                                |
| Innodb_buffer_pool_read_requests                                | 151144341                                         |
| Innodb_buffer_pool_reads                                        | 285854                                            |
| Innodb_buffer_pool_wait_free                                    | 0                                                 |
| Innodb_buffer_pool_write_requests                               | 1638842                                           |
| Innodb_data_fsyncs                                              | 174084                                            |
| Innodb_data_pending_fsyncs                                      | 0                                                 |
| Innodb_data_pending_reads                                       | 0                                                 |
| Innodb_data_pending_writes                                      | 0                                                 |
| Innodb_data_read                                                | 4812311040                                        |
| Innodb_data_reads                                               | 295315                                            |
| Innodb_data_writes                                              | 368513                                            |
| Innodb_data_written                                             | 7981386240                                        |
| Innodb_dblwr_pages_written                                      | 237448                                            |
| Innodb_dblwr_writes                                             | 16733                                             |
| Innodb_log_waits                                                | 0                                                 |
| Innodb_log_write_requests                                       | 234241                                            |
| Innodb_log_writes                                               | 91771                                             |
| Innodb_os_log_fsyncs                                            | 113605                                            |
| Innodb_os_log_pending_fsyncs                                    | 0                                                 |
| Innodb_os_log_pending_writes                                    | 0                                                 |
| Innodb_os_log_written                                           | 187362304                                         |
| Innodb_page_size                                                | 16384                                             |
| Innodb_pages_created                                            | 5819                                              |
| Innodb_pages_read                                               | 293715                                            |
| Innodb_pages0_read                                              | 795                                               |
| Innodb_pages_written                                            | 237579                                            |
| Innodb_row_lock_current_waits                                   | 0                                                 |
| Innodb_row_lock_time                                            | 0                                                 |
| Innodb_row_lock_time_avg                                        | 0                                                 |
| Innodb_row_lock_time_max                                        | 0                                                 |
| Innodb_row_lock_waits                                           | 1                                                 |
| Innodb_rows_deleted                                             | 10691                                             |
| Innodb_rows_inserted                                            | 247704                                            |
| Innodb_rows_read                                                | 131922731                                         |
| Innodb_rows_updated                                             | 23893                                             |
| Innodb_system_rows_deleted                                      | 0                                                 |
| Innodb_system_rows_inserted                                     | 0                                                 |
| Innodb_system_rows_read                                         | 0                                                 |
| Innodb_system_rows_updated                                      | 0                                                 |
| Innodb_num_open_files                                           | 431                                               |
| Innodb_truncated_status_writes                                  | 0                                                 |
| Innodb_available_undo_logs                                      | 128                                               |
| Innodb_page_compression_saved                                   | 0                                                 |
| Innodb_num_index_pages_written                                  | 0                                                 |
| Innodb_num_non_index_pages_written                              | 0                                                 |
| Innodb_num_pages_page_compressed                                | 0                                                 |
| Innodb_num_page_compressed_trim_op                              | 0                                                 |
| Innodb_num_pages_page_decompressed                              | 0                                                 |
| Innodb_num_pages_page_compression_error                         | 0                                                 |
| Innodb_num_pages_encrypted                                      | 0                                                 |
| Innodb_num_pages_decrypted                                      | 0                                                 |
| Innodb_have_lz4                                                 | OFF                                               |
| Innodb_have_lzo                                                 | OFF                                               |
| Innodb_have_lzma                                                | OFF                                               |
| Innodb_have_bzip2                                               | OFF                                               |
| Innodb_have_snappy                                              | OFF                                               |
| Innodb_have_punch_hole                                          | ON                                                |
| Innodb_defragment_compression_failures                          | 0                                                 |
| Innodb_defragment_failures                                      | 0                                                 |
| Innodb_defragment_count                                         | 0                                                 |
| Innodb_onlineddl_rowlog_rows                                    | 0                                                 |
| Innodb_onlineddl_rowlog_pct_used                                | 0                                                 |
| Innodb_onlineddl_pct_progress                                   | 0                                                 |
| Innodb_secondary_index_triggered_cluster_reads                  | 20809012                                          |
| Innodb_secondary_index_triggered_cluster_reads_avoided          | 0                                                 |
| Innodb_encryption_rotation_pages_read_from_cache                | 0                                                 |
| Innodb_encryption_rotation_pages_read_from_disk                 | 0                                                 |
| Innodb_encryption_rotation_pages_modified                       | 0                                                 |
| Innodb_encryption_rotation_pages_flushed                        | 0                                                 |
| Innodb_encryption_rotation_estimated_iops                       | 0                                                 |
| Innodb_encryption_key_rotation_list_length                      | 0                                                 |
| Innodb_encryption_n_merge_blocks_encrypted                      | 0                                                 |
| Innodb_encryption_n_merge_blocks_decrypted                      | 0                                                 |
| Innodb_encryption_n_rowlog_blocks_encrypted                     | 0                                                 |
| Innodb_encryption_n_rowlog_blocks_decrypted                     | 0                                                 |
| Innodb_scrub_background_page_reorganizations                    | 0                                                 |
| Innodb_scrub_background_page_splits                             | 0                                                 |
| Innodb_scrub_background_page_split_failures_underflow           | 0                                                 |
| Innodb_scrub_background_page_split_failures_out_of_filespace    | 0                                                 |
| Innodb_scrub_background_page_split_failures_missing_index       | 0                                                 |
| Innodb_scrub_background_page_split_failures_unknown             | 0                                                 |
| Innodb_scrub_log                                                | 0                                                 |
| Innodb_encryption_num_key_requests                              | 0                                                 |
| Key_blocks_not_flushed                                          | 0                                                 |
| Key_blocks_unused                                               | 107002                                            |
| Key_blocks_used                                                 | 21076                                             |
| Key_blocks_warm                                                 | 0                                                 |
| Key_read_requests                                               | 143091736                                         |
| Key_reads                                                       | 15150                                             |
| Key_write_requests                                              | 1029440                                           |
| Key_writes                                                      | 403149                                            |
| Last_query_cost                                                 | 10.499000                                         |
| Master_gtid_wait_count                                          | 0                                                 |
| Master_gtid_wait_time                                           | 0                                                 |
| Master_gtid_wait_timeouts                                       | 0                                                 |
| Max_statement_time_exceeded                                     | 0                                                 |
| Max_used_connections                                            | 17                                                |
| Memory_used                                                     | 68008                                             |
| Not_flushed_delayed_rows                                        | 0                                                 |
| Open_files                                                      | 226                                               |
| Open_streams                                                    | 0                                                 |
| Open_table_definitions                                          | 426                                               |
| Open_tables                                                     | 431                                               |
| Opened_files                                                    | 477200                                            |
| Opened_plugin_libraries                                         | 3                                                 |
| Opened_table_definitions                                        | 951                                               |
| Opened_tables                                                   | 951                                               |
| Opened_views                                                    | 0                                                 |
| Performance_schema_accounts_lost                                | 0                                                 |
| Performance_schema_cond_classes_lost                            | 0                                                 |
| Performance_schema_cond_instances_lost                          | 0                                                 |
| Performance_schema_digest_lost                                  | 0                                                 |
| Performance_schema_file_classes_lost                            | 0                                                 |
| Performance_schema_file_handles_lost                            | 0                                                 |
| Performance_schema_file_instances_lost                          | 0                                                 |
| Performance_schema_hosts_lost                                   | 0                                                 |
| Performance_schema_locker_lost                                  | 0                                                 |
| Performance_schema_mutex_classes_lost                           | 0                                                 |
| Performance_schema_mutex_instances_lost                         | 0                                                 |
| Performance_schema_rwlock_classes_lost                          | 0                                                 |
| Performance_schema_rwlock_instances_lost                        | 0                                                 |
| Performance_schema_session_connect_attrs_lost                   | 0                                                 |
| Performance_schema_socket_classes_lost                          | 0                                                 |
| Performance_schema_socket_instances_lost                        | 0                                                 |
| Performance_schema_stage_classes_lost                           | 0                                                 |
| Performance_schema_statement_classes_lost                       | 0                                                 |
| Performance_schema_table_handles_lost                           | 0                                                 |
| Performance_schema_table_instances_lost                         | 0                                                 |
| Performance_schema_thread_classes_lost                          | 0                                                 |
| Performance_schema_thread_instances_lost                        | 0                                                 |
| Performance_schema_users_lost                                   | 0                                                 |
| Prepared_stmt_count                                             | 0                                                 |
| Qcache_free_blocks                                              | 1                                                 |
| Qcache_free_memory                                              | 1031336                                           |
| Qcache_hits                                                     | 0                                                 |
| Qcache_inserts                                                  | 0                                                 |
| Qcache_lowmem_prunes                                            | 0                                                 |
| Qcache_not_cached                                               | 0                                                 |
| Qcache_queries_in_cache                                         | 0                                                 |
| Qcache_total_blocks                                             | 1                                                 |
| Queries                                                         | 17994982                                          |
| Questions                                                       | 6                                                 |
| Rocksdb_rows_deleted                                            | 0                                                 |
| Rocksdb_rows_inserted                                           | 0                                                 |
| Rocksdb_rows_read                                               | 0                                                 |
| Rocksdb_rows_updated                                            | 0                                                 |
| Rocksdb_rows_deleted_blind                                      | 0                                                 |
| Rocksdb_rows_expired                                            | 0                                                 |
| Rocksdb_system_rows_deleted                                     | 0                                                 |
| Rocksdb_system_rows_inserted                                    | 0                                                 |
| Rocksdb_system_rows_read                                        | 0                                                 |
| Rocksdb_system_rows_updated                                     | 0                                                 |
| Rocksdb_memtable_total                                          | 1512                                              |
| Rocksdb_memtable_unflushed                                      | 1512                                              |
| Rocksdb_queries_point                                           | 0                                                 |
| Rocksdb_queries_range                                           | 0                                                 |
| Rocksdb_covered_secondary_key_lookups                           | 0                                                 |
| Rocksdb_block_cache_add                                         | 8                                                 |
| Rocksdb_block_cache_data_hit                                    | 0                                                 |
| Rocksdb_block_cache_data_miss                                   | 4                                                 |
| Rocksdb_block_cache_filter_hit                                  | 0                                                 |
| Rocksdb_block_cache_filter_miss                                 | 0                                                 |
| Rocksdb_block_cache_hit                                         | 0                                                 |
| Rocksdb_block_cache_index_hit                                   | 0                                                 |
| Rocksdb_block_cache_index_miss                                  | 4                                                 |
| Rocksdb_block_cache_miss                                        | 8                                                 |
| Rocksdb_block_cachecompressed_hit                               | 0                                                 |
| Rocksdb_block_cachecompressed_miss                              | 0                                                 |
| Rocksdb_bloom_filter_prefix_checked                             | 0                                                 |
| Rocksdb_bloom_filter_prefix_useful                              | 0                                                 |
| Rocksdb_bloom_filter_useful                                     | 0                                                 |
| Rocksdb_bytes_read                                              | 0                                                 |
| Rocksdb_bytes_written                                           | 60                                                |
| Rocksdb_compact_read_bytes                                      | 0                                                 |
| Rocksdb_compact_write_bytes                                     | 0                                                 |
| Rocksdb_compaction_key_drop_new                                 | 0                                                 |
| Rocksdb_compaction_key_drop_obsolete                            | 0                                                 |
| Rocksdb_compaction_key_drop_user                                | 0                                                 |
| Rocksdb_flush_write_bytes                                       | 0                                                 |
| Rocksdb_getupdatessince_calls                                   | 0                                                 |
| Rocksdb_memtable_hit                                            | 0                                                 |
| Rocksdb_memtable_miss                                           | 2                                                 |
| Rocksdb_no_file_closes                                          | 0                                                 |
| Rocksdb_no_file_errors                                          | 0                                                 |
| Rocksdb_no_file_opens                                           | 4                                                 |
| Rocksdb_num_iterators                                           | 0                                                 |
| Rocksdb_number_block_not_compressed                             | 0                                                 |
| Rocksdb_number_deletes_filtered                                 | 0                                                 |
| Rocksdb_number_keys_read                                        | 2                                                 |
| Rocksdb_number_keys_updated                                     | 0                                                 |
| Rocksdb_number_keys_written                                     | 2                                                 |
| Rocksdb_number_merge_failures                                   | 0                                                 |
| Rocksdb_number_multiget_bytes_read                              | 0                                                 |
| Rocksdb_number_multiget_get                                     | 0                                                 |
| Rocksdb_number_multiget_keys_read                               | 0                                                 |
| Rocksdb_number_reseeks_iteration                                | 0                                                 |
| Rocksdb_number_sst_entry_delete                                 | 0                                                 |
| Rocksdb_number_sst_entry_merge                                  | 0                                                 |
| Rocksdb_number_sst_entry_other                                  | 0                                                 |
| Rocksdb_number_sst_entry_put                                    | 0                                                 |
| Rocksdb_number_sst_entry_singledelete                           | 0                                                 |
| Rocksdb_number_stat_computes                                    | 0                                                 |
| Rocksdb_number_superversion_acquires                            | 2                                                 |
| Rocksdb_number_superversion_cleanups                            | 0                                                 |
| Rocksdb_number_superversion_releases                            | 0                                                 |
| Rocksdb_rate_limit_delay_millis                                 | 0                                                 |
| Rocksdb_snapshot_conflict_errors                                | 0                                                 |
| Rocksdb_stall_l0_file_count_limit_slowdowns                     | 0                                                 |
| Rocksdb_stall_locked_l0_file_count_limit_slowdowns              | 0                                                 |
| Rocksdb_stall_l0_file_count_limit_stops                         | 0                                                 |
| Rocksdb_stall_locked_l0_file_count_limit_stops                  | 0                                                 |
| Rocksdb_stall_pending_compaction_limit_stops                    | 0                                                 |
| Rocksdb_stall_pending_compaction_limit_slowdowns                | 0                                                 |
| Rocksdb_stall_memtable_limit_stops                              | 0                                                 |
| Rocksdb_stall_memtable_limit_slowdowns                          | 0                                                 |
| Rocksdb_stall_total_stops                                       | 0                                                 |
| Rocksdb_stall_total_slowdowns                                   | 0                                                 |
| Rocksdb_stall_micros                                            | 0                                                 |
| Rocksdb_wal_bytes                                               | 60                                                |
| Rocksdb_wal_group_syncs                                         | 0                                                 |
| Rocksdb_wal_synced                                              | 2                                                 |
| Rocksdb_write_other                                             | 0                                                 |
| Rocksdb_write_self                                              | 2                                                 |
| Rocksdb_write_timedout                                          | 0                                                 |
| Rocksdb_write_wal                                               | 4                                                 |
| Rows_read                                                       | 0                                                 |
| Rows_sent                                                       | 83                                                |
| Rows_tmp_read                                                   | 2369                                              |
| Rpl_status                                                      | AUTH_MASTER                                       |
| Select_full_join                                                | 0                                                 |
| Select_full_range_join                                          | 0                                                 |
| Select_range                                                    | 0                                                 |
| Select_range_check                                              | 0                                                 |
| Select_scan                                                     | 1                                                 |
| Slave_connections                                               | 0                                                 |
| Slave_heartbeat_period                                          | 0.000                                             |
| Slave_open_temp_tables                                          | 0                                                 |
| Slave_received_heartbeats                                       | 0                                                 |
| Slave_retried_transactions                                      | 0                                                 |
| Slave_running                                                   | OFF                                               |
| Slave_skipped_errors                                            | 0                                                 |
| Slaves_connected                                                | 0                                                 |
| Slaves_running                                                  | 0                                                 |
| Slow_launch_threads                                             | 0                                                 |
| Slow_queries                                                    | 0                                                 |
| Sort_merge_passes                                               | 0                                                 |
| Sort_priority_queue_sorts                                       | 0                                                 |
| Sort_range                                                      | 0                                                 |
| Sort_rows                                                       | 0                                                 |
| Sort_scan                                                       | 0                                                 |
| Ssl_accept_renegotiates                                         | 0                                                 |
| Ssl_accepts                                                     | 0                                                 |
| Ssl_callback_cache_hits                                         | 0                                                 |
| Ssl_cipher                                                      |                                                   |
| Ssl_cipher_list                                                 |                                                   |
| Ssl_client_connects                                             | 0                                                 |
| Ssl_connect_renegotiates                                        | 0                                                 |
| Ssl_ctx_verify_depth                                            | 0                                                 |
| Ssl_ctx_verify_mode                                             | 0                                                 |
| Ssl_default_timeout                                             | 0                                                 |
| Ssl_finished_accepts                                            | 0                                                 |
| Ssl_finished_connects                                           | 0                                                 |
| Ssl_server_not_after                                            |                                                   |
| Ssl_server_not_before                                           |                                                   |
| Ssl_session_cache_hits                                          | 0                                                 |
| Ssl_session_cache_misses                                        | 0                                                 |
| Ssl_session_cache_mode                                          | NONE                                              |
| Ssl_session_cache_overflows                                     | 0                                                 |
| Ssl_session_cache_size                                          | 0                                                 |
| Ssl_session_cache_timeouts                                      | 0                                                 |
| Ssl_sessions_reused                                             | 0                                                 |
| Ssl_used_session_cache_entries                                  | 0                                                 |
| Ssl_verify_depth                                                | 0                                                 |
| Ssl_verify_mode                                                 | 0                                                 |
| Ssl_version                                                     |                                                   |
| Subquery_cache_hit                                              | 648                                               |
| Subquery_cache_miss                                             | 230788                                            |
| Syncs                                                           | 300                                               |
| Table_locks_immediate                                           | 16849433                                          |
| Table_locks_waited                                              | 55                                                |
| Tc_log_max_pages_used                                           | 0                                                 |
| Tc_log_page_size                                                | 4096                                              |
| Tc_log_page_waits                                               | 0                                                 |
| Threadpool_idle_threads                                         | 0                                                 |
| Threadpool_threads                                              | 0                                                 |
| Threads_cached                                                  | 5                                                 |
| Threads_connected                                               | 1                                                 |
| Threads_created                                                 | 30                                                |
| Threads_running                                                 | 1                                                 |
| Tokudb_db_opens                                                 | 0                                                 |
| Tokudb_db_closes                                                | 0                                                 |
| Tokudb_db_open_current                                          | 0                                                 |
| Tokudb_db_open_max                                              | 0                                                 |
| Tokudb_leaf_entry_max_committed_xr                              | 0                                                 |
| Tokudb_leaf_entry_max_provisional_xr                            | 0                                                 |
| Tokudb_leaf_entry_expanded                                      | 0                                                 |
| Tokudb_leaf_entry_max_memsize                                   | 0                                                 |
| Tokudb_leaf_entry_apply_gc_bytes_in                             | 0                                                 |
| Tokudb_leaf_entry_apply_gc_bytes_out                            | 0                                                 |
| Tokudb_leaf_entry_normal_gc_bytes_in                            | 0                                                 |
| Tokudb_leaf_entry_normal_gc_bytes_out                           | 0                                                 |
| Tokudb_checkpoint_period                                        | 60                                                |
| Tokudb_checkpoint_footprint                                     | 0                                                 |
| Tokudb_checkpoint_last_began                                    | Tue Sep  4 09:42:41 2018                          |
| Tokudb_checkpoint_last_complete_began                           | Tue Sep  4 09:42:41 2018                          |
| Tokudb_checkpoint_last_complete_ended                           | Tue Sep  4 09:42:41 2018                          |
| Tokudb_checkpoint_duration                                      | 0                                                 |
| Tokudb_checkpoint_duration_last                                 | 0                                                 |
| Tokudb_checkpoint_last_lsn                                      | 396147                                            |
| Tokudb_checkpoint_taken                                         | 3318                                              |
| Tokudb_checkpoint_failed                                        | 0                                                 |
| Tokudb_checkpoint_waiters_now                                   | 0                                                 |
| Tokudb_checkpoint_waiters_max                                   | 0                                                 |
| Tokudb_checkpoint_client_wait_on_mo                             | 0                                                 |
| Tokudb_checkpoint_client_wait_on_cs                             | 0                                                 |
| Tokudb_checkpoint_begin_time                                    | 50940                                             |
| Tokudb_checkpoint_long_begin_time                               | 0                                                 |
| Tokudb_checkpoint_long_begin_count                              | 0                                                 |
| Tokudb_checkpoint_end_time                                      | 4027459                                           |
| Tokudb_checkpoint_long_end_time                                 | 0                                                 |
| Tokudb_checkpoint_long_end_count                                | 0                                                 |
| Tokudb_cachetable_miss                                          | 3                                                 |
| Tokudb_cachetable_miss_time                                     | 48                                                |
| Tokudb_cachetable_prefetches                                    | 0                                                 |
| Tokudb_cachetable_size_current                                  | 1301                                              |
| Tokudb_cachetable_size_limit                                    | 4398454579                                        |
| Tokudb_cachetable_size_writing                                  | 0                                                 |
| Tokudb_cachetable_size_nonleaf                                  | 0                                                 |
| Tokudb_cachetable_size_leaf                                     | 1301                                              |
| Tokudb_cachetable_size_rollback                                 | 0                                                 |
| Tokudb_cachetable_size_cachepressure                            | 0                                                 |
| Tokudb_cachetable_size_cloned                                   | 0                                                 |
| Tokudb_cachetable_evictions                                     | 0                                                 |
| Tokudb_cachetable_cleaner_executions                            | 199023                                            |
| Tokudb_cachetable_cleaner_period                                | 1                                                 |
| Tokudb_cachetable_cleaner_iterations                            | 5                                                 |
| Tokudb_cachetable_wait_pressure_count                           | 0                                                 |
| Tokudb_cachetable_wait_pressure_time                            | 0                                                 |
| Tokudb_cachetable_long_wait_pressure_count                      | 0                                                 |
| Tokudb_cachetable_long_wait_pressure_time                       | 0                                                 |
| Tokudb_cachetable_pool_client_num_threads                       | 4                                                 |
| Tokudb_cachetable_pool_client_num_threads_active                | 0                                                 |
| Tokudb_cachetable_pool_client_queue_size                        | 0                                                 |
| Tokudb_cachetable_pool_client_max_queue_size                    | 0                                                 |
| Tokudb_cachetable_pool_client_total_items_processed             | 0                                                 |
| Tokudb_cachetable_pool_client_total_execution_time              | 0                                                 |
| Tokudb_cachetable_pool_cachetable_num_threads                   | 8                                                 |
| Tokudb_cachetable_pool_cachetable_num_threads_active            | 0                                                 |
| Tokudb_cachetable_pool_cachetable_queue_size                    | 0                                                 |
| Tokudb_cachetable_pool_cachetable_max_queue_size                | 0                                                 |
| Tokudb_cachetable_pool_cachetable_total_items_processed         | 0                                                 |
| Tokudb_cachetable_pool_cachetable_total_execution_time          | 0                                                 |
| Tokudb_cachetable_pool_checkpoint_num_threads                   | 1                                                 |
| Tokudb_cachetable_pool_checkpoint_num_threads_active            | 0                                                 |
| Tokudb_cachetable_pool_checkpoint_queue_size                    | 0                                                 |
| Tokudb_cachetable_pool_checkpoint_max_queue_size                | 0                                                 |
| Tokudb_cachetable_pool_checkpoint_total_items_processed         | 0                                                 |
| Tokudb_cachetable_pool_checkpoint_total_execution_time          | 0                                                 |
| Tokudb_locktree_memory_size                                     | 0                                                 |
| Tokudb_locktree_memory_size_limit                               | 499824384                                         |
| Tokudb_locktree_escalation_num                                  | 0                                                 |
| Tokudb_locktree_escalation_seconds                              | 0.000000                                          |
| Tokudb_locktree_latest_post_escalation_memory_size              | 0                                                 |
| Tokudb_locktree_open_current                                    | 2                                                 |
| Tokudb_locktree_pending_lock_requests                           | 0                                                 |
| Tokudb_locktree_sto_eligible_num                                | 0                                                 |
| Tokudb_locktree_sto_ended_num                                   | 0                                                 |
| Tokudb_locktree_sto_ended_seconds                               | 0.000000                                          |
| Tokudb_locktree_wait_count                                      | 0                                                 |
| Tokudb_locktree_wait_time                                       | 0                                                 |
| Tokudb_locktree_long_wait_count                                 | 0                                                 |
| Tokudb_locktree_long_wait_time                                  | 0                                                 |
| Tokudb_locktree_timeout_count                                   | 0                                                 |
| Tokudb_locktree_wait_escalation_count                           | 0                                                 |
| Tokudb_locktree_wait_escalation_time                            | 0                                                 |
| Tokudb_locktree_long_wait_escalation_count                      | 0                                                 |
| Tokudb_locktree_long_wait_escalation_time                       | 0                                                 |
| Tokudb_dictionary_updates                                       | 0                                                 |
| Tokudb_dictionary_broadcast_updates                             | 0                                                 |
| Tokudb_descriptor_set                                           | 0                                                 |
| Tokudb_messages_ignored_by_leaf_due_to_msn                      | 0                                                 |
| Tokudb_total_search_retries                                     | 0                                                 |
| Tokudb_search_tries_gt_height                                   | 0                                                 |
| Tokudb_search_tries_gt_heightplus3                              | 0                                                 |
| Tokudb_leaf_nodes_flushed_not_checkpoint                        | 0                                                 |
| Tokudb_leaf_nodes_flushed_not_checkpoint_bytes                  | 0                                                 |
| Tokudb_leaf_nodes_flushed_not_checkpoint_uncompressed_bytes     | 0                                                 |
| Tokudb_leaf_nodes_flushed_not_checkpoint_seconds                | 0.000000                                          |
| Tokudb_nonleaf_nodes_flushed_to_disk_not_checkpoint             | 0                                                 |
| Tokudb_nonleaf_nodes_flushed_to_disk_not_checkpoint_bytes       | 0                                                 |
| Tokudb_nonleaf_nodes_flushed_to_disk_not_checkpoint_uncompresse | 0                                                 |
| Tokudb_nonleaf_nodes_flushed_to_disk_not_checkpoint_seconds     | 0.000000                                          |
| Tokudb_leaf_nodes_flushed_checkpoint                            | 0                                                 |
| Tokudb_leaf_nodes_flushed_checkpoint_bytes                      | 0                                                 |
| Tokudb_leaf_nodes_flushed_checkpoint_uncompressed_bytes         | 0                                                 |
| Tokudb_leaf_nodes_flushed_checkpoint_seconds                    | 0.000000                                          |
| Tokudb_nonleaf_nodes_flushed_to_disk_checkpoint                 | 0                                                 |
| Tokudb_nonleaf_nodes_flushed_to_disk_checkpoint_bytes           | 0                                                 |
| Tokudb_nonleaf_nodes_flushed_to_disk_checkpoint_uncompressed_by | 0                                                 |
| Tokudb_nonleaf_nodes_flushed_to_disk_checkpoint_seconds         | 0.000000                                          |
| Tokudb_leaf_node_compression_ratio                              | 0.000000                                          |
| Tokudb_nonleaf_node_compression_ratio                           | 0.000000                                          |
| Tokudb_overall_node_compression_ratio                           | 0.000000                                          |
| Tokudb_nonleaf_node_partial_evictions                           | 0                                                 |
| Tokudb_nonleaf_node_partial_evictions_bytes                     | 0                                                 |
| Tokudb_leaf_node_partial_evictions                              | 0                                                 |
| Tokudb_leaf_node_partial_evictions_bytes                        | 0                                                 |
| Tokudb_leaf_node_full_evictions                                 | 0                                                 |
| Tokudb_leaf_node_full_evictions_bytes                           | 0                                                 |
| Tokudb_nonleaf_node_full_evictions                              | 0                                                 |
| Tokudb_nonleaf_node_full_evictions_bytes                        | 0                                                 |
| Tokudb_leaf_nodes_created                                       | 0                                                 |
| Tokudb_nonleaf_nodes_created                                    | 0                                                 |
| Tokudb_leaf_nodes_destroyed                                     | 0                                                 |
| Tokudb_nonleaf_nodes_destroyed                                  | 0                                                 |
| Tokudb_messages_injected_at_root_bytes                          | 0                                                 |
| Tokudb_messages_flushed_from_h1_to_leaves_bytes                 | 0                                                 |
| Tokudb_messages_in_trees_estimate_bytes                         | 0                                                 |
| Tokudb_messages_injected_at_root                                | 0                                                 |
| Tokudb_broadcase_messages_injected_at_root                      | 0                                                 |
| Tokudb_basements_decompressed_target_query                      | 0                                                 |
| Tokudb_basements_decompressed_prelocked_range                   | 0                                                 |
| Tokudb_basements_decompressed_prefetch                          | 0                                                 |
| Tokudb_basements_decompressed_for_write                         | 0                                                 |
| Tokudb_buffers_decompressed_target_query                        | 0                                                 |
| Tokudb_buffers_decompressed_prelocked_range                     | 0                                                 |
| Tokudb_buffers_decompressed_prefetch                            | 0                                                 |
| Tokudb_buffers_decompressed_for_write                           | 0                                                 |
| Tokudb_pivots_fetched_for_query                                 | 0                                                 |
| Tokudb_pivots_fetched_for_query_bytes                           | 0                                                 |
| Tokudb_pivots_fetched_for_query_seconds                         | 0.000000                                          |
| Tokudb_pivots_fetched_for_prefetch                              | 0                                                 |
| Tokudb_pivots_fetched_for_prefetch_bytes                        | 0                                                 |
| Tokudb_pivots_fetched_for_prefetch_seconds                      | 0.000000                                          |
| Tokudb_pivots_fetched_for_write                                 | 1                                                 |
| Tokudb_pivots_fetched_for_write_bytes                           | 512                                               |
| Tokudb_pivots_fetched_for_write_seconds                         | 0.000001                                          |
| Tokudb_basements_fetched_target_query                           | 0                                                 |
| Tokudb_basements_fetched_target_query_bytes                     | 0                                                 |
| Tokudb_basements_fetched_target_query_seconds                   | 0.000000                                          |
| Tokudb_basements_fetched_prelocked_range                        | 0                                                 |
| Tokudb_basements_fetched_prelocked_range_bytes                  | 0                                                 |
| Tokudb_basements_fetched_prelocked_range_seconds                | 0.000000                                          |
| Tokudb_basements_fetched_prefetch                               | 0                                                 |
| Tokudb_basements_fetched_prefetch_bytes                         | 0                                                 |
| Tokudb_basements_fetched_prefetch_seconds                       | 0.000000                                          |
| Tokudb_basements_fetched_for_write                              | 1                                                 |
| Tokudb_basements_fetched_for_write_bytes                        | 512                                               |
| Tokudb_basements_fetched_for_write_seconds                      | 0.000001                                          |
| Tokudb_buffers_fetched_target_query                             | 0                                                 |
| Tokudb_buffers_fetched_target_query_bytes                       | 0                                                 |
| Tokudb_buffers_fetched_target_query_seconds                     | 0.000000                                          |
| Tokudb_buffers_fetched_prelocked_range                          | 0                                                 |
| Tokudb_buffers_fetched_prelocked_range_bytes                    | 0                                                 |
| Tokudb_buffers_fetched_prelocked_range_seconds                  | 0.000000                                          |
| Tokudb_buffers_fetched_prefetch                                 | 0                                                 |
| Tokudb_buffers_fetched_prefetch_bytes                           | 0                                                 |
| Tokudb_buffers_fetched_prefetch_seconds                         | 0.000000                                          |
| Tokudb_buffers_fetched_for_write                                | 0                                                 |
| Tokudb_buffers_fetched_for_write_bytes                          | 0                                                 |
| Tokudb_buffers_fetched_for_write_seconds                        | 0.000000                                          |
| Tokudb_leaf_compression_to_memory_seconds                       | 0.000000                                          |
| Tokudb_leaf_serialization_to_memory_seconds                     | 0.000000                                          |
| Tokudb_leaf_decompression_to_memory_seconds                     | 0.000015                                          |
| Tokudb_leaf_deserialization_to_memory_seconds                   | 0.000011                                          |
| Tokudb_nonleaf_compression_to_memory_seconds                    | 0.000000                                          |
| Tokudb_nonleaf_serialization_to_memory_seconds                  | 0.000000                                          |
| Tokudb_nonleaf_decompression_to_memory_seconds                  | 0.000000                                          |
| Tokudb_nonleaf_deserialization_to_memory_seconds                | 0.000000                                          |
| Tokudb_promotion_roots_split                                    | 0                                                 |
| Tokudb_promotion_leaf_roots_injected_into                       | 0                                                 |
| Tokudb_promotion_h1_roots_injected_into                         | 0                                                 |
| Tokudb_promotion_injections_at_depth_0                          | 0                                                 |
| Tokudb_promotion_injections_at_depth_1                          | 0                                                 |
| Tokudb_promotion_injections_at_depth_2                          | 0                                                 |
| Tokudb_promotion_injections_at_depth_3                          | 0                                                 |
| Tokudb_promotion_injections_lower_than_depth_3                  | 0                                                 |
| Tokudb_promotion_stopped_nonempty_buffer                        | 0                                                 |
| Tokudb_promotion_stopped_at_height_1                            | 0                                                 |
| Tokudb_promotion_stopped_child_locked_or_not_in_memory          | 0                                                 |
| Tokudb_promotion_stopped_child_not_fully_in_memory              | 0                                                 |
| Tokudb_promotion_stopped_after_locking_child                    | 0                                                 |
| Tokudb_basement_deserialization_fixed_key                       | 2                                                 |
| Tokudb_basement_deserialization_variable_key                    | 1                                                 |
| Tokudb_pro_rightmost_leaf_shortcut_success                      | 0                                                 |
| Tokudb_pro_rightmost_leaf_shortcut_fail_pos                     | 0                                                 |
| Tokudb_rightmost_leaf_shortcut_fail_reactive                    | 0                                                 |
| Tokudb_cursor_skip_deleted_leaf_entry                           | 0                                                 |
| Tokudb_flusher_cleaner_total_nodes                              | 0                                                 |
| Tokudb_flusher_cleaner_h1_nodes                                 | 0                                                 |
| Tokudb_flusher_cleaner_hgt1_nodes                               | 0                                                 |
| Tokudb_flusher_cleaner_empty_nodes                              | 0                                                 |
| Tokudb_flusher_cleaner_nodes_dirtied                            | 0                                                 |
| Tokudb_flusher_cleaner_max_buffer_size                          | 0                                                 |
| Tokudb_flusher_cleaner_min_buffer_size                          | 18446744073709551615                              |
| Tokudb_flusher_cleaner_total_buffer_size                        | 0                                                 |
| Tokudb_flusher_cleaner_max_buffer_workdone                      | 0                                                 |
| Tokudb_flusher_cleaner_min_buffer_workdone                      | 18446744073709551615                              |
| Tokudb_flusher_cleaner_total_buffer_workdone                    | 0                                                 |
| Tokudb_flusher_cleaner_num_leaf_merges_started                  | 0                                                 |
| Tokudb_flusher_cleaner_num_leaf_merges_running                  | 0                                                 |
| Tokudb_flusher_cleaner_num_leaf_merges_completed                | 0                                                 |
| Tokudb_flusher_cleaner_num_dirtied_for_leaf_merge               | 0                                                 |
| Tokudb_flusher_flush_total                                      | 0                                                 |
| Tokudb_flusher_flush_in_memory                                  | 0                                                 |
| Tokudb_flusher_flush_needed_io                                  | 0                                                 |
| Tokudb_flusher_flush_cascades                                   | 0                                                 |
| Tokudb_flusher_flush_cascades_1                                 | 0                                                 |
| Tokudb_flusher_flush_cascades_2                                 | 0                                                 |
| Tokudb_flusher_flush_cascades_3                                 | 0                                                 |
| Tokudb_flusher_flush_cascades_4                                 | 0                                                 |
| Tokudb_flusher_flush_cascades_5                                 | 0                                                 |
| Tokudb_flusher_flush_cascades_gt_5                              | 0                                                 |
| Tokudb_flusher_split_leaf                                       | 0                                                 |
| Tokudb_flusher_split_nonleaf                                    | 0                                                 |
| Tokudb_flusher_merge_leaf                                       | 0                                                 |
| Tokudb_flusher_merge_nonleaf                                    | 0                                                 |
| Tokudb_flusher_balance_leaf                                     | 0                                                 |
| Tokudb_hot_num_started                                          | 0                                                 |
| Tokudb_hot_num_completed                                        | 0                                                 |
| Tokudb_hot_num_aborted                                          | 0                                                 |
| Tokudb_hot_max_root_flush_count                                 | 0                                                 |
| Tokudb_txn_begin                                                | 257                                               |
| Tokudb_txn_begin_read_only                                      | 0                                                 |
| Tokudb_txn_commits                                              | 129                                               |
| Tokudb_txn_aborts                                               | 128                                               |
| Tokudb_logger_next_lsn                                          | 396151                                            |
| Tokudb_logger_writes                                            | 3318                                              |
| Tokudb_logger_writes_bytes                                      | 773194                                            |
| Tokudb_logger_writes_uncompressed_bytes                         | 773194                                            |
| Tokudb_logger_writes_seconds                                    | 0.059987                                          |
| Tokudb_logger_wait_long                                         | 0                                                 |
| Tokudb_loader_num_created                                       | 0                                                 |
| Tokudb_loader_num_current                                       | 0                                                 |
| Tokudb_loader_num_max                                           | 0                                                 |
| Tokudb_memory_malloc_count                                      | 0                                                 |
| Tokudb_memory_free_count                                        | 0                                                 |
| Tokudb_memory_realloc_count                                     | 0                                                 |
| Tokudb_memory_malloc_fail                                       | 0                                                 |
| Tokudb_memory_realloc_fail                                      | 0                                                 |
| Tokudb_memory_requested                                         | 0                                                 |
| Tokudb_memory_used                                              | 0                                                 |
| Tokudb_memory_freed                                             | 0                                                 |
| Tokudb_memory_max_requested_size                                | 0                                                 |
| Tokudb_memory_last_failed_size                                  | 0                                                 |
| Tokudb_mem_estimated_maximum_memory_footprint                   | 0                                                 |
| Tokudb_memory_mallocator_version                                | 3.6.0-0-g46c0af68bd248b04df75e4f92d5fb804c3d75340 |
| Tokudb_memory_mmap_threshold                                    | 4194304                                           |
| Tokudb_filesystem_threads_blocked_by_full_disk                  | 0                                                 |
| Tokudb_filesystem_fsync_time                                    | 3920030                                           |
| Tokudb_filesystem_fsync_num                                     | 3319                                              |
| Tokudb_filesystem_long_fsync_time                               | 0                                                 |
| Tokudb_filesystem_long_fsync_num                                | 0                                                 |
| Update_scan                                                     | 0                                                 |
| Uptime                                                          | 199038                                            |
| Uptime_since_flush_status                                       | 199038                                            |
| wsrep_cluster_conf_id                                           | 18446744073709551615                              |
| wsrep_cluster_size                                              | 0                                                 |
| wsrep_cluster_state_uuid                                        |                                                   |
| wsrep_cluster_status                                            | Disconnected                                      |
| wsrep_connected                                                 | OFF                                               |
| wsrep_local_bf_aborts                                           | 0                                                 |
| wsrep_local_index                                               | 18446744073709551615                              |
| wsrep_provider_name                                             |                                                   |
| wsrep_provider_vendor                                           |                                                   |
| wsrep_provider_version                                          |                                                   |
| wsrep_ready                                                     | OFF                                               |
| wsrep_thread_count                                              | 0                                                 |
+-----------------------------------------------------------------+---------------------------------------------------+
830 rows in set (0.01 sec)

Open in new window

0
sharingsunshineAuthor Commented:
The server hasn't been restarted since Sunday afternoon.  Thanks for sticking with me.
0
gr8gonzoConsultantCommented:
You said before the memory leaks at about 0.3 gigabytes per day - have you seen that happen over the past day? If so, can you also elaborate on how you're measuring that?
0
arnoldCommented:
re-rurun mysqltunner and see what suggestions it makes near the end of the week. Do not restart mysql in the interim.

see if you have phpmyadmin is setup and use it to see what suggestions it is offering.
0
gr8gonzoConsultantCommented:
I'll be going through the status variables a little bit at a time today, but the first thing that catches my eye is that you have 179 "Aborted_connects" - that's unusual for that value to be above 0 if you are working within a single server (where the scripts and clients are all connecting locally). For example. phpMyAdmin runs from the server and it's connecting locally, so it shouldn't ever have an aborted connect.

If you haven't done so already, you should implement a firewall on your server (e.g. iptables) to block remote access to port 3306 (the port that MariaDB/MySQL runs on by default), along with other sensitive ports. If your home IP address is fixed/static, then you can just add that IP address to the firewall so that it can bypass all the blocking rules (so you can still access all the ports from your home IP). If it's dynamic or if you worry about it ever changing, you can set up a VPN on your server so you can connect securely to the server and THEN access all those ports (SSH and MySQL and so on).

If you don't have a firewall, then anyone has the ability to try to connect to your database server, and that leaves you exposed to problems such as denial-of-service attacks.
0
sharingsunshineAuthor Commented:
I have iptables enabled for fail2ban and ipsets.  Amazon has a hardware firewall and I have this entry

https://gyazo.com/a6ba6df85357117c85708a2373245261 - I have truncated the server ip which is the only thing allowed in.  

When I restarted the server I had 6.7 GB free and now I have 5.8 GB.  Seems it has increased to .53 GB per day if you figure from Sunday.

Regarding, phpMyAdmin I am not sure how to check if it is local only.
0
gr8gonzoConsultantCommented:
Another thing that catches my eye is that you have a pretty high write-to-read ratio:
Innodb_data_read                                                | 4812311040                                        |
Innodb_data_reads                                               | 295315                                            |
Innodb_data_writes                                              | 368513                                            |
Innodb_data_written                                             | 7981386240                                        |

Open in new window


Basically, it's written nearly 8 gigabytes of data and read just under 5 gigabytes. Usually most DB servers have much more reading than writing. It's possible that your particular scenario is different, but it's a little curious. You might want to look at the database and table sizes to see if anything looks unusual. I believe phpMyAdmin will do this for you already, but if you want a query to do it (you have to run it on each database), this is the one I use for my DBs:
SELECT
table_schema as `Database`,
table_name as `Table`,
AVG_ROW_LENGTH as `Average Row Length`,
FORMAT(TABLE_ROWS,0) as `Total Rows`,
FORMAT(round(((data_length + index_length) / 1024 / 1024), 0),0) as `Size in MB`
FROM information_schema.TABLES
WHERE table_schema=DATABASE()
ORDER BY (data_length + index_length) DESC

Open in new window


My gut feeling here is that you might have some pages that are getting hammered by bots. For example, bots love to try to leave spam comments on WordPress sites, so all those writes might indicate something like that happening. So try running that above query and recording the results in an Excel document, and then re-running the query after a few hours, and look for any tables that have significant increases in row counts (the "Total Rows" column).

I was thinking you might have some thread caching happening (one possible memory leak cause), but I only see 5 cached threads, so that's probably not it.

I'm not sure how many sites you're running and how busy they are, but you have about 17 million queries over about 2 days, which translates roughly to about 90 queries per second, but you've only had a max of 17 simultaneous connections, so it's a little unusual. Again, not impossibly abnormal, but it piques my interest in the output of the database/table size query differences.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
sharingsunshineAuthor Commented:
Thanks for the query.  However, I am not sure it is working correctly.

Current selection does not contain a unique column. Grid edit, checkbox, Edit, Copy and Delete features are not available. Documentation
Showing rows 0 - 24 (93 total, Query took 0.0038 seconds.)

SELECT
table_schema as `Database`,
table_name as `Table`,
AVG_ROW_LENGTH as `Average Row Length`,
FORMAT(TABLE_ROWS,0) as `Total Rows`,
FORMAT(round(((data_length + index_length) / 1024 / 1024), 0),0) as `Size in MB`
FROM information_schema.TABLES
WHERE table_schema=DATABASE()
ORDER BY (data_length + index_length) DESC

	
Database	Table	Average Row Length	Total Rows	Size in MB 	
information_schema 	EVENTS 	0 	NULL 	0
information_schema 	SYSTEM_VARIABLES 	0 	NULL 	0
information_schema 	PROCESSLIST 	0 	NULL 	0
information_schema 	ALL_PLUGINS 	0 	NULL 	0
information_schema 	PLUGINS 	0 	NULL 	0
information_schema 	VIEWS 	0 	NULL 	0
information_schema 	PARTITIONS 	0 	NULL 	0
information_schema 	PARAMETERS 	0 	NULL 	0
information_schema 	TRIGGERS 	0 	NULL 	0
information_schema 	ROUTINES 	0 	NULL 	0
information_schema 	COLUMNS 	0 	NULL 	0
information_schema 	CLIENT_STATISTICS 	386 	NULL 	0
information_schema 	INNODB_CMP 	25 	NULL 	0
information_schema 	TokuDB_background_job_status 	6276 	NULL 	0
information_schema 	COLLATIONS 	231 	NULL 	0
information_schema 	INNODB_TABLESPACES_SCRUBBING 	2020 	NULL 	0
information_schema 	INNODB_LOCK_WAITS 	599 	NULL 	0
information_schema 	FILES 	4022 	NULL 	0
information_schema 	ROCKSDB_INDEX_FILE_MAP 	3040 	NULL 	0
information_schema 	INNODB_TRX 	4534 	NULL 	0
information_schema 	TABLES 	14829 	NULL 	0
information_schema 	PROFILING 	308 	NULL 	0
information_schema 	INNODB_CMPMEM_RESET 	29 	NULL 	0
information_schema 	CHARACTER_SETS 	384 	NULL 	0
information_schema 	user_variables 	6630 	NULL 	0

Open in new window

0
gr8gonzoConsultantCommented:
You have to run the query once per user database. so if you're logged into the shell and you had a database called "thefruga_smf_test", then you'd first run:

USE thefruga_smf_test;

After that, then you'd run that above query, and it should show you the tables within that specific database in order of largest to smallest table.

Or if you want everything altogether, you can just remove the "WHERE table_schema=DATABASE()" line and then you should see all tables across all databases.
0
sharingsunshineAuthor Commented:
I understand and I have logged the entries.  I'll check them in 4 hours.
0
sharingsunshineAuthor Commented:
In the smf db there is a table called smf_log_errors and in 4 hours the number of rows increased by 6,000.
1
sharingsunshineAuthor Commented:
this is what is causing the problem.

[Sun Sep 02 22:03:46.171792 2018] [core:error] [pid 536:tid 139626199103232] (36)File name too long: [client 137.226.113.10:45186] AH00036: access to /YesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScann failed (filesystem path '/data/web/thefrugallife.com/YesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScann')
[Sun Sep 02 22:03:46.225410 2018] [core:error] [pid 467:tid 139625150539520] (36)File name too long: [client 137.226.113.10:35776] AH00036: access to /YesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScann failed (filesystem path '/data/web/thefrugallife.com/YesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScann')
[Sun Sep 02 22:03:46.268109 2018] [core:error] [pid 316:tid 139626526254848] (36)File name too long: [client 137.226.113.10:54596] AH00036: access to /YesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScann failed (filesystem path '/data/web/thefrugallife.com/YesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScann')

Open in new window

1
arnoldCommented:
An attempt to cause a buffer overflow.

Note client iP is the source, you can use arin.net, ripe.net, apnic.net to determine who is responsible for this IP and provide this log info to their abuse departments to have them address the issue with their user.
1
gr8gonzoConsultantCommented:
I agree with arnold, and I'd additionally suggest that you should block that IP address in your firewall so that their bots can't even reach the web server.
1
sharingsunshineAuthor Commented:
I have blocked at the vpc level so it won't even hit the server anymore.  Looking up in ripe.net it is an university and my experience reporting to them is a waste of time.  If you know differently let me know.

https://gyazo.com/023b9fdaf7326926bc0987134069e197 here is the ripe.net screenshot
0
gr8gonzoConsultantCommented:
Sounds good - let's do another snapshot of the table sizes to make sure no other tables are increasing like that, and see if there is any impact on the memory.
0
sharingsunshineAuthor Commented:
Since midnight last night it has increased by 18,000 rows.  I have looked in var/log/httpd/error_log my domain specific error logs and it shows the last activity was September 2nd.

So where can I look in addition?  I am concerned they may have more than one ip.

Here is a screenshot of the memory - https://gyazo.com/92072adb523808bd7859f568cb6a7eb0
0
gr8gonzoConsultantCommented:
So if those 3 example entries are the LAST activity, than that means that bot probably isn't responsible for all the SMF log errors (if the errors are still happening but the bot hasn't hit your server since 9/2).

So I'm guessing the entries into smf_log_errors are about a different error. Have you pulled some of the recent records from that table to see what they are?
0
gr8gonzoConsultantCommented:
Also, I'd still block that individual IP of that bot, but I wouldn't give it too much more thought (about it having more addresses). While you COULD look up the originating IP block of addresses and then block all of them with one command, that's probably overdoing it a bit.

That said, for sites that are specifically intended for North America, it can be effective to have the firewall block out any non-North-American IP address ranges. About 80%-90% of all of the bot attacks on any of my sites or my client sites tend to come from either Russia, China, or Poland, and then the remaining 10%-20% tends to come from other continents (a small handful occasionally come from Central and South American countries, but that's about it), with the exception of France and the UK (those don't seem to have a lot of originating attacks, but they also don't account for much valid traffic, either).

Again, that's just my own little world of sites that I manage - yours could be different, but it can be an effective safety measure to block out a lot of international traffic at the firewall level (which also grants a little more bandwidth to your web servers to handle more legitimate traffic).
0
arnoldCommented:
You should report it.
You could cite applicable law against hacking intrusion and check with the your local authorities about the process of filling a complaint.

When reporting, they will likely send you an auto-response. you should not expect here is jane and her contact info who has done this....

blocking the IP, segment. check your error log for similar errors and the source and block them if they are recent....

you could setup a process that looks at this log and adds ips to be blocked .....
0
sharingsunshineAuthor Commented:
I pulled those errors and I emptied the log file via the smf dashboard.  Many of the errors are saying functions are deprecated so I am in the process of working through those.  

Should I restart Maria or httpd?
0
arnoldCommented:
why are you considering a restart of either?
Are there numerous php or other indications that your system is being ....


Commonly, a restart of a service, especially mariadb/MySQL will deprive you of the assortment of stats that could help answer your question.

I've looked at what might use Smf db ...
0
sharingsunshineAuthor Commented:
I was thinking since I emptied the log table it might be good to restart Maria to free up what the table was taking up.  Also, at one point the memory red lined but it has since built itself back up.
0
gr8gonzoConsultantCommented:
For the deprecated functions, you mentioned you were working on those. Were they a core part of SMF or just custom code that was added to SMF somehow?
0
Bernard S.CTOCommented:
As far as I understand, the only IPs that should be able to reach your SQL directly are your WP site, localhost and its Apache, and your own IP
Use IPTables to block all other access to MYSQL (but not to Apache!)

How many simultaneous jobs do you allow for Apache? For MYSQL?.
How long is a php-page kept "open" after being fired-up?
0
arnoldCommented:
The files are binary data, there is no need to restart, shrinking of files consumes resources and is unnecessary.

You need to get the db server to continuously run as long as possible to get tunning if still needed.

Is your practice if you delete data from tables, databases you restart the db server?
0
sharingsunshineAuthor Commented:
The package was  a sanctioned mod.  I deleted it but it is still throwing errors.  So, I will be hastening my conversion to another forum.  Since, I can't get this to stop normally.

I don't have a practice this is the first time I have used the feature to delete log errors.
0
sharingsunshineAuthor Commented:
I have renamed the database, redirected the smf requests to the new BBpress forum software and ran the mysql reports again.  I can't see any row increase of any substantial amount.  But as you can see there is quite a bit of memory gone.

I also found out that smf isn't supported on php 7.2 nor Maria so that is why it was so throwing so many errors.

                                 --------------                                          10s elapsed
PRC | sys    0.02s | user   0.14s  |              | #proc    305 | #trun      1  | #tslpi   506 |              | #tslpu     0  | #zombie    0 | clones   176 |               | #exit    165 |
CPU | sys       1% | user      2%  | irq       0% |              | idle    397%  | wait      0% |              |               | steal     0% | guest     0% | curf 3.42GHz  | curscal   ?% |
cpu | sys       0% | user      1%  | irq       0% |              | idle     98%  | cpu000 w  0% |              |               | steal     0% | guest     0% | curf 3.41GHz  | curscal   ?% |
cpu | sys       0% | user      0%  | irq       0% |              | idle     99%  | cpu002 w  0% |              |               | steal     0% | guest     0% | curf 3.48GHz  | curscal   ?% |
cpu | sys       0% | user      0%  | irq       0% |              | idle     99%  | cpu003 w  0% |              |               | steal     0% | guest     0% | curf 3.41GHz  | curscal   ?% |
cpu | sys       0% | user      0%  | irq       0% |              | idle    100%  | cpu001 w  0% |              |               | steal     0% | guest     0% | curf 3.41GHz  | curscal   ?% |
CPL | avg1    0.02 | avg5    0.08  |              | avg15   0.03 |               |              | csw     4768 | intr    2790  |              |              | numcpu     4  |              |
MEM | tot     7.4G | free    3.9G  | cache   1.0G | dirty   0.3M | buff  130.4M  | slab  159.4M | slrec 121.6M | shmem   1.5M  | shrss   0.0M | vmbal   0.0M | hptot   0.0M  | hpuse   0.0M |
SWP | tot     0.0M | free    0.0M  |              |              |               |              |              |               |              |              | vmcom   5.5G  | vmlim   3.7G |
NET | transport    | tcpi      33  | tcpo      32 | udpi       1 | udpo       1  | tcpao      6 | tcppo      0 | tcprs      0  | tcpie      0 | tcpor      0 | udpnp      0  | udpie      0 |
NET | network      | ipi       37  | ipo       35 |              | ipfrw      0  | deliv     35 |              |               |              |              | icmpi      1  | icmpo      0 |
NET | eth0    ---- | pcki      37  | pcko      35 | sp    0 Mbps | si    3 Kbps  | so    3 Kbps | coll       0 | mlti       0  | erri       0 | erro       0 | drpi       0  | drpo       0 |

  PID      TID    MINFLT    MAJFLT     VSTEXT    VSLIBS     VDATA     VSTACK     VSIZE     RSIZE     PSIZE      VGROW     RGROW    SWAPSZ     RUID        EUID         MEM    CMD         1/9
14808        -         0         0     17452K    18448K      1.1G       132K      3.0G    436.3M        0K         0K        0K        0K     mysql       mysql         6%    mysqld
13697        -         3         0         4K    12088K    430.0M       132K      1.5G    322.3M        0K         0K        0K        0K     root        root          4%    fail2ban-serve
21180        -         0         0      4112K    25168K    207.8M       132K    575.3M    214.2M        0K         0K        0K        0K     apache      apache        3%    php-fpm
23207        -         0         0      4112K    25168K    154.3M       132K    521.8M    160.6M        0K         0K        0K        0K     apache      apache        2%    php-fpm
 3227        -         0         0      4112K    25168K    150.8M       132K    518.2M    157.1M        0K         0K        0K        0K     apache      apache        2%    php-fpm
20892        -         0         0      4112K    25168K    145.4M       332K    513.1M    152.0M        0K         0K        0K        0K     apache      apache        2%    php-fpm
19996        -         0         0      4112K    25168K    139.2M       132K    506.6M    145.5M        0K         0K        0K        0K     apache      apache        2%    php-fpm
20837        -         0         0      4112K    25168K    121.1M       132K    488.5M    127.5M        0K         0K        0K        0K     apache      apache        2%    php-fpm
20589        -         0         0      4112K    25168K    118.8M       132K    486.3M    124.5M        0K         0K        0K        0K     apache      apache        2%    php-fpm
20273        -         0         0      4112K    25168K    103.1M       132K    470.5M    109.3M        0K         0K        0K        0K     apache      apache        1%    php-fpm
20003        -         0         0      4112K    25168K    97420K       132K    462.6M    101.8M        0K         0K        0K        0K     apache      apache        1%    php-fpm
20958        -         0         0      4112K    25168K    91992K       132K    457.3M    97756K        0K         0K        0K        0K     apache      apache        1%    php-fpm
 1469        -        69         0       312K     3420K      712K       132K    142.6M    94488K        0K         0K        8K        0K     root        root          1%    systemd-journa
 2065        -         1         0       592K     5624K    19040K       132K    723.5M    75628K        0K         0K        0K        0K     root        root          1%    rsyslogd
  316        -         0         0       524K    12872K    714.2M       132K      2.8G    54364K        0K         0K        0K        0K     apache      apache        1%    httpd
25848        -         0         0       524K    12872K    592.8M       132K      2.7G    53648K        0K         0K        0K        0K     apache      apache        1%    httpd
  536        -         0         0       524K    12872K    579.5M       132K      2.7G    51688K        0K         0K        0K        0K     apache      apache        1%    httpd
  467        -         0         0       524K    12872K    578.6M       132K      2.7G    50784K        0K         0K        0K        0K     apache      apache        1%    httpd
  321        -         0         0       524K    12872K    577.7M       132K      2.7G    48040K        0K         0K        0K        0K     apache      apache        1%    httpd
28414        -         0         0      4112K    25084K     5508K       132K    304.9M     9944K        0K         0K        0K        0K     root        root          0%    php-fpm
24261        -         0         0      3456K    10472K     1768K       132K    171.6M     9804K        0K         0K        0K        0K     root        root          0%    mysql
21648        -         0         0       188K     3696K     5656K       708K    26688K     9596K        0K         0K        0K        0K     root        root          0%    atop
 3630        -         0         0       188K     3772K     4900K       132K    29528K     8908K        0K         0K        0K        0K     root        root          0%    atop
 3001        -         0         0       796K    12656K     1124K       132K    144.7M     8284K        0K         0K        0K        0K     root        root          0%    sshd
 3903        -         0         0       796K    12656K     1124K       132K    144.7M     8192K        0K         0K        0K        0K     root        root          0%    sshd
 3572        -         0         0       796K    12656K     1124K       132K    144.7M     8156K        0K         0K        0K        0K     root        root          0%    sshd
 2452        -        57         0       396K    14308K     4408K       132K    103.1M     6680K        0K         0K        0K        0K     root        root          0%    dhclient
 2251        -         0         0       524K    12872K     3132K       132K    269.4M     6392K        0K         0K        0K        0K     root        root          0%    httpd
 9403        -         0         0       256K    12844K      760K       132K    81940K     5508K        0K         0K        0K        0K     postfix     postfix       0%    pickup
    1        -         0         0      1404K     3776K    18088K       132K    186.8M     5132K        0K         0K        0K        0K     root        root          0%    systemd
  315        -         0         0       524K    12872K     3132K       132K    269.4M     5088K        0K         0K        0K        0K     apache      apache        0%    httpd
 3940        -         0         0       128K     6704K      864K       132K    191.6M     4936K        0K         0K        0K        0K     root        root          0%    sudo
 3608        -         0         0       128K     6704K      864K       132K    191.6M     4860K        0K         0K        0K        0K     root        root          0%    sudo
 3906        -         0         0       796K    12656K     1256K       132K    144.8M     4776K        0K         0K        0K        0K     ec2-user    ec2-user      0%    sshd

Open in new window


So, what should I do next?
0
arnoldCommented:
Memory gone does not really means anything.
The size of your database/s and the activity will dictate how much memory mysql will consume.

What is the issue you are experiencing that you are trying to resolve. memory utilization is not important if it does not impact the operation of the system.


Look at your APapche process you have 5 each reserving 2.3GB, but you are looking at the mysql instance that reserves 3GB
Usually the apache settigns are
number of child processes
number of requests each child process can respond to/handle
max child processes

each spawn php-fpm consumes 500MB

In the sum of things, your memory is consumed primarily by apache and the php-fm that is spawn.
0
gr8gonzoConsultantCommented:
I agree that the memory usage by Apache and php-fpm both seems unusual. if you restart the php-fpm service (usually just "service php-fpm restart" or whatever your php-fpm service is called), does the memory of each of those php-fpm processes drop, and if so, to what?

(Same question for Apache - what's the memory usage after a restart?)

The php-fpm service is basically designed to improve the performance of PHP requests by keeping the engine running, so to speak. However, if something is causing the memory of the child processes to bloat and never recover because it's not properly doing data cleanup, there might be a configuration issue at hand.  I would not expect the child processes to eat up more than a handful of megabytes each under normal circumstances.

You could probably change the process manager config (pm) to run "ondemand" instead of "dynamic", which could force the php-fpm processes to die a short while after they're done (so they'll still accommodate high bursts traffic really well but then they'll go away completely after a little bit of idle time). However, I'm also cautious about masking the problem accidentally.

Something else just occurred to me - are you running any custom PHP extensions? What method did you use to most recently install PHP?

Also, does your Apache setup implement any sort of memory-based caching? If it's not configured to run PHP as mod_php, then it's very unusual for its processes to be that large unless something else is being retained in memory.
0
gr8gonzoConsultantCommented:
Can you share your php-fpm.conf file?
0
sharingsunshineAuthor Commented:
[root@ip-172-31-8-214 etc]# cat php-fpm.conf
;;;;;;;;;;;;;;;;;;;;;
; FPM Configuration ;
;;;;;;;;;;;;;;;;;;;;;

; All relative paths in this configuration file are relative to PHP's install
; prefix.

; Include one or more files. If glob(3) exists, it is used to include a bunch of
; files from a glob(3) pattern. This directive can be used everywhere in the
; file.
include=/etc/php-fpm.d/*.conf

;;;;;;;;;;;;;;;;;;
; Global Options ;
;;;;;;;;;;;;;;;;;;

[global]
; Pid file
; Default Value: none
;pid = /run/php-fpm/php-fpm.pid

; Error log file
; If it's set to "syslog", log is sent to syslogd instead of being written
; in a local file.
; Default Value: /var/log/php-fpm.log
error_log = /var/log/php-fpm/error.log

; syslog_facility is used to specify what type of program is logging the
; message. This lets syslogd specify that messages from different facilities
; will be handled differently.
; See syslog(3) for possible values (ex daemon equiv LOG_DAEMON)
; Default Value: daemon
;syslog.facility = daemon

; syslog_ident is prepended to every message. If you have multiple FPM
; instances running on the same server, you can change the default value
; which must suit common needs.
; Default Value: php-fpm
;syslog.ident = php-fpm

; Log level
; Possible Values: alert, error, warning, notice, debug
; Default Value: notice
;log_level = notice

; If this number of child processes exit with SIGSEGV or SIGBUS within the time
; interval set by emergency_restart_interval then FPM will restart. A value
; of '0' means 'Off'.
; Default Value: 0
;emergency_restart_threshold = 0

; Interval of time used by emergency_restart_interval to determine when
; a graceful restart will be initiated.  This can be useful to work around
; accidental corruptions in an accelerator's shared memory.
; Available Units: s(econds), m(inutes), h(ours), or d(ays)
; Default Unit: seconds
; Default Value: 0
;emergency_restart_interval = 0

; Time limit for child processes to wait for a reaction on signals from master.
; Available units: s(econds), m(inutes), h(ours), or d(ays)
; Default Unit: seconds
; Default Value: 0
;process_control_timeout = 0

; The maximum number of processes FPM will fork. This has been designed to control
; the global number of processes when using dynamic PM within a lot of pools.
; Use it with caution.
; Note: A value of 0 indicates no limit
; Default Value: 0
;process.max = 128

; Specify the nice(2) priority to apply to the master process (only if set)
; The value can vary from -19 (highest priority) to 20 (lowest priority)
; Note: - It will only work if the FPM master process is launched as root
;       - The pool process will inherit the master process priority
;         unless specified otherwise
; Default Value: no set
;process.priority = -19

; Send FPM to background. Set to 'no' to keep FPM in foreground for debugging.
; Default Value: yes
daemonize = yes

; Set open file descriptor rlimit for the master process.
; Default Value: system defined value
;rlimit_files = 1024

; Set max core size rlimit for the master process.
; Possible Values: 'unlimited' or an integer greater or equal to 0
; Default Value: system defined value
;rlimit_core = 0

; Specify the event mechanism FPM will use. The following is available:
; - select     (any POSIX os)
; - poll       (any POSIX os)
; - epoll      (linux >= 2.5.44)
; Default Value: not set (auto detection)
;events.mechanism = epoll

; When FPM is built with systemd integration, specify the interval,
; in seconds, between health report notification to systemd.
; Set to 0 to disable.
; Available Units: s(econds), m(inutes), h(ours)
; Default Unit: seconds
; Default value: 10
;systemd_interval = 10

;;;;;;;;;;;;;;;;;;;;
; Pool Definitions ;
;;;;;;;;;;;;;;;;;;;;

; Multiple pools of child processes may be started with different listening
; ports and different management options.  The name of the pool will be
; used in logs and stats. There is no limitation on the number of pools which
; FPM can handle. Your system will tell you anyway :)

; See /etc/php-fpm.d/*.conf

Open in new window

0
sharingsunshineAuthor Commented:
before the restart php-fpm

          2018/09/08  12:25:01                                        --------------                                          10s elapsed
PRC | sys    0.02s | user   0.02s  |              | #proc    142 | #trun      1  | #tslpi   505 |              | #tslpu     0  | #zombie    0 | clones     8 |               | #exit      0 |
CPU | sys       0% | user      0%  | irq       0% |              | idle    400%  | wait      0% |              |               | steal     0% | guest     0% | curf 3.43GHz  | curscal   ?% |
cpu | sys       0% | user      0%  | irq       0% |              | idle    100%  | cpu003 w  0% |              |               | steal     0% | guest     0% | curf 3.40GHz  | curscal   ?% |
cpu | sys       0% | user      0%  | irq       0% |              | idle    100%  | cpu001 w  0% |              |               | steal     0% | guest     0% | curf 3.38GHz  | curscal   ?% |
CPL | avg1    0.01 | avg5    0.07  |              | avg15   0.03 |               |              | csw     3244 | intr    1734  |              |              | numcpu     4  |              |
MEM | tot     7.4G | free    2.1G  | cache   2.6G | dirty   0.0M | buff  154.5M  | slab  164.2M | slrec 125.7M | shmem   1.3M  | shrss   0.0M | vmbal   0.0M | hptot   0.0M  | hpuse   0.0M |
SWP | tot     0.0M | free    0.0M  |              |              |               |              |              |               |              |              | vmcom   5.9G  | vmlim   3.7G |
NET | transport    | tcpi       1  | tcpo       1 | udpi       0 | udpo       0  | tcpao      0 | tcppo      0 | tcprs      0  | tcpie      0 | tcpor      0 | udpnp      0  | udpie      0 |
NET | network      | ipi        4  | ipo        3 |              | ipfrw      0  | deliv      2 |              |               |              |              | icmpi      1  | icmpo      0 |
NET | eth0    ---- | pcki       4  | pcko       3 | sp    0 Mbps | si    0 Kbps  | so    1 Kbps | coll       0 | mlti       0  | erri       0 | erro       0 | drpi       0  | drpo       0 |
Window resized to 189x49...
  PID      TID    MINFLT    MAJFLT     VSTEXT    VSLIBS     VDATA     VSTACK     VSIZE     RSIZE     PSIZE      VGROW     RGROW    SWAPSZ     RUID        EUID         MEM    CMD         1/4
13449        -         0         0         4K    12088K    437.7M       132K      1.5G    331.3M        0K         0K        0K        0K     root        root          4%    fail2ban-serve
28052        -         0         0     17452K    18448K      1.0G       132K      3.0G    321.0M        0K         0K        0K        0K     mysql       mysql         4%    mysqld
26858        -         0         0      4112K    25168K    293.5M       132K    661.0M    298.9M        0K         0K        0K        0K     apache      apache        4%    php-fpm
26857        -         0         0      4112K    25168K    284.9M       132K    652.4M    289.8M        0K         0K        0K        0K     apache      apache        4%    php-fpm
26859        -         0         0      4112K    25168K    234.8M       132K    602.2M    240.2M        0K         0K        0K        0K     apache      apache        3%    php-fpm
27017        -         0         0      4112K    25168K    232.4M       132K    599.9M    237.7M        0K         0K        0K        0K     apache      apache        3%    php-fpm
26861        -         0         0      4112K    25168K    199.1M       132K    566.6M    204.5M        0K         0K        0K        0K     apache      apache        3%    php-fpm
30142        -         0         0      4112K    25168K    147.4M       132K    514.9M    152.8M        0K         0K        0K        0K     apache      apache        2%    php-fpm
 3708        -         0         0      4112K    25168K    141.9M       132K    509.4M    147.4M        0K         0K        0K        0K     apache      apache        2%    php-fpm
26860        -         0         0      4112K    25168K    124.4M       132K    491.9M    129.9M        0K         0K        0K        0K     apache      apache        2%    php-fpm
16388        -         0         0       524K    12872K    707.5M       132K      2.8G    55192K        0K         0K        0K        0K     apache      apache        1%    httpd
16609        -         0         0       524K    12872K    586.7M       132K      2.7G    54344K        0K         0K        0K        0K     apache      apache        1%    httpd
16389        -         0         0       524K    12872K    580.6M       132K      2.7G    52224K        0K         0K        0K        0K     apache      apache        1%    httpd
16403        -         0         0       524K    12872K    563.0M       132K      2.7G    42092K        0K         0K        0K        0K     apache      apache        1%    httpd
13363        -         0         0       524K    12872K    555.2M       132K      2.7G    35060K        0K         0K        0K        0K     apache      apache        0%    httpd
 2065        -         0         0       592K     5624K    19040K       132K    716.2M    27696K        0K         0K        0K        0K     root        root          0%    rsyslogd
26856        -         0         0      4112K    25084K     5508K       132K    304.9M    20128K        0K         0K        0K        0K     root        root          0%    php-fpm
 1469        -         0         0       312K     3420K      712K       132K    47640K    17524K        0K         0K        0K        0K     root        root          0%    systemd-journa
16386        -         0         0       524K    12872K     3028K       132K    269.3M    13112K        0K         0K        0K        0K     root        root          0%    httpd
13326        -         0         0       188K     3696K     6704K       708K    27736K    10656K        0K         0K        0K        0K     root        root          0%    atop
15321        -         0         0      3456K    10472K     1740K       132K    171.6M     9736K        0K         0K        0K        0K     root        root          0%    mysql
14350        -         0         0       188K     3772K     5440K       132K    30068K     9528K        0K         0K        0K        0K     root        root          0%    atop
13657        -         0         0       796K    12656K     1124K       132K    144.7M     8428K        0K         0K        0K        0K     root        root          0%    sshd
14252        -         0         0       796K    12656K     1124K       132K    144.7M     8328K        0K         0K        0K        0K     root        root          0%    sshd
14649        -         0         0       796K    12656K     1124K       132K    144.7M     8268K        0K         0K        0K        0K     root        root          0%    sshd
15260        -         0         0       796K    12656K     1124K       132K    144.7M     8252K        0K         0K        0K        0K     root        root          0%    sshd
 2452        -         0         0       396K    14308K     4532K       132K    103.2M     6900K        0K         0K        0K        0K     root        root          0%    dhclient
16387        -         0         0       524K    12872K     3028K       132K    269.1M     6868K        0K         0K        0K        0K     apache      apache        0%    httpd
31401        -         0         0       256K    12844K      760K       132K    81940K     5708K        0K         0K        0K        0K     postfix     postfix       0%    pickup
    1        -         0         0      1404K     3776K    18088K       132K    186.8M     5132K        0K         0K        0K        0K     root        root          0%    systemd
14327        -         0         0       128K     6704K      864K       132K    191.6M     4828K        0K         0K        0K        0K     root        root          0%    sudo
15292        -         0         0       128K     6704K      864K       132K    191.6M     4792K        0K         0K        0K        0K     root        root          0%    sudo
14704        -         0         0       128K     6704K      864K       132K    191.6M     4716K        0K         0K        0K        0K     root        root          0%    sudo
15262        -         0         0       796K    12656K     1124K       132K    144.7M     4692K        0K         0K        0K        0K     ec2-user    ec2-user      0%    sshd
13660        -         0         0       796K    12656K     1124K       132K    144.7M     4628K        0K         0K        0K        0K     ec2-user    ec2-user      0%    sshd
14659        -         0         0       796K    12656K     1124K       132K    144.7M     4548K        0K         0K        0K        0K     ec2-user    ec2-user      0%    sshd

Open in new window

0
sharingsunshineAuthor Commented:
after the restart php-fpm

                                      --------------                                          10s elapsed
PRC | sys    0.17s | user   0.47s  |              | #proc    149 | #trun      1  | #tslpi   510 |              | #tslpu     0  | #zombie    0 | clones     9 |               | #exit      1 |
CPU | sys       2% | user      5%  | irq       0% |              | idle    394%  | wait      0% |              |               | steal     0% | guest     0% | curf 3.42GHz  | curscal   ?% |
cpu | sys       0% | user      3%  | irq       0% |              | idle     96%  | cpu003 w  0% |              |               | steal     0% | guest     0% | curf 3.40GHz  | curscal   ?% |
cpu | sys       1% | user      0%  | irq       0% |              | idle     99%  | cpu002 w  0% |              |               | steal     0% | guest     0% | curf 3.48GHz  | curscal   ?% |
cpu | sys       1% | user      0%  | irq       0% |              | idle     99%  | cpu000 w  0% |              |               | steal     0% | guest     0% | curf 3.40GHz  | curscal   ?% |
cpu | sys       0% | user      1%  | irq       0% |              | idle     99%  | cpu001 w  0% |              |               | steal     0% | guest     0% | curf 3.40GHz  | curscal   ?% |
CPL | avg1    0.02 | avg5    0.06  |              | avg15   0.03 |               |              | csw     4437 | intr    2692  |              |              | numcpu     4  |              |
MEM | tot     7.4G | free    3.6G  | cache   2.6G | dirty   0.7M | buff  154.6M  | slab  164.1M | slrec 125.7M | shmem   1.3M  | shrss   0.0M | vmbal   0.0M | hptot   0.0M  | hpuse   0.0M |
SWP | tot     0.0M | free    0.0M  |              |              |               |              |              |               |              |              | vmcom   4.3G  | vmlim   3.7G |
NET | transport    | tcpi      19  | tcpo      20 | udpi       3 | udpo       3  | tcpao      0 | tcppo      2 | tcprs      2  | tcpie      0 | tcpor      0 | udpnp      0  | udpie      0 |
NET | network      | ipi       30  | ipo       24 |              | ipfrw      0  | deliv     23 |              |               |              |              | icmpi      1  | icmpo      0 |
NET | eth0    ---- | pcki      31  | pcko      27 | sp    0 Mbps | si    2 Kbps  | so    7 Kbps | coll       0 | mlti       0  | erri       0 | erro       0 | drpi       0  | drpo       0 |

  PID      TID    MINFLT    MAJFLT     VSTEXT    VSLIBS     VDATA     VSTACK     VSIZE     RSIZE     PSIZE      VGROW     RGROW    SWAPSZ     RUID        EUID         MEM    CMD         1/5
13449        -        38         0         4K    12088K    437.7M       132K      1.5G    331.4M        0K         0K        0K        0K     root        root          4%    fail2ban-serve
28052        -       246         0     17452K    18448K      1.0G       132K      3.0G    321.0M        0K         0K        0K        0K     mysql       mysql         4%    mysqld
16388        -         0         0       524K    12872K    707.5M       132K      2.8G    55192K        0K         0K        0K        0K     apache      apache        1%    httpd
16609        -         0         0       524K    12872K    587.4M       132K      2.7G    54200K        0K         0K        0K        0K     apache      apache        1%    httpd
16389        -         0         0       524K    12872K    580.6M       132K      2.7G    51028K        0K         0K        0K        0K     apache      apache        1%    httpd
16403        -         0         0       524K    12872K    563.0M       132K      2.7G    42092K        0K         0K        0K        0K     apache      apache        1%    httpd
13363        -         0         0       524K    12872K    555.2M       132K      2.7G    35060K        0K         0K        0K        0K     apache      apache        0%    httpd
 8703        -       796         0      4112K    25168K    20548K       132K    323.7M    31092K        0K     19232K    22988K        0K     apache      apache        0%    php-fpm
 8704        -       796         0      4112K    25168K    20548K       132K    323.7M    31092K        0K     19232K    22988K        0K     apache      apache        0%    php-fpm
 8705        -       796         0      4112K    25168K    20548K       132K    323.7M    31092K        0K     19232K    22988K        0K     apache      apache        0%    php-fpm
 2065        -         0         0       592K     5624K    19040K       132K    716.2M    27696K        0K         0K        0K        0K     root        root          0%    rsyslogd
 8702        -         6         0      4112K    25084K     5508K       132K    304.9M    20356K        0K         0K        0K        0K     root        root          0%    php-fpm
 1469        -         0         0       312K     3420K      712K       132K    47640K    17608K        0K         0K        0K        0K     root        root          0%    systemd-journa
16386        -         0         0       524K    12872K     3028K       132K    269.3M    13112K        0K         0K        0K        0K     root        root          0%    httpd
13326        -         0         0       188K     3696K     6704K       708K    27736K    10656K        0K         0K        0K        0K     root        root          0%    atop
15321        -         0         0      3456K    10472K     1740K       132K    171.6M     9736K        0K         0K        0K        0K     root        root          0%    mysql
14350        -         0         0       188K     3772K     5440K       132K    30068K     9528K        0K         0K        0K        0K     root        root          0%    atop
13657        -         0         0       796K    12656K     1124K       132K    144.7M     8428K        0K         0K        0K        0K     root        root          0%    sshd
14252        -         0         0       796K    12656K     1124K       132K    144.7M     8328K        0K         0K        0K        0K     root        root          0%    sshd
14649        -         0         0       796K    12656K     1124K       132K    144.7M     8268K        0K         0K        0K        0K     root        root          0%    sshd
15260        -         0         0       796K    12656K     1124K       132K    144.7M     8252K        0K         0K        0K        0K     root        root          0%    sshd
 8706        -         0         0      4112K    25084K     5508K       132K    304.9M     8048K        0K         0K        0K        0K     apache      apache        0%    php-fpm
 8707        -         0         0      4112K    25084K     5508K       132K    304.9M     8048K        0K         0K        0K        0K     apache      apache        0%    php-fpm
 8958        -        84         0      4112K    25084K     5508K       132K    304.9M     8048K        0K     304.9M     8048K        0K     apache      apache        0%    php-fpm
 2452        -         0         0       396K    14308K     4532K       132K    103.2M     6900K        0K         0K        0K        0K     root        root          0%    dhclient
16387        -         0         0       524K    12872K     3028K       132K    269.1M     6868K        0K         0K        0K        0K     apache      apache        0%    httpd
 8938        -         0         0       448K    12844K      764K       132K    82148K     6088K        0K         0K        0K        0K     postfix     postfix       0%    smtp
 8940        -         0         0       448K    12844K      764K       132K    82148K     6016K        0K         0K        0K        0K     postfix     postfix       0%    smtp
 8937        -         0         0       448K    12844K      764K       132K    82148K     5988K        0K         0K        0K        0K     postfix     postfix       0%    smtp
 8939        -         0         0       448K    12844K      764K       132K    82148K     5984K        0K         0K        0K        0K     postfix     postfix       0%    smtp
 8942        -         0         0       296K    12844K      760K       132K    81980K     5836K        0K         0K        0K        0K     postfix     postfix       0%    bounce
 8936        -         0         0       264K    12844K      760K       132K    81948K     5708K        0K         0K        0K        0K     postfix     postfix       0%    trivial-rewrit
31401        -         0         0       256K    12844K      760K       132K    81940K     5708K        0K         0K        0K        0K     postfix     postfix       0%    pickup
 8941        -         0         0       296K    12844K      760K       132K    81980K     5604K        0K         0K        0K        0K     postfix     postfix       0%    bounce
    1        -         0         0      1404K     3776K    18088K       132K    186.8M     5132K        0K         0K        0K        0K     root        root          0%    systemd

Open in new window

0
sharingsunshineAuthor Commented:
after a restart of apache

                                      --------------                                          10s elapsed
PRC | sys    0.05s | user   0.14s  |              | #proc    156 | #trun      1  | #tslpi   380 |              | #tslpu     0  | #zombie    0 | clones   230 |               | #exit     11 |
CPU | sys       1% | user      2%  | irq       0% |              | idle    397%  | wait      0% |              |               | steal     0% | guest     0% | curf 3.41GHz  | curscal   ?% |
cpu | sys       0% | user      1%  | irq       0% |              | idle     99%  | cpu001 w  0% |              |               | steal     0% | guest     0% | curf 3.40GHz  | curscal   ?% |
cpu | sys       0% | user      0%  | irq       0% |              | idle     99%  | cpu003 w  0% |              |               | steal     0% | guest     0% | curf 3.44GHz  | curscal   ?% |
cpu | sys       0% | user      0%  | irq       0% |              | idle    100%  | cpu002 w  0% |              |               | steal     0% | guest     0% | curf 3.40GHz  | curscal   ?% |
cpu | sys       0% | user      0%  | irq       0% |              | idle    100%  | cpu000 w  0% |              |               | steal     0% | guest     0% | curf 3.39GHz  | curscal   ?% |
CPL | avg1    0.05 | avg5    0.06  |              | avg15   0.03 |               |              | csw     5318 | intr    4036  |              |              | numcpu     4  |              |
MEM | tot     7.4G | free    3.7G  | cache   2.6G | dirty   0.3M | buff  154.6M  | slab  161.8M | slrec 125.6M | shmem   1.3M  | shrss   0.0M | vmbal   0.0M | hptot   0.0M  | hpuse   0.0M |
SWP | tot     0.0M | free    0.0M  |              |              |               |              |              |               |              |              | vmcom   3.4G  | vmlim   3.7G |
NET | transport    | tcpi      30  | tcpo      18 | udpi       6 | udpo       6  | tcpao      1 | tcppo      0 | tcprs      2  | tcpie      0 | tcpor      0 | udpnp      0  | udpie      0 |
NET | network      | ipi       44  | ipo       28 |              | ipfrw      0  | deliv     37 |              |               |              |              | icmpi      1  | icmpo      0 |
NET | eth0    ---- | pcki      44  | pcko      28 | sp    0 Mbps | si    3 Kbps  | so    2 Kbps | coll       0 | mlti       0  | erri       0 | erro       0 | drpi       0  | drpo       0 |

  PID      TID    MINFLT    MAJFLT     VSTEXT    VSLIBS     VDATA     VSTACK     VSIZE     RSIZE     PSIZE      VGROW     RGROW    SWAPSZ     RUID        EUID         MEM    CMD         1/5
13449        -        15         0         4K    12088K    437.7M       132K      1.5G    331.4M        0K         0K        0K        0K     root        root          4%    fail2ban-serve
28052        -         0         0     17452K    18448K      1.0G       132K      3.0G    321.0M        0K         0K        0K        0K     mysql       mysql         4%    mysqld
 8706        -         0         0      4112K    25084K    53584K       132K    351.8M    61964K        0K         0K        0K        0K     apache      apache        1%    php-fpm
 8703        -         0         0      4112K    25168K    20548K       132K    323.7M    31092K        0K         0K        0K        0K     apache      apache        0%    php-fpm
 8704        -         0         0      4112K    25168K    20548K       132K    323.7M    31092K        0K         0K        0K        0K     apache      apache        0%    php-fpm
 8705        -         0         0      4112K    25168K    20548K       132K    323.7M    31092K        0K         0K        0K        0K     apache      apache        0%    php-fpm
 2065        -         0         0       592K     5624K    19040K       132K    716.2M    27696K        0K         0K        0K        0K     root        root          0%    rsyslogd
 8702        -         0         0      4112K    25084K     5508K       132K    304.9M    20356K        0K         0K        0K        0K     root        root          0%    php-fpm
 1469        -        61         0       312K     3420K      712K       132K    47640K    17624K        0K         0K        8K        0K     root        root          0%    systemd-journa
 9243        -      1957        87       524K    12872K     3028K       132K    269.3M    13104K        0K     269.3M    13104K        0K     root        root          0%    httpd
13326        -         0         0       188K     3696K     6704K       708K    27736K    10656K        0K         0K        0K        0K     root        root          0%    atop
15321        -         0         0      3456K    10472K     1740K       132K    171.6M     9736K        0K         0K        0K        0K     root        root          0%    mysql
 9247        -       458         0       524K    12872K    526.8M       132K      2.4G     9632K        0K       2.4G     9632K        0K     apache      apache        0%    httpd
 9274        -       459         0       524K    12872K    527.0M       132K      2.5G     9632K        0K       2.5G     9632K        0K     apache      apache        0%    httpd
 9245        -       517         0       524K    12872K    655.1M       132K      2.6G     9628K        0K       2.6G     9628K        0K     apache      apache        0%    httpd
14350        -         0         0       188K     3772K     5440K       132K    30068K     9528K        0K         0K        0K        0K     root        root          0%    atop
13657        -         0         0       796K    12656K     1124K       132K    144.7M     8428K        0K         0K        0K        0K     root        root          0%    sshd
14252        -         0         0       796K    12656K     1124K       132K    144.7M     8328K        0K         0K        0K        0K     root        root          0%    sshd
14649        -         0         0       796K    12656K     1124K       132K    144.7M     8268K        0K         0K        0K        0K     root        root          0%    sshd
15260        -         0         0       796K    12656K     1124K       132K    144.7M     8252K        0K         0K        0K        0K     root        root          0%    sshd
 8707        -         0         0      4112K    25084K     5508K       132K    304.9M     8048K        0K         0K        0K        0K     apache      apache        0%    php-fpm
 8958        -         0         0      4112K    25084K     5508K       132K    304.9M     8048K        0K         0K        0K        0K     apache      apache        0%    php-fpm
 2452        -         0         0       396K    14308K     4532K       132K    103.2M     6900K        0K         0K        0K        0K     root        root          0%    dhclient
 9244        -       140         0       524K    12872K     3028K       132K    269.1M     6856K        0K     269.1M     6856K        0K     apache      apache        0%    httpd
 8938        -         0         0       448K    12844K      764K       132K    82148K     6088K        0K         0K        0K        0K     postfix     postfix       0%    smtp
 8940        -         0         0       448K    12844K      764K       132K    82148K     6016K        0K         0K        0K        0K     postfix     postfix       0%    smtp
 8937        -         0         0       448K    12844K      764K       132K    82148K     5988K        0K         0K        0K        0K     postfix     postfix       0%    smtp
 8939        -         0         0       448K    12844K      764K       132K    82148K     5984K        0K         0K        0K        0K     postfix     postfix       0%    smtp
 8942        -         0         0       296K    12844K      760K       132K    81980K     5836K        0K         0K        0K        0K     postfix     postfix       0%    bounce
 8936        -         0         0       264K    12844K      760K       132K    81948K     5708K        0K         0K        0K        0K     postfix     postfix       0%    trivial-rewrit
31401        -         0         0       256K    12844K      760K       132K    81940K     5708K        0K         0K        0K        0K     postfix     postfix       0%    pickup
 8941        -         0         0       296K    12844K      760K       132K    81980K     5604K        0K         0K        0K        0K     postfix     postfix       0%    bounce
    1        -       207         0      1404K     3776K    18088K       132K    186.8M     5132K        0K         0K        0K        0K     root        root          0%    systemd
14327        -         0         0       128K     6704K      864K       132K    191.6M     4828K        0K         0K        0K        0K     root        root          0%    sudo
15292        -         0         0       128K     6704K      864K       132K    191.6M     4792K        0K         0K        0K        0K     root        root          0%    sudo

Open in new window

0
arnoldCommented:
Posting data from atop without the requisite context on what issues you are experiencing, the only thing we have to go on is your determination that memory is being consumed.
0
sharingsunshineAuthor Commented:
no custom extensions that I know of

only cache is a wordpress plugin which is called W3 Total Cache
0
sharingsunshineAuthor Commented:
memory keeps being consumed until it will go redlined on atop and crash the server.  That is why I am concerned the memory keeps going down because if I don't intervene then the site will crash and has done that multiple times before I started tracking it.
0
arnoldCommented:
Check the configuration of the WP cache, that is likely what consumes your memory.
0
sharingsunshineAuthor Commented:
I just disabled the plugin completely

Here is atop after it was disabled.  I thought this could be a comparison

                                       --------------                                          10s elapsed
PRC | sys    2.43s | user   9.19s  |              | #proc    140 | #trun      4  | #tslpi   421 |              | #tslpu     0  | #zombie    0 | clones     8 |               | #exit      0 |
CPU | sys      24% | user     88%  | irq       0% |              | idle    289%  | wait      0% |              |               | steal     0% | guest     0% | curf 3.40GHz  | curscal   ?% |
cpu | sys       6% | user     24%  | irq       0% |              | idle     70%  | cpu002 w  0% |              |               | steal     0% | guest     0% | curf 3.40GHz  | curscal   ?% |
cpu | sys       4% | user     25%  | irq       0% |              | idle     71%  | cpu003 w  0% |              |               | steal     0% | guest     0% | curf 3.40GHz  | curscal   ?% |
cpu | sys       7% | user     21%  | irq       0% |              | idle     72%  | cpu001 w  0% |              |               | steal     0% | guest     0% | curf 3.40GHz  | curscal   ?% |
cpu | sys       6% | user     19%  | irq       0% |              | idle     75%  | cpu000 w  0% |              |               | steal     0% | guest     0% | curf 3.42GHz  | curscal   ?% |
CPL | avg1    0.59 | avg5    0.30  |              | avg15   0.18 |               |              | csw   238381 | intr  123748  |              |              | numcpu     4  |              |
MEM | tot     7.4G | free    3.0G  | cache   2.6G | dirty   0.1M | buff  155.7M  | slab  161.8M | slrec 125.2M | shmem   1.7M  | shrss   0.0M | vmbal   0.0M | hptot   0.0M  | hpuse   0.0M |
SWP | tot     0.0M | free    0.0M  |              |              |               |              |              |               |              |              | vmcom   4.5G  | vmlim   3.7G |
NET | transport    | tcpi      44  | tcpo      73 | udpi       3 | udpo       3  | tcpao      0 | tcppo      0 | tcprs      0  | tcpie      0 | tcpor      0 | udpnp      0  | udpie      0 |
NET | network      | ipi       48  | ipo       44 |              | ipfrw      0  | deliv     48 |              |               |              |              | icmpi      1  | icmpo      0 |
NET | eth0    ---- | pcki      48  | pcko      76 | sp    0 Mbps | si    3 Kbps  | so   66 Kbps | coll       0 | mlti       0  | erri       0 | erro       0 | drpi       0  | drpo       0 |

  PID      TID    MINFLT    MAJFLT     VSTEXT    VSLIBS     VDATA     VSTACK     VSIZE     RSIZE     PSIZE      VGROW     RGROW    SWAPSZ     RUID        EUID         MEM    CMD         1/4
11058        -     58924         0     17452K    18448K    984.1M       132K      3.0G    300.3M        0K         0K        0K        0K     mysql       mysql         4%    mysqld
 8704        -      3018         0      4112K    25168K    294.0M       132K    661.5M    299.1M        0K     166.1M    165.8M        0K     apache      apache        4%    php-fpm
 8707        -      1621         0      4112K    25168K    246.6M       132K    614.1M    252.0M        0K      -0.1G     -0.1G        0K     apache      apache        3%    php-fpm
10161        -         0         0      4112K    25168K    240.4M       132K    607.8M    245.8M        0K         0K        0K        0K     apache      apache        3%    php-fpm
 8703        -       642         0      4112K    25168K    113.7M       132K    481.2M    120.1M        0K         0K        0K        0K     apache      apache        2%    php-fpm
 8705        -         1         0      4112K    25168K    91056K       132K    456.4M    97612K        0K     -16.5M    -16.0M        0K     apache      apache        1%    php-fpm
 8958        -       642         0      4112K    25168K    64516K       132K    430.5M    70204K        0K     -6144K    -6144K        0K     apache      apache        1%    php-fpm
19324        -         0         0      4112K    25168K    63296K       132K    429.3M    68640K        0K         0K        0K        0K     apache      apache        1%    php-fpm
 8706        -         0         0      4112K    25168K    53708K       132K    419.9M    59756K        0K         0K        0K        0K     apache      apache        1%    php-fpm
 2065        -         0         0       592K     5624K    19040K       132K    716.2M    28524K        0K         0K        0K        0K     root        root          0%    rsyslogd
 9245        -         0         0       524K    12872K    668.7M       132K      2.8G    25068K        0K         0K        0K        0K     apache      apache        0%    httpd
 9247        -         0         0       524K    12872K    541.4M       132K      2.7G    23964K        0K         0K        0K        0K     apache      apache        0%    httpd
 9479        -        89         0       524K    12872K    538.5M       132K      2.7G    22648K        0K         0K      176K        0K     apache      apache        0%    httpd
 1469        -         0         0       312K     3420K      712K       132K    55832K    22308K        0K         0K        0K        0K     root        root          0%    systemd-journa
 8702        -         0         0      4112K    25084K     5508K       132K    304.9M    20460K        0K         0K        0K        0K     root        root          0%    php-fpm
 9274        -         1         0       524K    12872K    536.2M       132K      2.7G    19972K        0K         0K        0K        0K     apache      apache        0%    httpd
 9243        -         0         0       524K    12872K     3028K       132K    269.3M    13104K        0K         0K        0K        0K     root        root          0%    httpd
13326        -         0         0       188K     3696K     6704K       708K    27736K    10656K        0K         0K        0K        0K     root        root          0%    atop
19903        -        93         0       188K     3772K     4644K       132K    29272K     8720K        0K       -36K     -148K        0K     root        root          0%    atop
14606        -         0         0       796K    12656K     1124K       132K    144.7M     8308K        0K         0K        0K        0K     root        root          0%    sshd
19835        -         0         0       796K    12656K     1124K       132K    144.7M     8288K        0K         0K        0K        0K     root        root          0%    sshd
 2452        -         0         0       396K    14308K     4532K       132K    103.2M     6904K        0K         0K        0K        0K     root        root          0%    dhclient
 9244        -         0         0       524K    12872K     3028K       132K    269.1M     6856K        0K         0K        0K        0K     apache      apache        0%    httpd
18690        -         0         0       448K    12844K      764K       132K    82148K     6108K        0K         0K        0K        0K     postfix     postfix       0%    smtp
19297        -         0         0       352K    12844K      764K       132K    82044K     5996K        0K         0K        0K        0K     root        postfix       0%    local
19838        -         0         0       448K    12844K      764K       132K    82148K     5908K        0K         0K        0K        0K     postfix     postfix       0%    smtp
19837        -         0         0       448K    12844K      764K       132K    82148K     5888K        0K         0K        0K        0K     postfix     postfix       0%    smtp
19296        -         0         0       264K    12844K      760K       132K    81948K     5808K        0K         0K        0K        0K     postfix     postfix       0%    trivial-rewrit
19313        -         0         0       296K    12844K      760K       132K    81980K     5772K        0K         0K        0K        0K     postfix     postfix       0%    bounce
19294        -         0         0       396K    12844K      764K       132K    82088K     5744K        0K         0K        0K        0K     postfix     postfix       0%    cleanup
15854        -         0         0       256K    12844K      760K       132K    81940K     5608K        0K         0K        0K        0K     postfix     postfix       0%    pickup
19839        -         0         0       296K    12844K      760K       132K    81980K     5608K        0K         0K        0K        0K     postfix     postfix       0%    bounce
    1        -         0         0      1404K     3776K    18088K       132K    186.8M     5132K        0K         0K        0K        0K     root        root          0%    systemd
14640        -         0         0       128K     6704K      864K       132K    191.6M     4800K        0K         0K        0K        0K     root        root          0%    sudo
19881        -         0         0       128K     6704K      864K       132K    191.6M     4760K        0K         0K        0K        0K     root        root          0%    sudo

Open in new window

0
gr8gonzoConsultantCommented:
I've used the W3 Total Cache before on a really low-memory server - it shouldn't be the problem. If I recall correctly, that plugin just creates consolidated caches of content like CSS, Javascript, and so on in the filesystem. It's not usually in memory.

So it's REALLY concerning that:
#1. Your Apache processes are still several gigabytes after a restart.
#2. The php-fpm processes are still several hundred megabytes.

That tends to indicate there's something really heavy being loaded into memory, so let's look at the Apache modules and PHP extensions.

1. In the shell, find your httpd (the Apache executable) and run it with the -M flag:
/path/to/httpd -M

Open in new window


This should dump a list of all the Apache modules that get loaded. There will probably about a couple dozen or so - just paste that output here.

2. Create a new PHP script called info.php and have it contain just one line:
<?php phpinfo();

Open in new window

...and provide that URL here (or if you have a screen capture tool like Snagit that can do a scrolling capture of the entire page, that will work better). After this whole memory issue is fixed, you should probably remove that script. It's not dangerous by itself, but it's always good practice to limit what information you expose to the world.
0
gr8gonzoConsultantCommented:
It also might be helpful to check out the php-fpm error log in case there's anything relevant in there.
0
gr8gonzoConsultantCommented:
It's also noteworthy that there are some httpd processes that aren't huge - which makes me wonder if you're simply using an Apache MPM with some settings that allows the process to grow and you're just getting a ton of requests really quickly.

Have you checked your web access logs for your sites to look for any unusual patterns (e.g. tons of traffic from the same or similar IPs) ?
0
sharingsunshineAuthor Commented:
I am not clear how to do #1 this is a Linux 2 server.

[root@ip-172-31-8-214 modules]# /etc/selinux/targeted/active/modules/100/apache -M
-bash: /etc/selinux/targeted/active/modules/100/apache: Is a directory
[root@ip-172-31-8-214 modules]# find / -name httpd
/etc/logrotate.d/httpd
/etc/httpd
/run/httpd
/var/cache/httpd
/var/log/httpd
/var/lib/httpd
/usr/sbin/httpd
/usr/lib64/httpd
/usr/share/httpd
/usr/include/httpd
/usr/libexec/initscripts/legacy-actions/httpd

Open in new window


Searching for apache and httpd this is what I get.  Where do I go from here?
0
sharingsunshineAuthor Commented:
this is a list of the modules - will this do?

mod_access_compat.so  mod_authz_core.so       mod_cgi.so           mod_expires.so              mod_log_config.so    mod_proxy_balancer.so  mod_ratelimit.so         mod_speling.so
mod_actions.so        mod_authz_dbd.so        mod_charset_lite.so  mod_ext_filter.so           mod_log_debug.so     mod_proxy_connect.so   mod_reflector.so         mod_ssl.so
mod_alias.so          mod_authz_dbm.so        mod_data.so          mod_filter.so               mod_log_forensic.so  mod_proxy_express.so   mod_remoteip.so          mod_status.so
mod_allowmethods.so   mod_authz_groupfile.so  mod_dav_fs.so        mod_headers.so              mod_logio.so         mod_proxy_fcgi.so      mod_reqtimeout.so        mod_substitute.so
mod_asis.so           mod_authz_host.so       mod_dav_lock.so      mod_heartbeat.so            mod_lua.so           mod_proxy_fdpass.so    mod_request.so           mod_suexec.so
mod_auth_basic.so     mod_authz_owner.so      mod_dav.so           mod_heartmonitor.so         mod_macro.so         mod_proxy_ftp.so       mod_rewrite.so           mod_systemd.so
mod_auth_digest.so    mod_authz_user.so       mod_dbd.so           mod_http2.so                mod_mime_magic.so    mod_proxy_hcheck.so    mod_sed.so               mod_unique_id.so
mod_authn_anon.so     mod_autoindex.so        mod_deflate.so       mod_include.so              mod_mime.so          mod_proxy_http2.so     mod_setenvif.so          mod_unixd.so
mod_authn_core.so     mod_buffer.so           mod_dialup.so        mod_info.so                 mod_mpm_event.so     mod_proxy_http.so      mod_slotmem_plain.so     mod_userdir.so
mod_authn_dbd.so      mod_cache_disk.so       mod_dir.so           mod_lbmethod_bybusyness.so  mod_mpm_prefork.so   mod_proxy_scgi.so      mod_slotmem_shm.so       mod_usertrack.so
mod_authn_dbm.so      mod_cache.so            mod_dumpio.so        mod_lbmethod_byrequests.so  mod_mpm_worker.so    mod_proxy.so           mod_socache_dbm.so       mod_version.so
mod_authn_file.so     mod_cache_socache.so    mod_echo.so          mod_lbmethod_bytraffic.so   mod_negotiation.so   mod_proxy_uwsgi.so     mod_socache_memcache.so  mod_vhost_alias.so
mod_authn_socache.so  mod_cgid.so             mod_env.so           mod_lbmethod_heartbeat.so   mod_proxy_ajp.so     mod_proxy_wstunnel.so  mod_socache_shmcb.so     mod_watchdog.so

Open in new window

0
gr8gonzoConsultantCommented:
Try doing:
ps aux | grep "httpd"

It should show you the full path to the httpd binary.

At first, I thought you had actually found the binary and ran it and it had loaded all of those modules (!!!!), but I see now that it's just a file listing of the available modules, which isn't sufficient.

The available modules doesn't tell us what's actually loaded, which is what is important/relevant here.
0
sharingsunshineAuthor Commented:
0
sharingsunshineAuthor Commented:
[root@ip-172-31-8-214 httpd]# /usr/sbin/httpd -M
Loaded Modules:
 core_module (static)
 so_module (static)
 http_module (static)
 access_compat_module (shared)
 actions_module (shared)
 alias_module (shared)
 allowmethods_module (shared)
 auth_basic_module (shared)
 auth_digest_module (shared)
 authn_anon_module (shared)
 authn_core_module (shared)
 authn_dbd_module (shared)
 authn_dbm_module (shared)
 authn_file_module (shared)
 authn_socache_module (shared)
 authz_core_module (shared)
 authz_dbd_module (shared)
 authz_dbm_module (shared)
 authz_groupfile_module (shared)
 authz_host_module (shared)
 authz_owner_module (shared)
 authz_user_module (shared)
 autoindex_module (shared)
 cache_module (shared)
 cache_disk_module (shared)
 cache_socache_module (shared)
 data_module (shared)
 dbd_module (shared)
 deflate_module (shared)
 dir_module (shared)
 dumpio_module (shared)
 echo_module (shared)
 env_module (shared)
 expires_module (shared)
 ext_filter_module (shared)
 filter_module (shared)
 headers_module (shared)
 include_module (shared)
 info_module (shared)
 log_config_module (shared)
 logio_module (shared)
 macro_module (shared)
 mime_magic_module (shared)
 mime_module (shared)
 negotiation_module (shared)
 remoteip_module (shared)
 reqtimeout_module (shared)
 request_module (shared)
 rewrite_module (shared)
 setenvif_module (shared)
 slotmem_plain_module (shared)
 slotmem_shm_module (shared)
 socache_dbm_module (shared)
 socache_memcache_module (shared)
 socache_shmcb_module (shared)
 status_module (shared)
 substitute_module (shared)
 suexec_module (shared)
 unique_id_module (shared)
 unixd_module (shared)
 userdir_module (shared)
 version_module (shared)
 vhost_alias_module (shared)
 watchdog_module (shared)
 dav_module (shared)
 dav_fs_module (shared)
 dav_lock_module (shared)
 lua_module (shared)
 mpm_worker_module (shared)
 proxy_module (shared)
 lbmethod_bybusyness_module (shared)
 lbmethod_byrequests_module (shared)
 lbmethod_bytraffic_module (shared)
 lbmethod_heartbeat_module (shared)
 proxy_ajp_module (shared)
 proxy_balancer_module (shared)
 proxy_connect_module (shared)
 proxy_express_module (shared)
 proxy_fcgi_module (shared)
 proxy_fdpass_module (shared)
 proxy_ftp_module (shared)
 proxy_http_module (shared)
 proxy_hcheck_module (shared)
 proxy_scgi_module (shared)
 proxy_uwsgi_module (shared)
 proxy_wstunnel_module (shared)
 ssl_module (shared)
 systemd_module (shared)
 cgid_module (shared)
 http2_module (shared)
 proxy_http2_module (shared)

Open in new window

0
sharingsunshineAuthor Commented:
php-fpm error log
[28-Aug-2018 22:03:54] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 4 idle, and 14 total children
[05-Sep-2018 18:12:20] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 3 idle, and 13 total children
[05-Sep-2018 18:12:26] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 4 idle, and 18 total children
[05-Sep-2018 18:12:27] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 4 idle, and 19 total children
[

Open in new window

0
gr8gonzoConsultantCommented:
Okay, wow, yeah you have a lot of those modules loaded in, which would definitely pump up the size of the httpd processes. It's pretty doubtful you use all of them, though.

I pulled together the most common modules from a variety of servers and then came up with the modules that you have enabled that don't show up on my list:

 actions_module (shared)
 allowmethods_module (shared)
 auth_digest_module (shared)
 authn_anon_module (shared)
 authn_dbd_module (shared)
 authn_dbm_module (shared)
 authn_socache_module (shared)
 authz_dbd_module (shared)
 authz_dbm_module (shared)
 authz_owner_module (shared)
 cache_disk_module (shared)
 cache_module (shared)
 cache_socache_module (shared)
 cgid_module (shared)
 data_module (shared)
 dav_fs_module (shared)
 dav_lock_module (shared)
 dav_module (shared)
 dbd_module (shared)
 deflate_module (shared)
 dumpio_module (shared)
 echo_module (shared)
 ext_filter_module (shared)
 http2_module (shared)
 include_module (shared)
 info_module (shared)
 lbmethod_bybusyness_module (shared)
 lbmethod_byrequests_module (shared)
 lbmethod_bytraffic_module (shared)
 lbmethod_heartbeat_module (shared)
 logio_module (shared)
 lua_module (shared)
 macro_module (shared)
 mime_magic_module (shared)
 negotiation_module (shared)
 proxy_ajp_module (shared)
 proxy_balancer_module (shared)
 proxy_connect_module (shared)
 proxy_express_module (shared)
 proxy_fdpass_module (shared)
 proxy_ftp_module (shared)
 proxy_hcheck_module (shared)
 proxy_http_module (shared)
 proxy_http2_module (shared)
 proxy_scgi_module (shared)
 proxy_uwsgi_module (shared)
 proxy_wstunnel_module (shared)
 remoteip_module (shared)
 reqtimeout_module (shared)
 request_module (shared)
 slotmem_plain_module (shared)
 slotmem_shm_module (shared)
 socache_dbm_module (shared)
 socache_memcache_module (shared)
 substitute_module (shared)
 suexec_module (shared)
 systemd_module (shared)
 unique_id_module (shared)
 userdir_module (shared)
 vhost_alias_module (shared)
 watchdog_module (shared)

Open in new window


Now, just because you have something enabled that most of my servers don't doesn't mean that it's not in use. Every module has a purpose, so it's a question of whether you actually make use of it or not.

For example, proxy_ftp_module lets Apache be a proxy for other FTP sites.
https://httpd.apache.org/docs/2.4/mod/mod_proxy_ftp.html

It's not exactly "common" behavior, and proxy modules can be dangerous if you don't have the appropriate security measures in place (it can basically lead to your server being abused / hijacked).

The suexec_module lets Apache execute certain commands with elevated permissions (su = superuser / root).

All of the dav_ modules are for supporting file management via the WebDAV protocol, so if you don't use WebDAV to upload / download your files then it's just taking up memory and being a potential security hazard.

All the lb_ modules are for load balancing and work in conjunction with the proxy_balancer_module.

So on and so on.

It might be a lot of work to go through all of them to figure out what they each do, but it's probably worth it. You're very likely loading up a TON of functionality that isn't ever used and is ballooning your memory usage, especially modules that are intended to allocate memory for data caching (memcache, slotmem, etc).

So maybe make some backup copies of all your Apache config files, and then start stripping out everything you're unsure about and after the changes, save and run httpd -t to do a test of the configuration to make sure it all still looks okay (DON'T restart yet).

Once you get to a good spot in cleanup, then go ahead and restart Apache after saving, and if it comes back up properly, then immediately check your sites to make sure none of them have been impacted in any way. If they are, then swap out the configuration file for the backup and restart to get back to your original state, and then swap the configs back and try to determine which module removal caused the issue.

It's ugly, but if you don't have a separate, mirrored dev server, then this is pretty much the way to do it.

Now, the other big thing is that you're using the Worker MPM module:
 mpm_worker_module (shared)

You're using Apache 2.4 which means you have access to the Event MPM module, which is pretty much superior in both performance and memory requirements when compared to the Worker MPM. Unfortunately, your MPM modules aren't compiled in statically (which would give you a performance boost), but that's okay - it'll make it easy to switch to the event MPM.

There are several articles you can read on the differences between MPMs, but you'll find that everyone recommends using event if you can use it (and you should be able to). The only reason event wouldn't outperform worker is if you didn't have it configured correctly.
0
sharingsunshineAuthor Commented:
how do you disable modules because I don't do proxy or webdav for sure.
0
sharingsunshineAuthor Commented:
looking in httpd.conf I don't see the event or worker module.  I looked in /etc/sysconfig too and didn't find anything.  Any suggestions?
0
gr8gonzoConsultantCommented:
Now onto php-fpm - those error messages do indicate some spikes in activity, and some pretty heavy traffic to the PHP-FPM processes overall.

The phpinfo() output indicates you have a lot of PHP extensions enabled, too. It's not quite as crazy of a list as Apache's, but it's definitely worth figuring out if you really need all the functionality. For example, the bzip extension probably isn't used, especially since you have the more common zlib extension installed (which provides access to the more commonly-used deflate/gzip methods). Same goes for calendar, and probably soap, and ftp. These are pretty rarely-used PHP extensions, so unless you have software that explicitly uses them, they're probably not doing anything except taking up memory in every PHP-FPM child process.

I think the biggest thing is really determining whether you have a ton of legitimate traffic, or if you're getting hammered by some bots or if you have something running that shouldn't be (e.g. a hijacked endpoint). Many years ago, I had a database server that got hijacked and I didn't realize it until we ran out of disk space, and then realized that one vulnerability had led to an exploit that let someone take over the server, set up an FTP server, and turned the whole thing into a distribution point for pirated movies in German.

So checking the web access logs is going to be important here, but if you want to just take a quick snapshot of the current network traffic connections, you can use netstat:
netstat -npa

Open in new window


That should give you a peek at who's connected to what ports on your server, but if you see a lot of HTTP traffic, then you're going to want to check those logs. You can repeat the netstat command to see if there are any repeated IPs that are hammering any of your services.

If you don't already have a good log analyzer for Apache logs, Sawmill is an easy and powerful (but expensive) program. There are some free log analyzers like awstats, too. Or you can try to just eyeball it, but my guess is that you're going to have some pretty large log files.
0
gr8gonzoConsultantCommented:
To disable Apache modules, you'd go into the httpd.conf file and you should see a big list of "LoadModule" directives, like this:

LoadModule authn_file_module modules/mod_authn_file.so
#LoadModule authn_dbm_module modules/mod_authn_dbm.so
#LoadModule authn_anon_module modules/mod_authn_anon.so
#LoadModule authn_dbd_module modules/mod_authn_dbd.so
#LoadModule authn_socache_module modules/mod_authn_socache.so
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authz_host_module modules/mod_authz_host.so
...etc...

To disable a module from loading, just add a # in front of it to comment out that line. It will take effect the next time you save and restart (but again, test your configs before restarting).

Not all proxy modules are bad. For example, the fcgi proxy is necessary in order to allow Apache to talk to PHP-FPM, so try to stick to just the list I provided.
0
sharingsunshineAuthor Commented:
I found one bot that hit the site 1220 times in a 24 hour period.  I have stopped it and I will work on the modules and the extensions too.  But to do that I need to know where the modules are to be enabled.
0
gr8gonzoConsultantCommented:
Changing the MPM might be in a different config file. Look for a config file that has "mpm" in the name. You might have to change it there.
0
gr8gonzoConsultantCommented:
Sorry I can't be clearer - filenames and setups different from machine to machine, distribution to distribution, so it's hard to tell you exactly where to look.
0
gr8gonzoConsultantCommented:
If it helps at all, I have a source distribution where the MPM details are contained within the "conf/extra/http-mpm.conf" file, but since I usually compile in the MPM statically, I'm not sure off the top of my head if that's where you'd have the appropriate module load line.
0
David FavorLinux/LXD/WordPress/Hosting SavantCommented:
If you look at your initial question, mysqltuner tells you...

innodb_buffer_pool_size (>= 5.6G) if possible.

Open in new window


If you only have 3G of memory... you can't... magically make 3G into 5.6G, so nothing you do anywhere on your machine is going to fix your problem, until you increase your VPS memory.

There may be other problems to fix + memory is first fix, then consider if any other problems exist.
0
gr8gonzoConsultantCommented:
Sorry, I started writing a comment on my phone, but this post has become so long that my phone started acting sluggish and it ended up deleting most of what I wrote and then posted.

David,
1. Please read through the other comments before adding yours.

2. The OP stated he doubled memory, and it still crashed.

3. I've said this a couple times - don't blindly follow the instructions of MySQLTuner. Tuner is not a problem-solving tool, it's a performance-tweaking tool. Plus, you should never trust the results based on a short uptime, which is why the output of MySQLTuner also says this:
MySQL was started within the last 24 hours - recommendations may be inaccurate

Open in new window


4. The InnoDB buffer pool size is a performance-related item. It is not stating that MySQL requires 5.6G.
0
David FavorLinux/LXD/WordPress/Hosting SavantCommented:
Best to post new output of mysqltuner, which should show memory expansion.

The InnoDB buffer pool size is a performance-related item. It is not stating that MySQL requires 5.6G is true + give indication of... how to say this... If PHP processes begin to backup due to inability to quickly access data... the result can be an amplifying stampede of processes which overwhelm resources.

If your site is low traffic, just ignore the mysqltuner performance suggestions + focus on mysqltuner output relating to go/nogo info, like max_connections being too low.

If your site is high traffic, then pretty much do what mysqltuner tells you to do.

Post output of following when problem is surfacing...

1. mysqltuner full output

2. top output

3. ps fauxww
0
gr8gonzoConsultantCommented:
Again, please read thoroughly through the comments before posting any more. I'm not trying to be rude here - I'm just trying to eliminate repetition. One of the steps we did early on was to enable and monitor the slow query log. If there were issues with queries not running fast enough (and the InnoDB buffer pool would be one of the last things to change to fix that problem), then we would have seen the slow query log filling up with tons of slow queries and causing PHP process congestion. That is not the case here.

It's great if you are coming to the table with new ideas to try, but there's a lot of information in this thread, and if we start repeating things that can be discounted by looking at previously-posted data, it's just going to make the thread even longer.
0
sharingsunshineAuthor Commented:
I changed from worker to event and I have disabled several apache modules.  I restarted Maria, httpd and php-fpm at 4:05.  At that time, I had between free and cache a total of 6.6 GB.

At 4:16 I have a total of 6.1 GB and at 4:21 I have a total of 5.9 GB.

Is there any particular php extension that is making you think it could be leaking memory?
0
arnoldCommented:
You should not restart mariadb...

You need the stats.


What are you using?  The applications you are publishing, the visitors......

It seems you've under speced your server for what you need in terms of size of db, number of visitors, etc.

You are coming here after the fact and trying to roll resources back.

Without knowing what application you are running, which commonly includes which modules they need. You could disable mod_alias, mod_rewrite, mod_ssl (if you do not use HTTPS ...)

I am unfamiliar with whether the scenario you are under provides a benefit if using php-fpm.

Or whether not using it will reduce the memory consumption .......

Do you crunch the web access log to see what might be going on.

You have fail2ban that is also consuming resources along with iptables which presumably what fail2ban updates.....
0
sharingsunshineAuthor Commented:
These responses have answered the question of why mysql was growing exponentially.  Gr8Gonzo nailed it with having me compare the rows.  This showed me that smf was creating several thousand error entries per hour.  Removing smf and switching to BBpress has stopped Maria from growing so fast.

There is more to do on the memory leak and I will post a further question to get that resolved.

One thing, please don't bother commenting about under sized resources.  That has been answered many times already.  Nor, any suggestions of Mysql tuner.  If you have other helpful suggestions this is the url to the follow on question - https://www.experts-exchange.com/questions/29115269/Why-Does-Mysql-Keep-Increasing-Memory-Usage.html
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
MariaDB

From novice to tech pro — start learning today.