How To Find PHP or Apache Memory Leak

Good you finally figured out the problem.

The immediate fix, is to set a max_request limit so PHP processes are reaped/killed frequently...

# /etc/php/7.2/fpm/pool.d/www.conf
pm.max_requests = 100

Select allOpen in new window

The default of 0 means processes live forever, so if there's a leak, then your machine will eventually die.

That said, latest stable Apache + PHP contain no reported leaks at this time, so your next step is to upgrade your Apache + PHP.

Latest stable versions, as of today, with all minor patches applied...

Apache-2.4.34.1
PHP-7.2.9.1

Select allOpen in new window

Install latest + likely your memory leak will be fixed.

If the leak continues, likely problem is PHP or FPM, so you'll open a bug against... I'd likely open a PHP bug, since FPM + PHP are now tightly coupled.

I have the latest that Amazon offers.  In fact, I just ran their latest update.

Server version: Apache/2.4.34 ()
Server built:   Aug 17 2018 22:14:33
Server's Module Magic Number: 20120211:79
Server loaded:  APR 1.6.3, APR-UTIL 1.6.1
Compiled using: APR 1.6.3, APR-UTIL 1.6.1
Architecture:   64-bit
Server MPM:     event
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)

Select allOpen in new window

[root@ip-172-31-8-214 ~]# php -v
PHP 7.2.8 (cli) (built: Aug 14 2018 16:45:02) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies

Select allOpen in new window

I also ran Yum with the -y flag which updates anything that needs to be updated.

I just switched to ondemand instead of dynamic and will keep any eye on if this solves the issue or not.

http://linuxbsdos.com/2015/02/17/how-to-reduce-php-fpm-php5-fpm-ram-usage-by-about-50/

Select allOpen in new window

yes, I looked at all of the access logs and error logs.

I see a lot of proc files that are changing but the logs I see are what I have checked and the wordfence logs.

Here is the netstat command for httpd

[root@ip-172-31-8-214 ~]# netstat -npa | grep httpd
tcp6       0      0 :::80                   :::*                    LISTEN      830/httpd
tcp6       0      0 :::443                  :::*                    LISTEN      830/httpd
tcp6       0      0 172.31.9.125:443        24.115.158.88:63411     FIN_WAIT2   21100/httpd
unix  2      [ ACC ]     STREAM     LISTENING     34005020 21088/httpd          /run/httpd/cgisock.830
unix  3      [ ]         STREAM     CONNECTED     33349561 830/httpd
[root@ip-172-31-8-214 ~]#

Select allOpen in new window

the wordfence logs are something new to me and one comment on the internet I found said that there was a concentrated attack against wordfence.

I mention this because I set the command on 1 minute and the directory holding these files showed up.

[root@ip-172-31-8-214 wflogs]# ls -la
total 4652
drwxr-sr-x  2 apache apache    4096 Sep 10 15:09 .
drwxr-sr-x 12 apache apache    4096 Sep  8 14:39 ..
-rw-rw----  1 apache apache   40083 Sep 10 04:58 attack-data.php
-rw-rw----  1 apache apache 1113107 Sep 10 15:09 config.php
-rw-r--r--  1 apache apache 3399631 Jul 16 10:31 GeoLite2-Country.mmdb
-rw-r--r--  1 apache apache     133 Jul 11 11:22 .htaccess
-rw-rw----  1 apache apache      51 Sep 10 13:59 ips.php
-rw-rw-r--  1 apache apache  128057 Sep 10 10:41 rules.php
-rw-rw----  1 apache apache   58404 Sep 10 10:41 wafRules.rules

Select allOpen in new window

Is there a way to tell if someone is hitting these logs via ssh?  Or, do I just need to search for them in the log files manually?

Wordfence should send you a summary email at various points, that contains a list of all the IPs that tried to attack the site. If someone is attacking the site, then some of these files will get updated so it would be reasonable for these files to get updated frequently.

Since WP is a pretty popular platform, it's an attractive target to bots that are just out there to pound away. WordFence does a good job of keep them at bay, but I usually look for any huge offenders (thousands of hits from an IP) in the summary emails and then I block that IP in the firewall just to decrease load and filter out some of the noise.

I have a program that looks at the access logs for each domain and no one has hit the site over 500 times in a 12 hour period.

Yet, the leak has slowed but it isn't gone.  I removed all the excess modules so should I now start on the php extensions?

it seems the ondemand change fixed the problem.  It has ran now for nearly a month without any issues.  Thanks for all of the help getting me to this point.