How To Find PHP or Apache Memory Leak

This is a follow on question to https://www.experts-exchange.com/questions/29115269/Why-Does-Mysql-Keep-Increasing-Memory-Usage.html

Now it appears that the memory leaks are coming from php-fpm and Apache.

Please tell me how to isolate what is using up the memory so quickly.

In less than an hour I have lost .9 GB.
sharingsunshineAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

David FavorLinux/LXD/WordPress/Hosting SavantCommented:
Good you finally figured out the problem.

The immediate fix, is to set a max_request limit so PHP processes are reaped/killed frequently...

# /etc/php/7.2/fpm/pool.d/www.conf
pm.max_requests = 100

Open in new window


The default of 0 means processes live forever, so if there's a leak, then your machine will eventually die.

That said, latest stable Apache + PHP contain no reported leaks at this time, so your next step is to upgrade your Apache + PHP.

Latest stable versions, as of today, with all minor patches applied...

Apache-2.4.34.1
PHP-7.2.9.1

Open in new window


Install latest + likely your memory leak will be fixed.

If the leak continues, likely problem is PHP or FPM, so you'll open a bug against... I'd likely open a PHP bug, since FPM + PHP are now tightly coupled.
0
sharingsunshineAuthor Commented:
I have the latest that Amazon offers.  In fact, I just ran their latest update.

Server version: Apache/2.4.34 ()
Server built:   Aug 17 2018 22:14:33
Server's Module Magic Number: 20120211:79
Server loaded:  APR 1.6.3, APR-UTIL 1.6.1
Compiled using: APR 1.6.3, APR-UTIL 1.6.1
Architecture:   64-bit
Server MPM:     event
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)

Open in new window


[root@ip-172-31-8-214 ~]# php -v
PHP 7.2.8 (cli) (built: Aug 14 2018 16:45:02) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies

Open in new window


I also ran Yum with the -y flag which updates anything that needs to be updated.
0
sharingsunshineAuthor Commented:
I just switched to ondemand instead of dynamic and will keep any eye on if this solves the issue or not.

http://linuxbsdos.com/2015/02/17/how-to-reduce-php-fpm-php5-fpm-ram-usage-by-about-50/

Open in new window

0
10 Tips to Protect Your Business from Ransomware

Did you know that ransomware is the most widespread, destructive malware in the world today? It accounts for 39% of all security breaches, with ransomware gangsters projected to make $11.5B in profits from online extortion by 2019.

gr8gonzoConsultantCommented:
I mentioned the ondemand change in that previous question, and the word of caution still applies:
You could probably change the process manager config (pm) to run "ondemand" instead of "dynamic", which could force the php-fpm processes to die a short while after they're done (so they'll still accommodate high bursts traffic really well but then they'll go away completely after a little bit of idle time). However, I'm also cautious about masking the problem accidentally.

When you analyzed the access logs, did you check all of them, and did you run netstat to review the connections?

You can use the find tool to find all files that have been modified in the last X minutes like this:

find / -mmin -5

This might be helpful in identifying actively-changing files, in case there are logs or data files that we haven't previously identified that are constantly changing.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
sharingsunshineAuthor Commented:
yes, I looked at all of the access logs and error logs.

I see a lot of proc files that are changing but the logs I see are what I have checked and the wordfence logs.

Here is the netstat command for httpd

[root@ip-172-31-8-214 ~]# netstat -npa | grep httpd
tcp6       0      0 :::80                   :::*                    LISTEN      830/httpd
tcp6       0      0 :::443                  :::*                    LISTEN      830/httpd
tcp6       0      0 172.31.9.125:443        24.115.158.88:63411     FIN_WAIT2   21100/httpd
unix  2      [ ACC ]     STREAM     LISTENING     34005020 21088/httpd          /run/httpd/cgisock.830
unix  3      [ ]         STREAM     CONNECTED     33349561 830/httpd
[root@ip-172-31-8-214 ~]#

Open in new window

0
sharingsunshineAuthor Commented:
the wordfence logs are something new to me and one comment on the internet I found said that there was a concentrated attack against wordfence.

I mention this because I set the command on 1 minute and the directory holding these files showed up.

[root@ip-172-31-8-214 wflogs]# ls -la
total 4652
drwxr-sr-x  2 apache apache    4096 Sep 10 15:09 .
drwxr-sr-x 12 apache apache    4096 Sep  8 14:39 ..
-rw-rw----  1 apache apache   40083 Sep 10 04:58 attack-data.php
-rw-rw----  1 apache apache 1113107 Sep 10 15:09 config.php
-rw-r--r--  1 apache apache 3399631 Jul 16 10:31 GeoLite2-Country.mmdb
-rw-r--r--  1 apache apache     133 Jul 11 11:22 .htaccess
-rw-rw----  1 apache apache      51 Sep 10 13:59 ips.php
-rw-rw-r--  1 apache apache  128057 Sep 10 10:41 rules.php
-rw-rw----  1 apache apache   58404 Sep 10 10:41 wafRules.rules

Open in new window


Is there a way to tell if someone is hitting these logs via ssh?  Or, do I just need to search for them in the log files manually?
0
gr8gonzoConsultantCommented:
Wordfence should send you a summary email at various points, that contains a list of all the IPs that tried to attack the site. If someone is attacking the site, then some of these files will get updated so it would be reasonable for these files to get updated frequently.

Since WP is a pretty popular platform, it's an attractive target to bots that are just out there to pound away. WordFence does a good job of keep them at bay, but I usually look for any huge offenders (thousands of hits from an IP) in the summary emails and then I block that IP in the firewall just to decrease load and filter out some of the noise.
0
sharingsunshineAuthor Commented:
I have a program that looks at the access logs for each domain and no one has hit the site over 500 times in a 12 hour period.

Yet, the leak has slowed but it isn't gone.  I removed all the excess modules so should I now start on the php extensions?
0
sharingsunshineAuthor Commented:
it seems the ondemand change fixed the problem.  It has ran now for nearly a month without any issues.  Thanks for all of the help getting me to this point.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
PHP

From novice to tech pro — start learning today.