linux system out of memory kill processes

it is better to increase RAM of  the server.

MemTotal:       32880880 kB
MemFree:        29070960 kB

Select allOpen in new window

It should not be possible to eat up 29 GB of main memory with Apache on a normal system unless the system is serving hundreds of concurrent requests, has a memory leak, or (this would be my guess) an amazing number of children are being spawned by a single-threaded MPM.

Of course, nothing says that Apache is the guilty party eating up all the memory.  It's entirely possible that something else is ... but without evidence we must speculate.  Running a resource monitor on the system would be immensely helpful as that would point directly at the guilty party, or parties.

What multi-processing model of Apache is being used, and what limits have been placed on the number of children?

If you don't know what MPM is in use, find where httpd is located, and then invoke it with -V as seen below:

 

/usr/local/apache2/bin/httpd -V
Server version: Apache/2.2.34 (Unix)
Server built:   Aug  8 2018 23:18:20
Server's Module Magic Number: xxxxxxxx:43
Server loaded:  APR 1.5.2, APR-Util 1.5.4
Compiled using: APR 1.5.2, APR-Util 1.5.4
Architecture:   32-bit
Server MPM:     Worker
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/worker"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/usr/local/apache2"
 -D SUEXEC_BIN="/usr/local/apache2/bin/suexec"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

Select allOpen in new window

thanks for the reply , please find the below,

 httpd -V
Server version: Apache/2.2.15 (Unix)
Server built:   Oct 16 2014 14:48:21
Server's Module Magic Number: 20051115:25
Server loaded:  APR 1.3.9, APR-Util 1.3.9
Compiled using: APR 1.3.9, APR-Util 1.3.9
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/etc/httpd"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="run/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

As a first step, I suggest coming forward to 2.2.34, the final release.  Being 19 releases out of date has numerous effects and some of them influence the security of the server.

Move the present source kit to a safe location, bring in the 2.2.34 kit, build using the saved config file from the 2.2.19 source directory, then install 2.2.34 and see if the problem is alleviated.  If so, fine and good, problem solved.  If not, you know it is not a memory leak that was seen and fixed before the last release, the security holes that were fixed have been fixed, and you can then concentrate on other possibilities.

Side notes:
a) I suggest the worker MPM, but as you are using prefork now stick with that and save shifting to a different MPM until it is clear that the problem is alleviated or not in 2.2.34.
b) The fact that this prefork server is not threaded does tend to support the idea of way too many processes eating up memory.

Apache is memory hungry anyway..., i went to use lighttpd / nginx because of this. A system could definitely handle more requests using that (at least double the amount of connections).
(all modules are available in all processes).   With lighttpd / nginx the PHP sessions can be tuned to need (php-fpm) and you can connect to the right php backend from them. Same holds for other fastcgi serving servers.

please find the below config of httpd.conf, would be helpful on suggestions on below any values to be changed


# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers         4
MaxClients         300
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  0
</IfModule>

As an aside, it will be helpful if you can tell us the approximate number of simultaneous connections, the number of sessions per hour, and what is running other than unmodified Apache.  That includes MTAs, DHCP service, Samba, Avahi, Tomcat, SQL and Apache modules.

Well, ... yeah.  With a limit of 256 simultaneous servers, if they are doing anything other than serving up vanilla web pages, there could certainly be issues.

I still recommend coming forward to 2.2.34 first.  Then see what happens.  If the problem is not alleviated, then consider going to the worker MPM instead.

Are there "process limit reached" messages in the logs?

thanks for the reply, the total users of the application is 145 logins and I think simultaneous connections we can think about between 75 to 80 users , also this server has windows share, csv files are stored the windows share that is mounted to this linux server and web application reads it and pushes into web application and it gets executed as per the logic build, also this server does not have database instead it is pointed to another server for database, in the previous post I see both settings preform MPM and also worker MPM should I have to comment the worker MPM values.

The worker values are conditionalized on the MPM variable (IfModule).  Still, IMO having settings for both in the same file is confusing and might lead someone to assume that the server has been tested with, and should work with, either MPM.

yes "process limit reached" messages in the logs, have seen this, also if we perform yum update it updates all which I do not want to, also I do not want to update to 2.4, how to update from 2.2.14 to 2.2.34 , can you help me with this please.

thanks for the reply, shall I comment worker MPM values, also can you please help me the step to upgrade Apache/2.2.15 to 2.2.34 , disable repo and adding 2.4 repo then remove 2.2 and install 2.4 iam aware but how to upgrade from 2.2.15 to 2.2.34.

Download the V2.2.34 kit from the Apache web site

https://archive.apache.org/dist/httpd/

httpd-2.2.34.tar.bz2

Confirm the hashes after download.  Then proceed as in my earlier comment.

This assumes that you are running a built version of Apache where you still have the config files and not a distributed version that came with whatever linux is in use on the host.  If you are running a distributed version without saved config files there will be a very steep learning curve.  You'll have to figure out what options you want in the server, and some of that will be trial and error.  Best to have your cold-spare copy of the server hardware readied with a copy of the current server to do trials on.

If all of this is Greek, then it may be best to hire someone to do it for you, preferably on-site so that you can both answer and ask questions.

can you please help me to understand if the following values are okay before going upgrading to the 2.2.34 , total physical RAM 64G and I can see from the log  between 15 to 20 httpd process and the highest memory consumption process observed is 9.5M, simultaneous users using application between 75 to 80 this is at higher side, please help on the following values is this okay or please change it to the best value please.

I have set  prefork MPM settings as following
<IfModule prefork.c>
StartServers       4
MinSpareServers    20
MaxSpareServers    40
# ServerLimit      256
MaxClients        200
MaxRequestsPerChild  4500

See the commentary on MaxRequestsPerChild at the link below.  Everything else looks OK to me.

https://serverfault.com/questions/415879/apache-prefork-optimization-choosing-the-right-maxrequestsperchild-value

When the OOM Killer runs, your system is in serious trouble, as you've completely exhausted both RAM + Swap space.

Unlikely increasing Swap space will fix your problem.

Doubtful updating Apache will fix your problem. To clarify what Dr. Klahn mentioned, there's simply no way you can generate enough traffic to raw Apache (no database + no PHP) to take down a 32G server. At least no way I can imagine.

More likely you have database resource starvation + slow running PHP processes, which will aggravate database resource usage by leaving connections open for a very long time.

To fix your problem, first run mysqltuner. If you get back message saying something like you require 64G of memory, then you'll have to retool your App or just add more memory, as Rahul suggested.

My suggestions.

Increase your memory to 128G-256G (memory is super cheap).

Upgrade to latest Apache-2.4, as Apache-2.2 is no longer supported (.34 was last releases).

Also update to use FPM-PHP which will help massively.

Also upgrade to MariaDB, which tends to use memory more efficiently than MySQL... especially whatever old version of MySQL you might be running with Apache-2.2 running.

Start with mysqltuner... For PHP... with FPM, PHP is easy to debug, because you can get an execution time + memory usage related to every PHP file that executes. I'm unsure if using libapache2-mod-php you can get this level of detail. If you can setup libapache2-mod-php to log this type of data, this will also allow you to possibly pinpoint the problem.

thank you all for such great inputs, can you please provide me steps or link on how to configure FPM-PHP think should install fpm php package then how to configure it to php/apache please