Apache server crashes randomly

My apache server crashes randomly with no apparent reason.

This is my configuration:

OS:
OpenSuSE 13.1 (Linux apache 3.11.10-29-default x86_64 GNU/Linux)
Apache:
apache2-2.4.6-6.47.1.x86_64
Apache Modules:

 core_module (static)
 access_compat_module (static)
 so_module (static)
 http_module (static)
 mpm_prefork_module (static)
 unixd_module (static)
 systemd_module (static)
 actions_module (shared)
 alias_module (shared)
 auth_basic_module (shared)
 authn_file_module (shared)
 authz_host_module (shared)
 authz_groupfile_module (shared)
 authz_user_module (shared)
 autoindex_module (shared)
 dir_module (shared)
 env_module (shared)
 expires_module (shared)
 include_module (shared)
 log_config_module (shared)
 mime_module (shared)
 negotiation_module (shared)
 setenvif_module (shared)
 ssl_module (shared)
 userdir_module (shared)
 reqtimeout_module (shared)
 authn_core_module (shared)
 authz_core_module (shared)
 php5_module (shared)

PHP:
5.4.20

Apache Error Log:
*** Error in `/usr/sbin/httpd2-prefork': double free or corruption (out): 0x00007fa17d8ea040 ***
LVL 6
Ludwig DiehlSystems ArchitectAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

gr8gonzoConsultantCommented:
How did you install Apache? Was it via an RPM or other package, or did you compile it from source?
arnoldCommented:
Double check your configuration/settings dealing with resource allocation.

Likely memory related, when it crashes it lacks the needed memory.

Enable core dump for Apache, then look at the dump for info...
Ludwig DiehlSystems ArchitectAuthor Commented:
I installed it via zypper (rpm) .
Active Protection takes the fight to cryptojacking

While there were several headline-grabbing ransomware attacks during in 2017, another big threat started appearing at the same time that didn’t get the same coverage – illicit cryptomining.

Ludwig DiehlSystems ArchitectAuthor Commented:
How can I know if it is related to memory?. How do I enable core dump?
junipllcCommented:
This certainly points to a bug in either apache or one of its modules. Basically, Apache is keeping a pointer around which points to a block of memory that has already been freed.

Are you running the latest versions of everything in the stack?

Mike
gheistCommented:
I would test hardware (zypper in memtest86, then reboot and let memtest run at least couple of passes)
Apache and php does not crash so bad in generic configuration.
arnoldCommented:
Add CoreDumpDirectory to httpd.conf with a directory to which the core should saved.

See
http://httpd.apache.org/dev/debugging.html
Ludwig DiehlSystems ArchitectAuthor Commented:
By the way I forgot to mention. It is intalled on Hyper-V with 32Gb RAM assigned and 16 virtual processors.
arnoldCommented:
Does it crash on a consistent frequency every 30 hours, etc.

Capture a core once, and see what lead to the crash.

Is your httpd.con spawns more than 5child process, or configured for too many compared to what your system can support?
gheistCommented:
Please tell hyper-V version at least.
openSUSE 13.1 is not supported by Hyper-V (only 12.3 is)
Ludwig DiehlSystems ArchitectAuthor Commented:
Hyper V Version  6.3.9600.16384
gheistCommented:
Still not supported. Wait for crash dump.
Have you tested server memory before installing windows?
Ludwig DiehlSystems ArchitectAuthor Commented:
Yes. I did, actually I've got 10 VM running without any prob.
gr8gonzoConsultantCommented:
1. Check your access and error logs and copy the entries that occur around the time of the crash. The error logs should contain entries showing Apache's startup messages, so it should be easy to see the timestamps of the crash. If the crash is related to any specific functionality (e.g. maybe a PHP request that uses a function from an extension that isn't working right), then it will help to establish what those requests are so that you have a reproducible test.

2. I would suggest compiling Apache and PHP from source, so everything is compiled against your own installed libraries. It doesn't take very long to do, and you can use the --prefix parameter during your configure to build and install everything to specific folders (so they don't overwrite any existing configuration or libraries).

3. Once you've compiled Apache and PHP, just set up their configuration to match the existing RPM-installed versions (same php.ini, same virtual hosts, same port, etc...), and then stop the existing Apache daemon and start the one for your compiled version so that the compiled one starts taking over the requests, then watch to see if it also crashes.

Personally, I'm not a fan of using RPMs for the most mission-critical pieces of functionality. There's always a tiny risk of compatibility issues, so I'll use them for some very common libraries that are always available in the right version and build, but I always compile Apache, PHP, and MySQL from source, and I've had no problems since I've started doing that.
gheistCommented:
I suggest to not compile anything from source, because that makes crash non-repeatable and drops you 10km out of any support options.
gr8gonzoConsultantCommented:
Not sure what you mean by making the crash non-repeatable. My very first suggestion was to try and find steps to make the crash repeatable - before switching to a different version. The idea is to verify that the same steps that crash on Build A do not crash on Build B (or if they do, then you now have more information on where to go next).

As far as support options go, maybe it's just me, but I haven't run across many support options that are dependent on the RPM -AND- are actually helpful. I tend to find it's either one or the other - either you get official support that doesn't really help very well (they are often less experienced admins who are trying to run through a canned script of options to try), or you get helpful support that doesn't really care too much about whether you compiled from the official source or installed from an RPM.

Not saying that it's impossible to find that scenario - just saying I haven't seen it yet.
gheistCommented:
Why dont you capture crash dump and upload to attachmate bugzilla? You dont have to do more.
Ludwig DiehlSystems ArchitectAuthor Commented:
I will try that and let you know
gheistCommented:
If you have repeatable problem they are very supportive
Ludwig DiehlSystems ArchitectAuthor Commented:
It is weird,so far it has not happened again...
gheistCommented:
I would say - take http:#a40964977 as answer and keep directory for cores around in case if it ever crashes again. As you might suspect without crash dump we or support cannot help with them.
gr8gonzoConsultantCommented:
I'd disagree - 40964977 isn't an answer - it's steps to determining more information. Keep this open until it happens again.
gheistCommented:
Sounds like eternity...
arnoldCommented:
This question once the person has the coredump includes the means by which to try to identify the cause.


Without precise timing when the crash occurs as well as data from snapshots of what was going on the system, process list, memory use, io use...
Ludwig DiehlSystems ArchitectAuthor Commented:
SO what I've done so far:

1.
mkdir /var/log/coredump

Open in new window

2.added to my apache configuration
CoreDumpDirectory /var/log/coredump

Open in new window


3.
sysctl -w fs.suid_dumpable=2

Open in new window

4.
sysctl -w kernel.core_pattern=/var/log/coredump

Open in new window

5.
ulimit -c unlimited

Open in new window


6.
systemctl stop apache2

Open in new window

7.
systemctl start apache2

Open in new window


Nothing happens yet...
gheistCommented:
You scared it...
Ludwig DiehlSystems ArchitectAuthor Commented:
Can you suggest anything to replicate it?. Now I am confused...
gr8gonzoConsultantCommented:
See my comment about checking the access logs. Typically, crashes ARE caused by some kind of repeatable steps, and in a web server, that tends to come from code that is executed from a module. So I'd say your first step is try to find access logs that indicate requests that occurred before the crashes, and try to trace those requests back (e.g. find the POST or GET data) so you can repeat them.
Ludwig DiehlSystems ArchitectAuthor Commented:
Thanks for that. I will search for those entries to see if something can be done
Ludwig DiehlSystems ArchitectAuthor Commented:
Finally I got this. I got this extract:

[Tue Oct 06 16:30:02.021240 2015] [ssl:warn] [pid 56957] AH01873: Init: Session Cache is not configured [hint: SSLSessionCache]
[Tue Oct 06 16:30:02.037402 2015] [core:warn] [pid 56957] AH00098: pid file /run/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Tue Oct 06 16:30:02.039236 2015] [mpm_prefork:notice] [pid 56957] AH00163: Apache/2.4.6 (Linux/SUSE) OpenSSL/1.0.1k PHP/5.4.20 configured -- resuming norma$
[Tue Oct 06 16:30:02.039272 2015] [core:notice] [pid 56957] AH00094: Command line: '/usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf'
[Tue Oct 06 16:30:07.016522 2015] [php5:error] [pid 56983] [client 190.237.6.236:49368] PHP Notice:  Undefined variable: mSource in /srv/www/htdocs/online/m$
[Tue Oct 06 16:30:07.016753 2015] [php5:error] [pid 56983] [client 190.237.6.236:49368] PHP Notice:  Undefined variable: nTotalPages in /srv/www/htdocs/comm$
[Tue Oct 06 16:30:07.016770 2015] [php5:error] [pid 56983] [client 190.237.6.236:49368] PHP Notice:  Undefined variable: mData in /srv/www/htdocs/common/cla$
[Tue Oct 06 16:30:07.043751 2015] [php5:error] [pid 56981] [client 190.237.6.236:49364] PHP Notice:  Undefined variable: mSource in /srv/www/htdocs/online/m$
[Tue Oct 06 16:30:07.043853 2015] [php5:error] [pid 56981] [client 190.237.6.236:49364] PHP Notice:  Undefined variable: nTotalPages in /srv/www/htdocs/comm$
[Tue Oct 06 16:30:07.043872 2015] [php5:error] [pid 56981] [client 190.237.6.236:49364] PHP Notice:  Undefined variable: mData in /srv/www/htdocs/common/cla$
[Tue Oct 06 16:30:07.088511 2015] [php5:error] [pid 56985] [client 190.237.6.236:49366] PHP Notice:  Undefined variable: aOrder in /srv/www/htdocs/online/mo$
[Tue Oct 06 16:30:07.303493 2015] [php5:error] [pid 56981] [client 190.237.6.236:49364] PHP Notice:  Undefined variable: mSource in /srv/www/htdocs/online/m$
[Tue Oct 06 16:30:07.303563 2015] [php5:error] [pid 56981] [client 190.237.6.236:49364] PHP Notice:  Undefined variable: nTotalPages in /srv/www/htdocs/comm$
[Tue Oct 06 16:30:07.303575 2015] [php5:error] [pid 56981] [client 190.237.6.236:49364] PHP Notice:  Undefined variable: mData in /srv/www/htdocs/common/cla$
[Tue Oct 06 16:30:07.338158 2015] [php5:error] [pid 56985] [client 190.237.6.236:49366] PHP Notice:  Undefined variable: nCoordGral in /srv/www/htdocs/onlin$
[Tue Oct 06 16:30:07.338243 2015] [php5:error] [pid 56985] [client 190.237.6.236:49366] PHP Notice:  Undefined variable: nCoordGral in /srv/www/htdocs/onlin$
[Tue Oct 06 16:30:07.825327 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825368 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825382 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825390 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825401 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825409 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825421 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825429 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825439 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825447 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825457 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825464 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825489 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825497 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825507 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825515 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.844569 2015] [php5:error] [pid 56984] [client 190.237.6.236:49365] PHP Notice:  Undefined variable: nCoordGral in /srv/www/htdocs/onlin$
[Tue Oct 06 16:30:07.844693 2015] [php5:error] [pid 56984] [client 190.237.6.236:49365] PHP Notice:  Undefined variable: nCoordGral in /srv/www/htdocs/onlin$
[Tue Oct 06 16:30:07.844721 2015] [php5:error] [pid 56984] [client 190.237.6.236:49365] PHP Notice:  Undefined variable: nCoordGral in /srv/www/htdocs/onlin$
[Tue Oct 06 16:30:07.912946 2015] [php5:error] [pid 56982] [client 190.237.6.236:49367] PHP Notice:  Undefined index: id_rol_personal in /srv/www/htdocs/onl$
*** Error in `/usr/sbin/httpd2-prefork': double free or corruption (out): 0x00007f48a1c2f040 ***
[Tue Oct 06 16:30:08.178012 2015] [php5:error] [pid 56988] [client 190.216.175.10:6400] PHP Warning:  Creating default object from empty value in /srv/www/h$
[Tue Oct 06 16:30:08.178057 2015] [php5:error] [pid 56988] [client 190.216.175.10:6400] PHP Notice:  Undefined index: paginas in /srv/www/htdocs/online/modu$
[Tue Oct 06 16:30:08.178073 2015] [php5:error] [pid 56988] [client 190.216.175.10:6400] PHP Notice:  Undefined index: nrofilas in /srv/www/htdocs/online/mod$
[Tue Oct 06 16:30:09.124175 2015] [core:notice] [pid 56957] AH00052: child pid 56992 exit signal Aborted (6)
*** Error in `/usr/sbin/httpd2-prefork': double free or corruption (out): 0x00007f48a1c2f040 ***
[Tue Oct 06 16:30:09.844199 2015] [php5:error] [pid 56958] [client 190.234.57.98:59926] PHP Notice:  Undefined variable: data2 in /srv/www/htdocs/online/mod$
[Tue Oct 06 16:30:09.884062 2015] [php5:error] [pid 56958] [client 190.234.57.98:59926] PHP Notice:  Undefined variable: data2 in /srv/www/htdocs/online/mod$
[Tue Oct 06 16:30:10.136282 2015] [core:notice] [pid 56957] AH00052: child pid 56996 exit signal Aborted (6)
*** Error in `/usr/sbin/httpd2-prefork': double free or corruption (out): 0x00007f48a1c2f040 ***

Open in new window



Any idea?
arnoldCommented:
Do you have the core dump file?
gr8gonzoConsultantCommented:
I see another PHP bug that complains about the OpenSUSE package for PHP causing the same problems, although that bug is listed for PHP version 5.2.5 on SUSE 10.3. Still, the same error is being reported:

/usr/sbin/httpd2-prefork: double free or corruption (out): 0x00007fff56094780

I would again suggest just compiling PHP from source. It's not difficult to do, and given that SUSE's official packages seem to have this problem, it seems like a reasonable way to avoid the problem.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
gheistCommented:
The address is fixed. It means that system memory is bad and you need to run memtest. There is no bug at all.
Ludwig DiehlSystems ArchitectAuthor Commented:
That's weird. I scanned it using memtest and no error was found.
gheistCommented:
Maybe you have some httpd hung on shutdown
Thry "service apache2 stop" (check with ps no httpds left) and then start it back up.
Ludwig DiehlSystems ArchitectAuthor Commented:
It has not happened again ever since.
gheistCommented:
I think 3 apache updates later it might be silently fixed and you can close the ticket accepting your own finding that it suddenly works well, or give some small points where you feel it could help diagnostics.
Ludwig DiehlSystems ArchitectAuthor Commented:
Thanks everyone for ur assistance.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Apache Web Server

From novice to tech pro — start learning today.