Apache server crashes randomly

How did you install Apache? Was it via an RPM or other package, or did you compile it from source?

Double check your configuration/settings dealing with resource allocation.

Likely memory related, when it crashes it lacks the needed memory.

Enable core dump for Apache, then look at the dump for info...

I installed it via zypper (rpm) .

How can I know if it is related to memory?. How do I enable core dump?

This certainly points to a bug in either apache or one of its modules. Basically, Apache is keeping a pointer around which points to a block of memory that has already been freed.

Are you running the latest versions of everything in the stack?

Mike

I would test hardware (zypper in memtest86, then reboot and let memtest run at least couple of passes)
Apache and php does not crash so bad in generic configuration.

Add CoreDumpDirectory to httpd.conf with a directory to which the core should saved.

See
http://httpd.apache.org/dev/debugging.html

By the way I forgot to mention. It is intalled on Hyper-V with 32Gb RAM assigned and 16 virtual processors.

Does it crash on a consistent frequency every 30 hours, etc.

Capture a core once, and see what lead to the crash.

Is your httpd.con spawns more than 5child process, or configured for too many compared to what your system can support?

Hyper V Version  6.3.9600.16384

Still not supported. Wait for crash dump.
Have you tested server memory before installing windows?

Yes. I did, actually I've got 10 VM running without any prob.

1. Check your access and error logs and copy the entries that occur around the time of the crash. The error logs should contain entries showing Apache's startup messages, so it should be easy to see the timestamps of the crash. If the crash is related to any specific functionality (e.g. maybe a PHP request that uses a function from an extension that isn't working right), then it will help to establish what those requests are so that you have a reproducible test.

2. I would suggest compiling Apache and PHP from source, so everything is compiled against your own installed libraries. It doesn't take very long to do, and you can use the --prefix parameter during your configure to build and install everything to specific folders (so they don't overwrite any existing configuration or libraries).

3. Once you've compiled Apache and PHP, just set up their configuration to match the existing RPM-installed versions (same php.ini, same virtual hosts, same port, etc...), and then stop the existing Apache daemon and start the one for your compiled version so that the compiled one starts taking over the requests, then watch to see if it also crashes.

Personally, I'm not a fan of using RPMs for the most mission-critical pieces of functionality. There's always a tiny risk of compatibility issues, so I'll use them for some very common libraries that are always available in the right version and build, but I always compile Apache, PHP, and MySQL from source, and I've had no problems since I've started doing that.

I suggest to not compile anything from source, because that makes crash non-repeatable and drops you 10km out of any support options.

Not sure what you mean by making the crash non-repeatable. My very first suggestion was to try and find steps to make the crash repeatable - before switching to a different version. The idea is to verify that the same steps that crash on Build A do not crash on Build B (or if they do, then you now have more information on where to go next).

As far as support options go, maybe it's just me, but I haven't run across many support options that are dependent on the RPM -AND- are actually helpful. I tend to find it's either one or the other - either you get official support that doesn't really help very well (they are often less experienced admins who are trying to run through a canned script of options to try), or you get helpful support that doesn't really care too much about whether you compiled from the official source or installed from an RPM.

Not saying that it's impossible to find that scenario - just saying I haven't seen it yet.

Why dont you capture crash dump and upload to attachmate bugzilla? You dont have to do more.

I will try that and let you know

If you have repeatable problem they are very supportive

It is weird,so far it has not happened again...

I would say - take http:#a40964977 as answer and keep directory for cores around in case if it ever crashes again. As you might suspect without crash dump we or support cannot help with them.

I'd disagree - 40964977 isn't an answer - it's steps to determining more information. Keep this open until it happens again.

Sounds like eternity...

SO what I've done so far:

1.

mkdir /var/log/coredump

Select allOpen in new window

2.added to my apache configuration

CoreDumpDirectory /var/log/coredump

Select allOpen in new window

3.

sysctl -w fs.suid_dumpable=2

Select allOpen in new window

4.

sysctl -w kernel.core_pattern=/var/log/coredump

Select allOpen in new window

5.

ulimit -c unlimited

Select allOpen in new window

6.

systemctl stop apache2

Select allOpen in new window

7.

systemctl start apache2

Select allOpen in new window

Nothing happens yet...

You scared it...

Can you suggest anything to replicate it?. Now I am confused...

See my comment about checking the access logs. Typically, crashes ARE caused by some kind of repeatable steps, and in a web server, that tends to come from code that is executed from a module. So I'd say your first step is try to find access logs that indicate requests that occurred before the crashes, and try to trace those requests back (e.g. find the POST or GET data) so you can repeat them.

Thanks for that. I will search for those entries to see if something can be done

Finally I got this. I got this extract:

[Tue Oct 06 16:30:02.021240 2015] [ssl:warn] [pid 56957] AH01873: Init: Session Cache is not configured [hint: SSLSessionCache]
[Tue Oct 06 16:30:02.037402 2015] [core:warn] [pid 56957] AH00098: pid file /run/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Tue Oct 06 16:30:02.039236 2015] [mpm_prefork:notice] [pid 56957] AH00163: Apache/2.4.6 (Linux/SUSE) OpenSSL/1.0.1k PHP/5.4.20 configured -- resuming norma$
[Tue Oct 06 16:30:02.039272 2015] [core:notice] [pid 56957] AH00094: Command line: '/usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf'
[Tue Oct 06 16:30:07.016522 2015] [php5:error] [pid 56983] [client 190.237.6.236:49368] PHP Notice:  Undefined variable: mSource in /srv/www/htdocs/online/m$
[Tue Oct 06 16:30:07.016753 2015] [php5:error] [pid 56983] [client 190.237.6.236:49368] PHP Notice:  Undefined variable: nTotalPages in /srv/www/htdocs/comm$
[Tue Oct 06 16:30:07.016770 2015] [php5:error] [pid 56983] [client 190.237.6.236:49368] PHP Notice:  Undefined variable: mData in /srv/www/htdocs/common/cla$
[Tue Oct 06 16:30:07.043751 2015] [php5:error] [pid 56981] [client 190.237.6.236:49364] PHP Notice:  Undefined variable: mSource in /srv/www/htdocs/online/m$
[Tue Oct 06 16:30:07.043853 2015] [php5:error] [pid 56981] [client 190.237.6.236:49364] PHP Notice:  Undefined variable: nTotalPages in /srv/www/htdocs/comm$
[Tue Oct 06 16:30:07.043872 2015] [php5:error] [pid 56981] [client 190.237.6.236:49364] PHP Notice:  Undefined variable: mData in /srv/www/htdocs/common/cla$
[Tue Oct 06 16:30:07.088511 2015] [php5:error] [pid 56985] [client 190.237.6.236:49366] PHP Notice:  Undefined variable: aOrder in /srv/www/htdocs/online/mo$
[Tue Oct 06 16:30:07.303493 2015] [php5:error] [pid 56981] [client 190.237.6.236:49364] PHP Notice:  Undefined variable: mSource in /srv/www/htdocs/online/m$
[Tue Oct 06 16:30:07.303563 2015] [php5:error] [pid 56981] [client 190.237.6.236:49364] PHP Notice:  Undefined variable: nTotalPages in /srv/www/htdocs/comm$
[Tue Oct 06 16:30:07.303575 2015] [php5:error] [pid 56981] [client 190.237.6.236:49364] PHP Notice:  Undefined variable: mData in /srv/www/htdocs/common/cla$
[Tue Oct 06 16:30:07.338158 2015] [php5:error] [pid 56985] [client 190.237.6.236:49366] PHP Notice:  Undefined variable: nCoordGral in /srv/www/htdocs/onlin$
[Tue Oct 06 16:30:07.338243 2015] [php5:error] [pid 56985] [client 190.237.6.236:49366] PHP Notice:  Undefined variable: nCoordGral in /srv/www/htdocs/onlin$
[Tue Oct 06 16:30:07.825327 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825368 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825382 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825390 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825401 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825409 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825421 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825429 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825439 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825447 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825457 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825464 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825489 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825497 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825507 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.825515 2015] [php5:error] [pid 56979] [client 190.237.6.236:49363] PHP Warning:  Division by zero in /srv/www/htdocs/online/modules/onl$
[Tue Oct 06 16:30:07.844569 2015] [php5:error] [pid 56984] [client 190.237.6.236:49365] PHP Notice:  Undefined variable: nCoordGral in /srv/www/htdocs/onlin$
[Tue Oct 06 16:30:07.844693 2015] [php5:error] [pid 56984] [client 190.237.6.236:49365] PHP Notice:  Undefined variable: nCoordGral in /srv/www/htdocs/onlin$
[Tue Oct 06 16:30:07.844721 2015] [php5:error] [pid 56984] [client 190.237.6.236:49365] PHP Notice:  Undefined variable: nCoordGral in /srv/www/htdocs/onlin$
[Tue Oct 06 16:30:07.912946 2015] [php5:error] [pid 56982] [client 190.237.6.236:49367] PHP Notice:  Undefined index: id_rol_personal in /srv/www/htdocs/onl$
*** Error in `/usr/sbin/httpd2-prefork': double free or corruption (out): 0x00007f48a1c2f040 ***
[Tue Oct 06 16:30:08.178012 2015] [php5:error] [pid 56988] [client 190.216.175.10:6400] PHP Warning:  Creating default object from empty value in /srv/www/h$
[Tue Oct 06 16:30:08.178057 2015] [php5:error] [pid 56988] [client 190.216.175.10:6400] PHP Notice:  Undefined index: paginas in /srv/www/htdocs/online/modu$
[Tue Oct 06 16:30:08.178073 2015] [php5:error] [pid 56988] [client 190.216.175.10:6400] PHP Notice:  Undefined index: nrofilas in /srv/www/htdocs/online/mod$
[Tue Oct 06 16:30:09.124175 2015] [core:notice] [pid 56957] AH00052: child pid 56992 exit signal Aborted (6)
*** Error in `/usr/sbin/httpd2-prefork': double free or corruption (out): 0x00007f48a1c2f040 ***
[Tue Oct 06 16:30:09.844199 2015] [php5:error] [pid 56958] [client 190.234.57.98:59926] PHP Notice:  Undefined variable: data2 in /srv/www/htdocs/online/mod$
[Tue Oct 06 16:30:09.884062 2015] [php5:error] [pid 56958] [client 190.234.57.98:59926] PHP Notice:  Undefined variable: data2 in /srv/www/htdocs/online/mod$
[Tue Oct 06 16:30:10.136282 2015] [core:notice] [pid 56957] AH00052: child pid 56996 exit signal Aborted (6)
*** Error in `/usr/sbin/httpd2-prefork': double free or corruption (out): 0x00007f48a1c2f040 ***

Select allOpen in new window

Any idea?

Do you have the core dump file?

The address is fixed. It means that system memory is bad and you need to run memtest. There is no bug at all.

That's weird. I scanned it using memtest and no error was found.

It has not happened again ever since.

I think 3 apache updates later it might be silently fixed and you can close the ticket accepting your own finding that it suddenly works well, or give some small points where you feel it could help diagnostics.

Thanks everyone for ur assistance.