Diagnose: [core:notice] [pid xxx] AH00052: child pid xxx exit signal Segmentation fault (11)

Yeah, there are countless results because it's an extremely generic problem.

A segfault is the equivalent of one person trying to stand in the exact same spot as another person at the same time. Physics tells us that two different people cannot occupy the same space at the same time, so the second person throws his hands into the air in frustrated failure and screams, "I CANNOT DO THIS!"

That's what's happening in the computing world, but in more geeky terms. Basically, you have this big, open space called memory that is used to store different bits of data, and all applications have to share it nicely like a bunch of kids with OCD. To this end, each application usually grabs chunks of this open area for itself (a term called "memory allocation"), and then everyone plays within their own allocated space.

A segfault happens when one application tries to access a part of the memory that is already used / allocated by another application. If this were allowed to happen, terrible things could occur. For example, application B could change data that application A depends on, and application A would go crazy trying to figure out why their data doesn't look right. So whenever this unauthorized/unexpected access occurs, the system throws a segfault error and intentionally crashes things to prevent worse things from happening.

That all said, I re-iterate that segfaults are extremely generic - it can theoretically happen with any two applications, which is why you see a bazillion different responses on the topic, since there are a bazillion different programs. PHP itself has lots of built-in extensions, and each extension is sort of like its own mini-application, so the problem could be a bad extension, or the use of the wrong library, blah blah blah.

Now, the first step to fixing the problem is figuring out the root cause. To do that, you need to start by observing the behavior of the web server and try to find patterns that describe WHEN the problem happens:
1. Does it happen on every web hit, like when a browser asks Apache for an image or a Javascript file?
2. Does it happen only on PHP scripts?
3. Does it happen on EVERY PHP script all the time?
4. Does it happen on only certain PHP scripts all the time?
5. Does it happen only happen to random PHP scripts at seemingly random times (e.g. x.php works fine 9 times out of 10, but every 10th time, it fails, etc) ?
6. Does it happen within certain circumstances (e.g. only when user X is logged in and running some report or whatever)?
7. Does it happen at certain time intervals, like at the beginning of every hour, or maybe once every midnight?
8. Does it only happen when you're running a PHP script that processes a large amount of data?
9. Does it only happen when you're running a PHP script that uses a particular extension?
...etc...

There is almost always some kind of pattern to segfaults if you look closely enough.

Other questions to answer:
1. Are Apache and PHP both compiled as 64-bit, or are they both 32-bit, or are they mixed?
2. How did you obtain Apache and/or PHP - did you compile from source, or get them via yum (and if so, did you use any extra repositories), or did you copy the existing binaries from another server, or something else?

Interesting read, well explained but it leaves as many questions as when I started, which is of course, as you say, generic.

Still, no, I've not seen patterns yet.
No, I did not compile anything, I installed using yum and everything is 64bit.

Looking at the access logs, I can't find connections which are at the exact same time but there are a few which are very close and all of them are wordpress related, xmlrpc.php and wp-login.php accesses.

One VERY odd thing is that the logs seem to have rotated at around the exact same time and most interesting is that looking in the previous log shows nothing what so ever for anything on or around the same times.

All of the previous logs end at April 3rd. Maybe the problem is related to the rotation of the logs or something then.

Still looking.

you could enable coredumps on the system, then when this happens a dump of the program memory is done into a file. this can then be examined to try and identify the program

So far, finding information on how to actually enable CoreDumpDirectory is a joke. I've found countless posts and articles and not one thing works. No matter what I do, I cannot enable it on centos7.

If you know for sure how to enable this on Centos 7 using apache 2.4 and php 70w, I'd love to know it.

well apache & php are not relevant, this is pure OS stuff. This seems a straight-forward description

http://wangpidong.blogspot.co.uk/2016/06/enable-core-dumps-for-systemd-services.html

Isn't this an apache problem since the error is showing in the apache error.log.

In the log;
[Tue Apr 11 09:02:50.797373 2017] [core:notice] [pid 7175] AH00052: child pid 8531 exit signal Segmentation fault (11)

Looking at the process mentioned;
root      7175  0.0  0.1 1036492 40332 ?       Ss   Apr08   2:07 /usr/sbin/httpd -DFOREGROUND

but the corefile setting is an OS one, not apache specific. But then the point of the core dump was to identify the process crashing which you have now done. Now you need to work out why apache is crashing, you could still aim for a coredump if you have experience with source code and gdb or sdb, if not you have to turn up the debug level for apache and work out what it is doing at the time it crashes

Still a bit confused about what you're saying since the CoreDumpDirectory directive is an apache one. Everything I've read is specifically about setting up a core dump for apache, not the OS.

Even though I can see the process, that doesn't tell me anything what so ever. I need to enable more logging to find the source of the problem. Once I can get a core dump, then I can use gbd to try and find the problem. I've never seen any mention of having to go through any source code.

gdb of the curedump will show you the internal of the code

ok, you are trying to tell apache specifically to coredump within apache, I'm telling everything on the machine to coredump if necessary

Yes, I'm not asking or wanting to coredump the entire machine, I know it's only an apache problem. It's all I want to core dump.

Ehh, maybe I'm missing something about the core dump recommendation, but usually core dumps aren't that helpful without compiling Apache with symbol info and being able to reproduce the segfault it in a single process instead of in a child worker (usually done with httpd -X).

In all honesty, I'm going to guess that php70w just has some kind of incompatibility with your Apache setup or is an unstable build in some way. I don't know too much about Webtatic, but if PHP is your primary language/engine for your scripts, I'd recommend just downloading and compiling the latest PHP 7 code with PHP-FPM and running that in a separate process, and once it's compiled and running, just tell Apache to handle PHP files by sending them to your FastCGI process and comment out the loading of the php70w module.

Running your PHP scripts through FastCGI should give you much better performance than mod_php anyway, and it makes it a LOT easier to maintain and upgrade (in my opinion).

Basically, the process is:
1. Get into the shell.

2. Download the latest stable source:
wget -O php713.tar.gz http://php.net/get/php-7.1.3.tar.gz/from/this/mirror

3. Extract the source:
tar -zxvf php713.tar.gz

4. Go into the source folder:
cd php-7.1.3

5. Run configure:
./configure --prefix=/usr/local/php-7.1.3 --with-openssl --with-curl --enable-fpm --with-mysqli

6. Compile:
make

7. If it all compiles fine, optionally run make test (you might have a few minor tests fail due to known bugs, etc):
make test

8. Run the make install to copy everything into /usr/local/php-7.1.3
make install

9. Copy the FPM init.d script into place and set it up to start at boot:
cp sapi/fpm/init.d.php-fpm /etc/init.d/php-fpm
chmod 755 /etc/init.d/php-fpm
chkconfig php-fpm on

10. Create a new file called /usr/local/php-7.1.3/etc/php-fpm.conf and set it up with something like this:

[global]
error_log = /var/log/php-fpm.log

[www]
user = daemon
group = daemon
listen = 127.0.0.1:9000
pm = dynamic
pm.max_children = 25
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 10

Select allOpen in new window

(Replace "daemon" values in user/group with whatever user/group you want PHP to run as)

11. Set up your php.ini file in /usr/local/php-7.1.3/lib/php.ini however you want (you might be able to just copy over your existing php.ini file)

12. Start the PHP-FPM process:
/etc/init.d/php-fpm start

Once all that's done, you should just need to tell Apache to hand off PHP requests to it, so edit the Apache config and comment out existing references to PHP being loaded as a module, and then somewhere near the end of your Apache config, add:

TimeOut 300
ProxyTimeout 300

<FilesMatch \.php$>
 SetHandler "proxy:fcgi://127.0.0.1:9000"
</FilesMatch>

Select allOpen in new window

Replace the TimeOut and ProxyTimeout values with whatever time limits you want to enforce on PHP processes (e.g. 300 allows PHP and Apache processes to run up to a max of 5 minutes each, but that might be overkill for what you need).

Gracefully restart Apache and test out your PHP pages thoroughly, and then see if the problem goes away.

It also looks like there's an FPM package for php70w, too, but again, I'm not familiar with Webtatic so I can't tell you why you would want to use their package over the standard distribution. However, if you needed to stick with the Webtatic packages for some reason, you could always try out their FPM package to separate out the PHP to the FastCGI worker instead of Apache. In case there's any sort of incompatibility between the worker and the PHP module for Apache (mod_php) approach, the FastCGI would resolve that.

I certainly appreciate the input on going to a new version but that would be way too disruptive. We have too many things working in concert to start changing so many parts.

I just need to figure out how to generate a core dump using the apache method, anything else is overkill.