Diagnose: [core:notice] [pid xxx] AH00052: child pid xxx exit signal Segmentation fault (11)

I've searched and tried a number of things. There are countless results when searching for this problem but I've yet to solve it.

PLEASE, do not send me any links to more reading. I am looking for hands on help to find and fix this problem once and for all.

The server is Centos 7 running php 70w and apache 2.4.
LVL 1
Mark LewisAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

gr8gonzoConsultantCommented:
Yeah, there are countless results because it's an extremely generic problem.

A segfault is the equivalent of one person trying to stand in the exact same spot as another person at the same time. Physics tells us that two different people cannot occupy the same space at the same time, so the second person throws his hands into the air in frustrated failure and screams, "I CANNOT DO THIS!"

That's what's happening in the computing world, but in more geeky terms. Basically, you have this big, open space called memory that is used to store different bits of data, and all applications have to share it nicely like a bunch of kids with OCD. To this end, each application usually grabs chunks of this open area for itself (a term called "memory allocation"), and then everyone plays within their own allocated space.

A segfault happens when one application tries to access a part of the memory that is already used / allocated by another application. If this were allowed to happen, terrible things could occur. For example, application B could change data that application A depends on, and application A would go crazy trying to figure out why their data doesn't look right. So whenever this unauthorized/unexpected access occurs, the system throws a segfault error and intentionally crashes things to prevent worse things from happening.

That all said, I re-iterate that segfaults are extremely generic - it can theoretically happen with any two applications, which is why you see a bazillion different responses on the topic, since there are a bazillion different programs. PHP itself has lots of built-in extensions, and each extension is sort of like its own mini-application, so the problem could be a bad extension, or the use of the wrong library, blah blah blah.

Now, the first step to fixing the problem is figuring out the root cause. To do that, you need to start by observing the behavior of the web server and try to find patterns that describe WHEN the problem happens:
1. Does it happen on every web hit, like when a browser asks Apache for an image or a Javascript file?
2. Does it happen only on PHP scripts?
3. Does it happen on EVERY PHP script all the time?
4. Does it happen on only certain PHP scripts all the time?
5. Does it happen only happen to random PHP scripts at seemingly random times (e.g. x.php works fine 9 times out of 10, but every 10th time, it fails, etc) ?
6. Does it happen within certain circumstances (e.g. only when user X is logged in and running some report or whatever)?
7. Does it happen at certain time intervals, like at the beginning of every hour, or maybe once every midnight?
8. Does it only happen when you're running a PHP script that processes a large amount of data?
9. Does it only happen when you're running a PHP script that uses a particular extension?
...etc...

There is almost always some kind of pattern to segfaults if you look closely enough.

Other questions to answer:
1. Are Apache and PHP both compiled as 64-bit, or are they both 32-bit, or are they mixed?
2. How did you obtain Apache and/or PHP - did you compile from source, or get them via yum (and if so, did you use any extra repositories), or did you copy the existing binaries from another server, or something else?
0
Mark LewisAuthor Commented:
Interesting read, well explained but it leaves as many questions as when I started, which is of course, as you say, generic.

Still, no, I've not seen patterns yet.
No, I did not compile anything, I installed using yum and everything is 64bit.

Looking at the access logs, I can't find connections which are at the exact same time but there are a few which are very close and all of them are wordpress related, xmlrpc.php and wp-login.php accesses.

One VERY odd thing is that the logs seem to have rotated at around the exact same time and most interesting is that looking in the previous log shows nothing what so ever for anything on or around the same times.

All of the previous logs end at April 3rd. Maybe the problem is related to the rotation of the logs or something then.

Still looking.
0
Nick UpsonPrincipal Operations EngineerCommented:
you could enable coredumps on the system, then when this happens a dump of the program memory is done into a file. this can then be examined to try and identify the program
0
IT Pros Agree: AI and Machine Learning Key

We’d all like to think our company’s data is well protected, but when you ask IT professionals they admit the data probably is not as safe as it could be.

Mark LewisAuthor Commented:
So far, finding information on how to actually enable CoreDumpDirectory is a joke. I've found countless posts and articles and not one thing works. No matter what I do, I cannot enable it on centos7.

If you know for sure how to enable this on Centos 7 using apache 2.4 and php 70w, I'd love to know it.
0
Nick UpsonPrincipal Operations EngineerCommented:
well apache & php are not relevant, this is pure OS stuff. This seems a straight-forward description

http://wangpidong.blogspot.co.uk/2016/06/enable-core-dumps-for-systemd-services.html
0
Mark LewisAuthor Commented:
Isn't this an apache problem since the error is showing in the apache error.log.

In the log;
[Tue Apr 11 09:02:50.797373 2017] [core:notice] [pid 7175] AH00052: child pid 8531 exit signal Segmentation fault (11)

Looking at the process mentioned;
root      7175  0.0  0.1 1036492 40332 ?       Ss   Apr08   2:07 /usr/sbin/httpd -DFOREGROUND
0
Nick UpsonPrincipal Operations EngineerCommented:
but the corefile setting is an OS one, not apache specific. But then the point of the core dump was to identify the process crashing which you have now done. Now you need to work out why apache is crashing, you could still aim for a coredump if you have experience with source code and gdb or sdb, if not you have to turn up the debug level for apache and work out what it is doing at the time it crashes
0
Mark LewisAuthor Commented:
Still a bit confused about what you're saying since the CoreDumpDirectory directive is an apache one. Everything I've read is specifically about setting up a core dump for apache, not the OS.

Even though I can see the process, that doesn't tell me anything what so ever. I need to enable more logging to find the source of the problem. Once I can get a core dump, then I can use gbd to try and find the problem. I've never seen any mention of having to go through any source code.
0
Nick UpsonPrincipal Operations EngineerCommented:
gdb of the curedump will show you the internal of the code

ok, you are trying to tell apache specifically to coredump within apache, I'm telling everything on the machine to coredump if necessary
0
Mark LewisAuthor Commented:
Yes, I'm not asking or wanting to coredump the entire machine, I know it's only an apache problem. It's all I want to core dump.
0
gr8gonzoConsultantCommented:
Ehh, maybe I'm missing something about the core dump recommendation, but usually core dumps aren't that helpful without compiling Apache with symbol info and being able to reproduce the segfault it in a single process instead of in a child worker (usually done with httpd -X).

In all honesty, I'm going to guess that php70w just has some kind of incompatibility with your Apache setup or is an unstable build in some way. I don't know too much about Webtatic, but if PHP is your primary language/engine for your scripts, I'd recommend just downloading and compiling the latest PHP 7 code with PHP-FPM and running that in a separate process, and once it's compiled and running, just tell Apache to handle PHP files by sending them to your FastCGI process and comment out the loading of the php70w module.

Running your PHP scripts through FastCGI should give you much better performance than mod_php anyway, and it makes it a LOT easier to maintain and upgrade (in my opinion).

Basically, the process is:
1. Get into the shell.

2. Download the latest stable source:
wget -O php713.tar.gz http://php.net/get/php-7.1.3.tar.gz/from/this/mirror

3. Extract the source:
tar -zxvf php713.tar.gz

4. Go into the source folder:
cd php-7.1.3

5. Run configure:
./configure --prefix=/usr/local/php-7.1.3 --with-openssl --with-curl --enable-fpm --with-mysqli

6. Compile:
make

7. If it all compiles fine, optionally run make test (you might have a few minor tests fail due to known bugs, etc):
make test

8. Run the make install to copy everything into /usr/local/php-7.1.3
make install

9. Copy the FPM init.d script into place and set it up to start at boot:
cp sapi/fpm/init.d.php-fpm /etc/init.d/php-fpm
chmod 755 /etc/init.d/php-fpm
chkconfig php-fpm on

10. Create a new file called /usr/local/php-7.1.3/etc/php-fpm.conf and set it up with something like this:
[global]
error_log = /var/log/php-fpm.log

[www]
user = daemon
group = daemon
listen = 127.0.0.1:9000
pm = dynamic
pm.max_children = 25
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 10

Open in new window


(Replace "daemon" values in user/group with whatever user/group you want PHP to run as)

11. Set up your php.ini file in /usr/local/php-7.1.3/lib/php.ini however you want (you might be able to just copy over your existing php.ini file)

12. Start the PHP-FPM process:
/etc/init.d/php-fpm start

Once all that's done, you should just need to tell Apache to hand off PHP requests to it, so edit the Apache config and comment out existing references to PHP being loaded as a module, and then somewhere near the end of your Apache config, add:

TimeOut 300
ProxyTimeout 300

<FilesMatch \.php$>
 SetHandler "proxy:fcgi://127.0.0.1:9000"
</FilesMatch>

Open in new window


Replace the TimeOut and ProxyTimeout values with whatever time limits you want to enforce on PHP processes (e.g. 300 allows PHP and Apache processes to run up to a max of 5 minutes each, but that might be overkill for what you need).

Gracefully restart Apache and test out your PHP pages thoroughly, and then see if the problem goes away.
0
gr8gonzoConsultantCommented:
It also looks like there's an FPM package for php70w, too, but again, I'm not familiar with Webtatic so I can't tell you why you would want to use their package over the standard distribution. However, if you needed to stick with the Webtatic packages for some reason, you could always try out their FPM package to separate out the PHP to the FastCGI worker instead of Apache. In case there's any sort of incompatibility between the worker and the PHP module for Apache (mod_php) approach, the FastCGI would resolve that.
0
Mark LewisAuthor Commented:
I certainly appreciate the input on going to a new version but that would be way too disruptive. We have too many things working in concert to start changing so many parts.

I just need to figure out how to generate a core dump using the apache method, anything else is overkill.
0
gr8gonzoConsultantCommented:
I understand the hesitancy to make a change but let me offer up 3 points:
1. You would be swapping out PHP 7.x for PHP 7.x - just running as FastCGI instead of an Apache module. So the same scripts would be running in the same major version of PHP - a pretty safe change (and performs faster, too).
2. I'd bet money that the error will ultimately be a bug within the compiled php70w libraries, which means you'd have to change it anyway to fix the problem.
3. You should be comfortable with updating the PHP binaries regularly so you're staying on top of security fixes.

Also, the core dump is going to be difficult to get. Apache config isn't the problem - that part is simple - you just have to specify the CoreDumpDirectory setting in httpd.conf and restart Apache.

The problem is that the security measures on Linux distros like CentOS will actively prevent core dumps from being generated by setuid processes (like Apache child processes) to prevent leaking of data that could be sensitive. You have to jump through a lot of hoops to disable the security measures to allow this to happen, and that's not a good thing to do.

Even if you do get a core dump, you'd have to actually hope it has the right info to lead you to the root cause, which would almost certainly be bad code in the compiled PHP engine anyway (at which point you'd have to replace it to fix it).
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Apache Web Server

From novice to tech pro — start learning today.