Updating php causes apache to "hang" on restart
I compiled a newer version of PHP, now apache will not reload or stop, because the child processes and main apache process just "hang". I have to use 'kill -9' to remove the process. So far that is the only defect that we've noticed, but this is not something we want to put into production.
No php code is executed when testing this, just a restart or reload at anytime, will have the problem. I see this in the log file:
Nov 8 14:54:30 qa31 apache2: child process 19037 still did not exit, sending a SIGKILL
The child processes are eventually killed. Googling around, I see the error msg comes from server/mpm_common.c. Not much help, it just means the parent process has noticed the child process didn't die.
I used strace on the processes to see the last OS call where they stopped. Setting up a gdb env. isn't really possible, because our servers are still on the old etch Debian base, so the gdb packages are not available anymore.
The child and parent processes all stop with:
futex(0xb6d6e520, FUTEX_WAIT, 2, NULL
Googling with: apache2 prefork php futex
shows other people have the "futex" symptom, but the fixes suggested haven't been relevant or don't work.
A third-party interface requires at least php-5.2.6. I'll probably try earlier versions to see when the problem starts, then I can do a diff across the source files, and maybe I'll find the relevant change.
Any ideas on the cause, fix, or what to try next, to diagnose how to fix this would be great.
I've compiled several versons of php now:
php-5.2.0 - works
php-5.2.6 - processes don't die
php-5.2.12 - processes don't die
php-5.2.14 - processes don't die
php-5.3.3 - processes don't die
OS: Debian 4.0 etch
Kernel: 2.6.18-5-686 #1 SMP
ii apache2 2.2.3-4+etch11 Next generation, scalable, extendable web
ii apache2.2-common 2.2.3-4+etch11 Next gen., scalable, extendable web
ii apache2-mpm-prefork 2.2.3-4+etch11 Traditional model for Apache HTTPD 2.1
ii apache2-utils 2.2.3-4+etch11 utility programs for webservers
ii libapache2-mod-security 2 2.1.1-0 Tighten web app. security for Apache
ii libapr1 1.2.7-9 The Apache Portable Runtime Library
ii libaprutil1 1.2.7+dfsg-2+e The Apache Portable Runtime Utility Library
ii libmysqlclient15off 5.0.32-7etch12 mysql database client library
ii mysql-client 5.0.32-7etch12 mysql database client (meta package
ii mysql-client-5.0 5.0.32-7etch12 mysql database client binaries
ii mysql-common 5.0.32-7etch12 mysql database common files
PHP extentions (custom compiled with the new php versions):
ii php-ssh2 0.11.0-1 Bindings, for the libssh2 library, for PHP
ii php-zip 1.10.2-1 A zip management extension
ii memcache 2.2.6-1 memcached module for PHP
Thanks gheist for your suggestions. I will be trying them out, then update this question.
It *is correct* to build and test with obsolete software, if that is what is in production, because it exposes exactly this kind of problem. The main error is not keeping production updated, and that will be a top priority now. (I see the latest Debian apache is on version 2.2.16)
The solution that works for our particular set of packages: compile php, using only one of these options, not both:
PHP is a widely-used server-side scripting language especially suited for web development, powering tens of millions of sites from Facebook to personal WordPress blogs. PHP is often paired with the MySQL relational database, but includes support for most other mainstream databases. By utilizing different Server APIs, PHP can work on many different web servers as a server-side scripting language.