Updating php causes apache to "hang" on restart
I compiled a newer version of PHP, now apache will not reload or stop, because the child processes and main apache process just "hang". I have to use 'kill -9' to remove the process. So far that is the only defect that we've noticed, but this is not something we want to put into production.
No php code is executed when testing this, just a restart or reload at anytime, will have the problem. I see this in the log file:
Nov 8 14:54:30 qa31 apache2[19028]: child process 19037 still did not exit, sending a SIGKILL
The child processes are eventually killed. Googling around, I see the error msg comes from server/mpm_common.c. Not much help, it just means the parent process has noticed the child process didn't die.
I used strace on the processes to see the last OS call where they stopped. Setting up a gdb env. isn't really possible, because our servers are still on the old etch Debian base, so the gdb packages are not available anymore.
The child and parent processes all stop with:
futex(0xb6d6e520, FUTEX_WAIT, 2, NULL
Googling with: apache2 prefork php futex
shows other people have the "futex" symptom, but the fixes suggested haven't been relevant or don't work.
A third-party interface requires at least php-5.2.6. I'll probably try earlier versions to see when the problem starts, then I can do a diff across the source files, and maybe I'll find the relevant change.
Any ideas on the cause, fix, or what to try next, to diagnose how to fix this would be great.
Best regards,
BruceRafn
--------------------------
Versions:
I've compiled several versons of php now:
php-5.2.0 - works
php-5.2.6 - processes don't die
php-5.2.12 - processes don't die
php-5.2.14 - processes don't die
php-5.3.3 - processes don't die
OS: Debian 4.0 etch
Kernel: 2.6.18-5-686 #1 SMP
Package Versions:
ii apache2 2.2.3-4+etch11 Next generation, scalable, extendable web
ii apache2.2-common 2.2.3-4+etch11 Next gen., scalable, extendable web
ii apache2-mpm-prefork 2.2.3-4+etch11 Traditional model for Apache HTTPD 2.1
ii apache2-utils 2.2.3-4+etch11 utility programs for webservers
ii libapache2-mod-security 2 2.1.1-0 Tighten web app. security for Apache
ii libapr1 1.2.7-9 The Apache Portable Runtime Library
ii libaprutil1 1.2.7+dfsg-2+e The Apache Portable Runtime Utility Library
ii libmysqlclient15off 5.0.32-7etch12 mysql database client library
ii mysql-client 5.0.32-7etch12 mysql database client (meta package
ii mysql-client-5.0 5.0.32-7etch12 mysql database client binaries
ii mysql-common 5.0.32-7etch12 mysql database common files
PHP extentions (custom compiled with the new php versions):
ii php-ssh2 0.11.0-1 Bindings, for the libssh2 library, for PHP
ii php-zip 1.10.2-1 A zip management extension
ii memcache 2.2.6-1 memcached module for PHP
More config details are in the attached file.
config-files.txt
No php code is executed when testing this, just a restart or reload at anytime, will have the problem. I see this in the log file:
Nov 8 14:54:30 qa31 apache2[19028]: child process 19037 still did not exit, sending a SIGKILL
The child processes are eventually killed. Googling around, I see the error msg comes from server/mpm_common.c. Not much help, it just means the parent process has noticed the child process didn't die.
I used strace on the processes to see the last OS call where they stopped. Setting up a gdb env. isn't really possible, because our servers are still on the old etch Debian base, so the gdb packages are not available anymore.
The child and parent processes all stop with:
futex(0xb6d6e520, FUTEX_WAIT, 2, NULL
Googling with: apache2 prefork php futex
shows other people have the "futex" symptom, but the fixes suggested haven't been relevant or don't work.
A third-party interface requires at least php-5.2.6. I'll probably try earlier versions to see when the problem starts, then I can do a diff across the source files, and maybe I'll find the relevant change.
Any ideas on the cause, fix, or what to try next, to diagnose how to fix this would be great.
Best regards,
BruceRafn
--------------------------
Versions:
I've compiled several versons of php now:
php-5.2.0 - works
php-5.2.6 - processes don't die
php-5.2.12 - processes don't die
php-5.2.14 - processes don't die
php-5.3.3 - processes don't die
OS: Debian 4.0 etch
Kernel: 2.6.18-5-686 #1 SMP
Package Versions:
ii apache2 2.2.3-4+etch11 Next generation, scalable, extendable web
ii apache2.2-common 2.2.3-4+etch11 Next gen., scalable, extendable web
ii apache2-mpm-prefork 2.2.3-4+etch11 Traditional model for Apache HTTPD 2.1
ii apache2-utils 2.2.3-4+etch11 utility programs for webservers
ii libapache2-mod-security 2 2.1.1-0 Tighten web app. security for Apache
ii libapr1 1.2.7-9 The Apache Portable Runtime Library
ii libaprutil1 1.2.7+dfsg-2+e The Apache Portable Runtime Utility Library
ii libmysqlclient15off 5.0.32-7etch12 mysql database client library
ii mysql-client 5.0.32-7etch12 mysql database client (meta package
ii mysql-client-5.0 5.0.32-7etch12 mysql database client binaries
ii mysql-common 5.0.32-7etch12 mysql database common files
PHP extentions (custom compiled with the new php versions):
ii php-ssh2 0.11.0-1 Bindings, for the libssh2 library, for PHP
ii php-zip 1.10.2-1 A zip management extension
ii memcache 2.2.6-1 memcached module for PHP
More config details are in the attached file.
config-files.txt
upgrade apache to 2.2.17 to plug millions of security holes. nobody tests builds with obsolete insecure software.
ASKER CERTIFIED SOLUTION
membership
Create a free account to see this answer
Signing up is free and takes 30 seconds. No credit card required.
ASKER
Thanks gheist for your suggestions. I will be trying them out, then update this question.
It *is correct* to build and test with obsolete software, if that is what is in production, because it exposes exactly this kind of problem. The main error is not keeping production updated, and that will be a top priority now. (I see the latest Debian apache is on version 2.2.16)
It *is correct* to build and test with obsolete software, if that is what is in production, because it exposes exactly this kind of problem. The main error is not keeping production updated, and that will be a top priority now. (I see the latest Debian apache is on version 2.2.16)
SOLUTION
membership
Create a free account to see this answer
Signing up is free and takes 30 seconds. No credit card required.
ASKER
More information: The problem will solve itself when you rename /lib/tls, so that those so files are not used. (mysql and mysqli extentions can both be compiled into php).
Solution found at the bottom of this page:
http://bugs.php.net/bug.php?id=42625
( Use this, if link broken: http://www.webcitation.org/5uSq0irej )
Solution found at the bottom of this page:
http://bugs.php.net/bug.php?id=42625
( Use this, if link broken: http://www.webcitation.org/5uSq0irej )
ASKER
The solution that works for our particular set of packages: compile php, using only one of these options, not both:
--with-mysql=
--with-mysqli=
--with-mysql=
--with-mysqli=
ASKER
php-5.2.1 - works
php-5.2.2 - works
php-5.2.3 - works
php-5.2.4 - processes don't die
I'll next look at the PHP release notes, and diff between the source files to see if there are changes the might have caused this problem. Low probability that I'll notice anything though.