andycas
asked on
Troubleshooting intermittent performance issues for LAMP server
We have an e-learning application developed in PHP/MySQL running on a Linux VPS with Apache. We are having issues where at some times of the day pages are being served very slowly, not just PHP pages but also static images, JS files, etc. Normally it's slowest when there are more users on the site, which is normally between 3-6pm, other time of the day it seems to be very fast.
We are aware of some particularly problematic pages which run slowly and execute multiple SQL queries. However I'm not convinced that this is the reason for the slowdown of the server.
I've been looking at this problem on and off over the last few weeks and my gut feeling is that it's not necessarily a problem directly with PHP or MySQL but something to do with Apache serving the requests and running out of threads. The server isn't particularly busy, it sits normally at 0% CPU and has 400MB RAM free. I just don't have enough expertise to be able to pin-point the exact problem and don't know what at the best tools to help me.
I would like to confirm that the server config is optimal and hopefully work out where the bottle neck is (i.e. is it a particular PHP page or MySQL query, or a problem with the Apache config) so that we can improve the performance and also know what to do better the next time!
I'm not sure what details are required to troubleshoot this, I'm including a netstat, top and mysql slow log output. What else is needed?
Final remark, I added the %T flag to Apache's LogFormat to be able to see the time taken for each request. While some requests are taking longer, ones which I know took a long time to load are still showing zero seconds, making me thing that the problem is that the requests might be getting queued up before Apache received them or acknowledged them.
Any help / suggestions / pointers in the right direction are much appreciated.
Andy
We are aware of some particularly problematic pages which run slowly and execute multiple SQL queries. However I'm not convinced that this is the reason for the slowdown of the server.
I've been looking at this problem on and off over the last few weeks and my gut feeling is that it's not necessarily a problem directly with PHP or MySQL but something to do with Apache serving the requests and running out of threads. The server isn't particularly busy, it sits normally at 0% CPU and has 400MB RAM free. I just don't have enough expertise to be able to pin-point the exact problem and don't know what at the best tools to help me.
I would like to confirm that the server config is optimal and hopefully work out where the bottle neck is (i.e. is it a particular PHP page or MySQL query, or a problem with the Apache config) so that we can improve the performance and also know what to do better the next time!
I'm not sure what details are required to troubleshoot this, I'm including a netstat, top and mysql slow log output. What else is needed?
Final remark, I added the %T flag to Apache's LogFormat to be able to see the time taken for each request. While some requests are taking longer, ones which I know took a long time to load are still showing zero seconds, making me thing that the problem is that the requests might be getting queued up before Apache received them or acknowledged them.
Any help / suggestions / pointers in the right direction are much appreciated.
Andy
Output from "top" during a particulary slow period:
---------------------------------------------------
top - 18:06:54 up 17:28, 3 users, load average: 0.43, 0.33, 0.18
Tasks: 54 total, 1 running, 53 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0% us, 0.0% sy, 0.0% ni, 99.8% id, 0.1% wa, 0.0% hi, 0.0% si
Mem: 524288k total, 126260k used, 398028k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19887 root 16 0 2092 1104 856 R 0 0.2 0:00.20 top
1 root 16 0 1800 668 564 S 0 0.1 0:00.08 init
17832 root 16 0 1496 576 456 S 0 0.1 0:00.57 syslogd
17902 root 19 0 2544 1340 1068 S 0 0.3 0:00.01 mysqld_safe
18010 mysql 16 0 134m 23m 5728 S 0 4.6 55:47.32 mysqld
18011 root 15 0 1424 496 432 S 0 0.1 0:00.00 logger
18209 root 18 0 3992 840 640 S 0 0.2 0:00.00 couriertcpd
18212 root 18 0 2908 736 564 S 0 0.1 0:00.00 courierlogger
18242 root 18 0 3992 840 640 S 0 0.2 0:00.00 couriertcpd
18245 root 18 0 2908 736 564 S 0 0.1 0:00.00 courierlogger
18255 root 21 0 3992 840 640 S 0 0.2 0:00.00 couriertcpd
18257 root 18 0 2908 736 564 S 0 0.1 0:00.00 courierlogger
18266 root 21 0 3992 840 640 S 0 0.2 0:00.00 couriertcpd
18269 root 18 0 2908 736 564 S 0 0.1 0:00.00 courierlogger
18309 root 15 0 2112 832 664 S 0 0.2 0:00.08 xinetd
18418 bind 19 0 36276 2968 1888 S 0 0.6 0:00.09 named
19512 qmails 17 0 1484 400 312 S 0 0.1 0:00.03 qmail-send
19515 qmaill 15 0 1436 456 392 S 0 0.1 0:00.00 splogger
19516 root 18 0 1464 360 272 S 0 0.1 0:00.00 qmail-lspawn
19517 qmailr 18 0 1460 356 268 S 0 0.1 0:00.00 qmail-rspawn
19518 qmailq 18 0 1432 336 276 S 0 0.1 0:00.00 qmail-clean
19780 root 16 0 30788 12m 7764 S 0 2.5 0:01.58 apache2
19960 root 16 0 36996 5476 3500 S 0 1.0 0:00.02 httpsd
19963 psaadm 15 0 44008 28m 19m S 0 5.5 0:01.62 httpsd
20096 root 16 0 4772 1084 748 S 0 0.2 0:01.21 sshd
20382 root 15 0 2040 852 676 S 0 0.2 0:00.07 cron
27796 root 17 0 8896 2536 1984 S 0 0.5 0:00.16 sshd
27939 netlangu 15 0 9064 1816 1240 S 0 0.3 0:02.29 sshd
13966 psaadm 16 0 45856 30m 19m S 0 5.9 0:01.38 httpsd
14048 psaadm 15 0 43516 26m 17m S 0 5.1 0:01.59 httpsd
7236 root 16 0 8896 2532 1980 S 0 0.5 0:00.13 sshd
7556 netlangu 16 0 9060 1816 1232 S 0 0.3 0:01.00 sshd
7567 netlangu 15 0 3176 1912 1248 S 0 0.4 0:00.09 bash
7964 root 16 0 8896 2532 1980 S 0 0.5 0:00.05 sshd
8026 netlangu 16 0 8900 1788 1228 S 0 0.3 0:00.21 sshd
8029 netlangu 15 0 3224 1956 1244 S 0 0.4 0:00.06 bash
20010 www-data 16 0 35040 14m 5372 S 0 2.8 0:03.48 apache2
9869 www-data 15 0 31280 9988 4228 S 0 1.9 0:00.75 apache2
14091 www-data 16 0 35068 14m 5420 S 0 2.9 0:00.69 apache2
27664 root 16 0 5040 1528 1136 S 0 0.3 0:00.00 su
27737 root 16 0 2668 1652 1244 S 0 0.3 0:00.01 bash
9667 www-data 16 0 35056 14m 5348 S 0 2.8 0:00.81 apache2
9684 www-data 16 0 36664 15m 4664 S 0 3.0 0:00.69 apache2
15741 www-data 16 0 31348 10m 4992 S 0 2.1 0:00.63 apache2
15746 www-data 16 0 34968 13m 4512 S 0 2.7 0:00.51 apache2
15747 www-data 16 0 31220 10m 5040 S 0 2.1 0:00.67 apache2
20285 www-data 16 0 35056 13m 4580 S 0 2.7 0:00.69 apache2
21864 www-data 16 0 31204 10m 4964 S 0 2.0 0:00.47 apache2
Output from "netstat" during a slow period:
-------------------------------------------
root@netlanguages:/var/www/vhosts/netlanguages.com# netstat -t -n
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 705 0 79.170.194.137:80 84.127.74.175:48754 ESTABLISHED
tcp 0 0 79.170.194.137:80 90.163.59.214:49338 ESTABLISHED
tcp 0 0 79.170.194.137:80 90.163.59.214:49336 ESTABLISHED
tcp 0 0 79.170.194.137:80 90.163.59.214:49334 ESTABLISHED
tcp 0 0 79.170.194.137:80 90.163.59.214:49332 FIN_WAIT2
tcp 0 1 79.170.194.137:80 85.58.10.19:3592 LAST_ACK
tcp 0 0 79.170.194.137:80 62.57.44.213:4887 ESTABLISHED
tcp 0 0 79.170.194.137:80 62.57.44.213:4886 ESTABLISHED
tcp 0 0 79.170.194.137:80 62.57.44.213:4888 ESTABLISHED
tcp 0 0 79.170.194.126:22 80.33.133.83:3494 ESTABLISHED
tcp 0 0 79.170.194.137:80 83.34.133.42:17604 TIME_WAIT
tcp 684 0 79.170.194.137:80 80.174.164.61:63348 ESTABLISHED
tcp 0 0 79.170.194.137:80 80.174.164.61:63347 TIME_WAIT
tcp 557 0 79.170.194.137:80 85.58.36.65:61210 ESTABLISHED
tcp 444 0 79.170.194.137:80 88.22.67.218:1845 ESTABLISHED
tcp 0 0 79.170.194.126:22 80.33.133.83:3728 ESTABLISHED
tcp 0 1 79.170.194.137:80 77.208.139.120:1449 FIN_WAIT1
tcp 552 0 79.170.194.137:80 77.208.139.120:1450 ESTABLISHED
tcp 0 3033 79.170.194.137:80 212.59.221.14:50281 FIN_WAIT1
tcp 0 2248 79.170.194.126:22 80.33.133.83:3397 ESTABLISHED
tcp 0 0 79.170.194.126:22 80.33.133.83:2592 ESTABLISHED
tcp 797 0 79.170.194.137:80 84.88.36.27:28340 ESTABLISHED
tcp 0 0 79.170.194.137:80 77.224.48.105:1364 ESTABLISHED
tcp 0 0 79.170.194.137:80 77.224.48.105:1370 TIME_WAIT
tcp 0 0 79.170.194.137:80 77.224.48.105:1368 ESTABLISHED
tcp 536 0 79.170.194.137:80 77.224.48.105:1374 ESTABLISHED
tcp 0 0 79.170.194.137:80 77.224.48.105:1372 ESTABLISHED
tcp 0 2577 79.170.194.137:80 88.15.147.165:29644 LAST_ACK
tcp 0 2577 79.170.194.137:80 88.15.147.165:29645 LAST_ACK
tcp 0 2588 79.170.194.137:80 88.15.147.165:29646 LAST_ACK
tcp 0 2577 79.170.194.137:80 88.15.147.165:29647 LAST_ACK
tcp 0 5143 79.170.194.137:80 88.15.147.165:29640 LAST_ACK
tcp 0 5143 79.170.194.137:80 88.15.147.165:29641 LAST_ACK
tcp 0 2577 79.170.194.137:80 88.15.147.165:29642 LAST_ACK
tcp 0 2577 79.170.194.137:80 88.15.147.165:29643 LAST_ACK
tcp 0 0 79.170.194.137:80 83.35.158.152:1776 FIN_WAIT2
tcp 0 0 79.170.194.137:80 88.18.60.176:10041 FIN_WAIT2
tcp 121 0 79.170.194.137:80 213.27.221.227:2178 CLOSE_WAIT
tcp 607 0 79.170.194.137:80 87.217.138.252:1310 ESTABLISHED
tcp 0 0 79.170.194.137:80 81.34.86.92:2225 FIN_WAIT2
tcp 0 0 79.170.194.137:80 81.34.86.92:2197 TIME_WAIT
tcp 529 0 79.170.194.137:80 80.33.133.83:3420 ESTABLISHED
tcp 451 0 79.170.194.137:80 80.174.133.95:1958 ESTABLISHED
tcp 0 0 79.170.194.137:80 80.174.133.95:1955 FIN_WAIT2
tcp 0 0 79.170.194.137:80 80.174.133.95:1953 TIME_WAIT
tcp 0 0 79.170.194.137:80 87.219.67.151:1073 TIME_WAIT
tcp 554 0 79.170.194.137:80 87.219.67.151:1076 ESTABLISHED
tcp 571 0 79.170.194.137:80 81.34.86.92:2265 ESTABLISHED
tcp 822 0 79.170.194.137:80 193.130.97.35:25637 ESTABLISHED
A sample of some queries from MySQL slow log:
---------------------------------------------
# Time: 081022 16:47:07
# User@Host: discovia[discovia] @ localhost []
# Query_time: 11 Lock_time: 0 Rows_sent: 39 Rows_examined: 374
SELECT id, name FROM student_groups WHERE student_groups.id IN
(SELECT users.student_group FROM users
INNER JOIN user_assigned_units ON users.id = user_assigned_units.user_id
WHERE user_assigned_units.tutor_id = 321) ORDER BY name;
# Time: 081022 17:00:44
# User@Host: discovia[discovia] @ localhost []
# Query_time: 12 Lock_time: 0 Rows_sent: 39 Rows_examined: 374
SELECT id, name FROM student_groups WHERE student_groups.id IN
(SELECT users.student_group FROM users
INNER JOIN user_assigned_units ON users.id = user_assigned_units.user_id
WHERE user_assigned_units.tutor_id = 321) ORDER BY name;
# Time: 081022 17:03:05
# User@Host: discovia[discovia] @ localhost []
# Query_time: 11 Lock_time: 0 Rows_sent: 37 Rows_examined: 374
SELECT id, name FROM student_groups WHERE student_groups.id IN
(SELECT users.student_group FROM users
INNER JOIN user_assigned_units ON users.id = user_assigned_units.user_id
WHERE user_assigned_units.tutor_id = 327) ORDER BY name;
# Time: 081022 17:13:38
# User@Host: discovia[discovia] @ localhost []
# Query_time: 11 Lock_time: 0 Rows_sent: 39 Rows_examined: 374
SELECT id, name FROM student_groups WHERE student_groups.id IN
(SELECT users.student_group FROM users
INNER JOIN user_assigned_units ON users.id = user_assigned_units.user_id
WHERE user_assigned_units.tutor_id = 321) ORDER BY name;
# Time: 081022 17:14:02
# User@Host: discovia[discovia] @ localhost []
# Query_time: 11 Lock_time: 0 Rows_sent: 39 Rows_examined: 374
SELECT id, name FROM student_groups WHERE student_groups.id IN
(SELECT users.student_group FROM users
INNER JOIN user_assigned_units ON users.id = user_assigned_units.user_id
WHERE user_assigned_units.tutor_id = 321) ORDER BY name;
# Time: 081022 17:14:36
# User@Host: discovia[discovia] @ localhost []
# Query_time: 11 Lock_time: 0 Rows_sent: 51 Rows_examined: 374
SELECT id, name FROM student_groups WHERE student_groups.id IN
(SELECT users.student_group FROM users
INNER JOIN user_assigned_units ON users.id = user_assigned_units.user_id
WHERE user_assigned_units.tutor_id = 322) ORDER BY name;
# Time: 081022 17:14:55
# User@Host: discovia[discovia] @ localhost []
# Query_time: 11 Lock_time: 0 Rows_sent: 51 Rows_examined: 374
SELECT id, name FROM student_groups WHERE student_groups.id IN
(SELECT users.student_group FROM users
INNER JOIN user_assigned_units ON users.id = user_assigned_units.user_id
WHERE user_assigned_units.tutor_id = 322) ORDER BY name;
# Time: 081022 17:43:51
# User@Host: discovia[discovia] @ localhost []
# Query_time: 11 Lock_time: 0 Rows_sent: 51 Rows_examined: 375
SELECT id, name FROM student_groups WHERE student_groups.id IN
(SELECT users.student_group FROM users
INNER JOIN user_assigned_units ON users.id = user_assigned_units.user_id
WHERE user_assigned_units.tutor_id = 322) ORDER BY name;
ASKER
@Xyptilon2:
Many thanks for the reply and suggestions.
1. The server doesn't want to let me turn on swap - maybe because it's a virtual server:
root@netlanguages:/# swapon /swap1
swapon: /swap1: Operation not permitted
It has burst memory available, it has 512MB normally but automatically has access to more if necessary. As it sits at 400MB free (according to "top") even when it's running slowly I don't think it's a memory problem.
2. I've attached the output from server-status below, again at a point when it's running slowly. The server-status page itself took about 30 seconds to display! Is "0 idle workers" a sign that I need to increase something in my apache settings?
3. I'll check the error_log shortly and post what I find in it here.
Thanks,
Andy
Many thanks for the reply and suggestions.
1. The server doesn't want to let me turn on swap - maybe because it's a virtual server:
root@netlanguages:/# swapon /swap1
swapon: /swap1: Operation not permitted
It has burst memory available, it has 512MB normally but automatically has access to more if necessary. As it sits at 400MB free (according to "top") even when it's running slowly I don't think it's a memory problem.
2. I've attached the output from server-status below, again at a point when it's running slowly. The server-status page itself took about 30 seconds to display! Is "0 idle workers" a sign that I need to increase something in my apache settings?
3. I'll check the error_log shortly and post what I find in it here.
Thanks,
Andy
Server Version: Apache/2.0.55 (Ubuntu) mod_python/3.2.8 Python/2.4.4c1 PHP/5.1.6 mod_ssl/2.0.55 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8
Server Built: Aug 16 2007 22:41:15
Current Time: Thursday, 23-Oct-2008 18:37:30 CEST
Restart Time: Thursday, 23-Oct-2008 15:55:54 CEST
Parent Server Generation: 2
Server uptime: 2 hours 41 minutes 35 seconds
10 requests currently being processed, 0 idle workers
RRKRRKRWKK......................................................
................................................................
................................................................
................................................................
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process
PID Key:
17642 in state: R , 3268 in state: R , 17907 in state: K
3342 in state: R , 32685 in state: R , 32739 in state: K
17668 in state: R , 3453 in state: W , 3461 in state: K
22259 in state: K ,
Perhaps yes, what are the settings for your StartServers, SpareServers? They are in your apache2.conf or httpd.conf file.
ASKER
Okay, I've run apache -V and it looks like apache is in prefork mode (which seems to be what it should be from what I've read). It says:
Server compiled with....
-D APACHE_MPM_DIR="server/mpm /prefork"
That means it's prefork mode, right?
Below are the relevant (I think) bits of what's in my apache2.conf.
<IfModule prefork.c>
StartServers 1
MinSpareServers 1
MaxSpareServers 5
MaxClients 10
MaxRequestsPerChild 0
</IfModule>
According to the comments in the file MaxClients is the "maximum number of server processes allowed to start" but according to the docs it seems to be the number of simultaneous requests to the server. And 10 seems to be incredibly low!
FYI, these are the Apache settings as the server was provided to us by the VPS hosting provider. I've just looked at a different VPS from a different provider and that one has very different values (like MaxClients of 256). What would you recommend that I change them to?
Thanks.
Server compiled with....
-D APACHE_MPM_DIR="server/mpm
That means it's prefork mode, right?
Below are the relevant (I think) bits of what's in my apache2.conf.
<IfModule prefork.c>
StartServers 1
MinSpareServers 1
MaxSpareServers 5
MaxClients 10
MaxRequestsPerChild 0
</IfModule>
According to the comments in the file MaxClients is the "maximum number of server processes allowed to start" but according to the docs it seems to be the number of simultaneous requests to the server. And 10 seems to be incredibly low!
FYI, these are the Apache settings as the server was provided to us by the VPS hosting provider. I've just looked at a different VPS from a different provider and that one has very different values (like MaxClients of 256). What would you recommend that I change them to?
Thanks.
Relevant bits from apache2.conf file:
-------------------------------------
# Timeout: The number of seconds before receives and sends time out.
Timeout 300
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
KeepAlive On
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
MaxKeepAliveRequests 100
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
KeepAliveTimeout 15
##
## Server-Pool Size Regulation (MPM specific)
##
# prefork MPM
# StartServers ......... number of server processes to start
# MinSpareServers ...... minimum number of server processes which are kept spare
# MaxSpareServers ...... maximum number of server processes which are kept spare
# MaxClients ........... maximum number of server processes allowed to start
# MaxRequestsPerChild .. maximum number of requests a server process serves
<IfModule prefork.c>
StartServers 1
MinSpareServers 1
MaxSpareServers 5
MaxClients 10
MaxRequestsPerChild 0
</IfModule>
# pthread MPM
# StartServers ......... initial number of server processes to start
# MaxClients ........... maximum number of server processes allowed to start
# MinSpareThreads ...... minimum number of worker threads which are kept spare
# MaxSpareThreads ...... maximum number of worker threads which are kept spare
# ThreadsPerChild ...... constant number of worker threads in each server process
# MaxRequestsPerChild .. maximum number of requests a server process serves
<IfModule worker.c>
StartServers 1
MaxClients 10
MinSpareThreads 1
MaxSpareThreads 4
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>
# perchild MPM
# NumServers ........... constant number of server processes
# StartThreads ......... initial number of worker threads in each server process
# MinSpareThreads ...... minimum number of worker threads which are kept spare
# MaxSpareThreads ...... maximum number of worker threads which are kept spare
# MaxThreadsPerChild ... maximum number of worker threads in each server process
# MaxRequestsPerChild .. maximum number of connections per server process (then it dies)
<IfModule perchild.c>
NumServers 5
StartThreads 5
MinSpareThreads 1
MaxSpareThreads 4
MaxThreadsPerChild 20
MaxRequestsPerChild 0
AcceptMutex fcntl
</IfModule>
ASKER
Just wanted to add the response to your original third question, that the error_log doesn't seem to have too many warnings, which surprises me a bit. I've included it for all of October with unrelated errors removed (file not founds, etc).
The only errors of note I see are four of these:
[error] server reached MaxClients setting, consider raising the MaxClients setting
What's interesting is that there have been many times when the server has been running very slowly but none of these MaxClients errors have appears. Is that normal?
The only errors of note I see are four of these:
[error] server reached MaxClients setting, consider raising the MaxClients setting
What's interesting is that there have been many times when the server has been running very slowly but none of these MaxClients errors have appears. Is that normal?
[Tue Oct 07 10:30:53 2008] [notice] Graceful restart requested, doing restart
[Tue Oct 07 10:30:54 2008] [notice] mod_python: Creating 8 session mutexes based on 10 max processes and 0 max threads.
[Tue Oct 07 10:30:55 2008] [notice] Apache/2.0.55 (Ubuntu) mod_python/3.2.8 Python/2.4.4c1 PHP/5.1.6 mod_ssl/2.0.55 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Tue Oct 07 12:10:08 2008] [notice] Graceful restart requested, doing restart
[Tue Oct 07 12:10:09 2008] [notice] mod_python: Creating 8 session mutexes based on 10 max processes and 0 max threads.
[Tue Oct 07 12:10:09 2008] [notice] Apache/2.0.55 (Ubuntu) mod_python/3.2.8 Python/2.4.4c1 PHP/5.1.6 mod_ssl/2.0.55 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Tue Oct 07 12:11:02 2008] [notice] Graceful restart requested, doing restart
[Tue Oct 07 12:11:02 2008] [notice] mod_python: Creating 8 session mutexes based on 10 max processes and 0 max threads.
[Tue Oct 07 12:11:03 2008] [notice] Apache/2.0.55 (Ubuntu) mod_python/3.2.8 Python/2.4.4c1 PHP/5.1.6 mod_ssl/2.0.55 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Tue Oct 07 19:06:49 2008] [notice] caught SIGTERM, shutting down
[Tue Oct 07 19:08:38 2008] [warn] Init: Session Cache is not configured [hint: SSLSessionCache]
[Tue Oct 07 19:08:38 2008] [notice] suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec2)
[Tue Oct 07 19:08:38 2008] [notice] mod_python: Creating 8 session mutexes based on 10 max processes and 0 max threads.
[Tue Oct 07 19:08:40 2008] [notice] Apache/2.0.55 (Ubuntu) mod_python/3.2.8 Python/2.4.4c1 PHP/5.1.6 mod_ssl/2.0.55 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Tue Oct 07 19:08:47 2008] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Mon Oct 13 18:33:47 2008] [notice] caught SIGTERM, shutting down
[Mon Oct 13 18:35:53 2008] [warn] Init: Session Cache is not configured [hint: SSLSessionCache]
[Mon Oct 13 18:35:53 2008] [notice] suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec2)
[Mon Oct 13 18:35:53 2008] [notice] mod_python: Creating 8 session mutexes based on 10 max processes and 0 max threads.
[Mon Oct 13 18:35:57 2008] [notice] Apache/2.0.55 (Ubuntu) mod_python/3.2.8 Python/2.4.4c1 PHP/5.1.6 mod_ssl/2.0.55 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Mon Oct 13 18:36:02 2008] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Mon Oct 13 18:41:39 2008] [notice] SIGHUP received. Attempting to restart
[Mon Oct 13 18:41:39 2008] [notice] mod_python: Creating 8 session mutexes based on 10 max processes and 0 max threads.
[Mon Oct 13 18:41:40 2008] [notice] Apache/2.0.55 (Ubuntu) mod_python/3.2.8 Python/2.4.4c1 PHP/5.1.6 mod_ssl/2.0.55 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Wed Oct 15 17:57:44 2008] [notice] SIGHUP received. Attempting to restart
[Wed Oct 15 17:57:44 2008] [notice] mod_python: Creating 8 session mutexes based on 10 max processes and 0 max threads.
[Wed Oct 15 17:57:45 2008] [notice] Apache/2.0.55 (Ubuntu) mod_python/3.2.8 Python/2.4.4c1 PHP/5.1.6 mod_ssl/2.0.55 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Thu Oct 16 16:41:25 2008] [notice] SIGHUP received. Attempting to restart
[Thu Oct 16 16:41:25 2008] [notice] mod_python: Creating 8 session mutexes based on 10 max processes and 0 max threads.
[Thu Oct 16 16:41:26 2008] [notice] Apache/2.0.55 (Ubuntu) mod_python/3.2.8 Python/2.4.4c1 PHP/5.1.6 mod_ssl/2.0.55 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Thu Oct 16 16:56:32 2008] [notice] SIGHUP received. Attempting to restart
[Thu Oct 16 16:56:32 2008] [notice] mod_python: Creating 8 session mutexes based on 10 max processes and 0 max threads.
[Thu Oct 16 16:56:32 2008] [notice] Apache/2.0.55 (Ubuntu) mod_python/3.2.8 Python/2.4.4c1 PHP/5.1.6 mod_ssl/2.0.55 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Thu Oct 16 17:08:34 2008] [notice] SIGHUP received. Attempting to restart
[Thu Oct 16 17:08:34 2008] [notice] mod_python: Creating 8 session mutexes based on 10 max processes and 0 max threads.
[Thu Oct 16 17:08:34 2008] [notice] Apache/2.0.55 (Ubuntu) mod_python/3.2.8 Python/2.4.4c1 PHP/5.1.6 mod_ssl/2.0.55 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Thu Oct 16 17:12:44 2008] [notice] SIGHUP received. Attempting to restart
[Thu Oct 16 17:12:44 2008] [notice] mod_python: Creating 8 session mutexes based on 10 max processes and 0 max threads.
[Thu Oct 16 17:12:44 2008] [notice] Apache/2.0.55 (Ubuntu) mod_python/3.2.8 Python/2.4.4c1 PHP/5.1.6 mod_ssl/2.0.55 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Thu Oct 16 17:33:16 2008] [notice] Graceful restart requested, doing restart
[Thu Oct 16 17:33:16 2008] [notice] mod_python: Creating 8 session mutexes based on 10 max processes and 0 max threads.
[Thu Oct 16 17:33:17 2008] [notice] Apache/2.0.55 (Ubuntu) mod_python/3.2.8 Python/2.4.4c1 PHP/5.1.6 mod_ssl/2.0.55 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Wed Oct 22 00:38:11 2008] [notice] caught SIGTERM, shutting down
[Wed Oct 22 00:40:39 2008] [warn] Init: Session Cache is not configured [hint: SSLSessionCache]
[Wed Oct 22 00:40:39 2008] [notice] suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec2)
[Wed Oct 22 00:40:39 2008] [notice] mod_python: Creating 8 session mutexes based on 10 max processes and 0 max threads.
[Wed Oct 22 00:40:42 2008] [notice] Apache/2.0.55 (Ubuntu) mod_python/3.2.8 Python/2.4.4c1 PHP/5.1.6 mod_ssl/2.0.55 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Wed Oct 22 00:53:05 2008] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Wed Oct 22 18:45:17 2008] [notice] SIGHUP received. Attempting to restart
[Wed Oct 22 18:45:17 2008] [notice] mod_python: Creating 8 session mutexes based on 10 max processes and 0 max threads.
[Wed Oct 22 18:45:18 2008] [notice] Apache/2.0.55 (Ubuntu) mod_python/3.2.8 Python/2.4.4c1 PHP/5.1.6 mod_ssl/2.0.55 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Wed Oct 22 18:51:58 2008] [notice] SIGHUP received. Attempting to restart
[Wed Oct 22 18:51:58 2008] [notice] mod_python: Creating 8 session mutexes based on 10 max processes and 0 max threads.
[Wed Oct 22 18:51:58 2008] [notice] Apache/2.0.55 (Ubuntu) mod_python/3.2.8 Python/2.4.4c1 PHP/5.1.6 mod_ssl/2.0.55 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Wed Oct 22 18:57:30 2008] [notice] SIGHUP received. Attempting to restart
[Wed Oct 22 18:57:30 2008] [notice] mod_python: Creating 8 session mutexes based on 10 max processes and 0 max threads.
[Wed Oct 22 18:57:30 2008] [notice] Apache/2.0.55 (Ubuntu) mod_python/3.2.8 Python/2.4.4c1 PHP/5.1.6 mod_ssl/2.0.55 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Wed Oct 22 19:12:54 2008] [notice] caught SIGTERM, shutting down
[Wed Oct 22 19:14:43 2008] [warn] Init: Session Cache is not configured [hint: SSLSessionCache]
[Wed Oct 22 19:14:43 2008] [notice] suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec2)
[Wed Oct 22 19:14:43 2008] [notice] mod_python: Creating 8 session mutexes based on 10 max processes and 0 max threads.
[Wed Oct 22 19:14:43 2008] [notice] Apache/2.0.55 (Ubuntu) mod_python/3.2.8 Python/2.4.4c1 PHP/5.1.6 mod_ssl/2.0.55 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Wed Oct 22 19:14:48 2008] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Thu Oct 23 15:54:11 2008] [notice] SIGHUP received. Attempting to restart
[Thu Oct 23 15:54:12 2008] [notice] mod_python: Creating 8 session mutexes based on 10 max processes and 0 max threads.
[Thu Oct 23 15:54:13 2008] [notice] Apache/2.0.55 (Ubuntu) mod_python/3.2.8 Python/2.4.4c1 PHP/5.1.6 mod_ssl/2.0.55 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Thu Oct 23 15:55:54 2008] [notice] SIGHUP received. Attempting to restart
[Thu Oct 23 15:55:54 2008] [notice] mod_python: Creating 8 session mutexes based on 10 max processes and 0 max threads.
[Thu Oct 23 15:55:54 2008] [notice] Apache/2.0.55 (Ubuntu) mod_python/3.2.8 Python/2.4.4c1 PHP/5.1.6 mod_ssl/2.0.55 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Thu Oct 23 21:28:15 2008] [notice] SIGHUP received. Attempting to restart
[Thu Oct 23 21:28:16 2008] [notice] mod_python: Creating 8 session mutexes based on 10 max processes and 0 max threads.
[Thu Oct 23 21:28:17 2008] [notice] Apache/2.0.55 (Ubuntu) mod_python/3.2.8 Python/2.4.4c1 PHP/5.1.6 mod_ssl/2.0.55 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Thu Oct 23 21:31:18 2008] [notice] SIGHUP received. Attempting to restart
[Thu Oct 23 21:31:18 2008] [notice] mod_python: Creating 8 session mutexes based on 10 max processes and 0 max threads.
[Thu Oct 23 21:31:19 2008] [notice] Apache/2.0.55 (Ubuntu) mod_python/3.2.8 Python/2.4.4c1 PHP/5.1.6 mod_ssl/2.0.55 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Thu Oct 23 21:39:32 2008] [notice] SIGHUP received. Attempting to restart
[Thu Oct 23 21:39:32 2008] [notice] mod_python: Creating 8 session mutexes based on 30 max processes and 0 max threads.
[Thu Oct 23 21:39:32 2008] [notice] Apache/2.0.55 (Ubuntu) mod_python/3.2.8 Python/2.4.4c1 PHP/5.1.6 mod_ssl/2.0.55 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Many thanks, just what I wanted. I solved my immediate problem and learnt a lot in the mean time.
ASKER
I'll have to check out iostat at some stage - it's not installed on the server at the moment. However, raising MaxClients to 50 seems to have made a huge difference. It's now responding well even during heavier periods of traffic. I'm currently running the settings below and CPU and RAM are both low so I've probably got room to increase more if necessary or else start to tune MySQL too.
Thanks!
Thanks!
<IfModule prefork.c>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 50
MaxRequestsPerChild 250
</IfModule>
cd /
dd if=/dev/zero of=/swap1 bs=1024 count=1048576 # Make a 1GB file
mkswap /swap1 1048576 # Make file swap device
sync; sync # Flush file system buffers
swapon /swap1 # Activate swap page
Second, you can enable the status logger in Apache. The relevant portion is probably commented out in your apache config, but it looks something like
# Allow server status reports, with the URL of http://servername/server-status
# Change the ".your_domain.com" to match your domain to enable.
#
#<Location /server-status>
# SetHandler server-status
# Order deny,allow
# Deny from all
# Allow from .your_domain.com
#</Location>
You can then go to http://www.yourdomain.com/server-status and see some useful "per request" information.
Third...do you have any information in your apache error_log ? For example that there are too few available servers?