SimonBrook
asked on
High load average all of a sudden - Apache2.
Hello,
We have a Debian server running apache2. Our load average used to be below 1.0, however this week it is now at anywhere between 5-10.
It is causing real big problems. It is our front end clinet facing HTTP server. Does anyone have any advice? If I disable the apache2 service, the load average drops back down to an acceptable level, re-enabling it rockets the load average back up.
I'm not extremely well versed in linux so be gentle!
Thanks,
We have a Debian server running apache2. Our load average used to be below 1.0, however this week it is now at anywhere between 5-10.
It is causing real big problems. It is our front end clinet facing HTTP server. Does anyone have any advice? If I disable the apache2 service, the load average drops back down to an acceptable level, re-enabling it rockets the load average back up.
I'm not extremely well versed in linux so be gentle!
Thanks,
It's more than likely that Apache is getting hit relatively hard by someone (could be legitimate or malicious).
Do a
tail /var/log/apache2/error.log
to see if you can see anything that looks unusual.
Also do
tail -100 /var/log/apache2/access.lo
to view the last 100 lines of the Apache access log to see if you can see anything unusual, eg: multiple requests all coming from the same IP or a particular bot hitting your site.
Do a
tail /var/log/apache2/error.log
to see if you can see anything that looks unusual.
Also do
tail -100 /var/log/apache2/access.lo
to view the last 100 lines of the Apache access log to see if you can see anything unusual, eg: multiple requests all coming from the same IP or a particular bot hitting your site.
ASKER
I did the tail -100.
And the following came back. Bizarre considering the debians system time is.
Mars:~# date
Mon Jul 12 13:21:47 BST 2010
And the following came back. Bizarre considering the debians system time is.
Mars:~# date
Mon Jul 12 13:21:47 BST 2010
Mars:~# tail -100 /var/log/apache2/access.log | less
81.171.153.68 - - [11/Jul/2010:07:48:11 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=D:/clips/mp3/flo-rida%20feat.%20david%20guett_0.m3e&completed=1&s
erialNo=K214330&entityID=_EID_&type=MUSICCLIP&file=flo-rida%20feat.%20david%20guett_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:12 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/clips/mp3/kylie%20-%20all%20the%20lovers-1.m3e&completed=1&ser
ialNo=K333530&entityID=_EID_&type=MUSICCLIP&file=kylie%20-%20all%20the%20lovers-1.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:12 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/clips/mp3/travie%20mccoy%20-billionaire%20_0.m3e&completed=1&s
erialNo=K332730&entityID=_EID_&type=MUSICCLIP&file=travie%20mccoy%20-billionaire%20_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:12 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/clips/mp3/game%20feat.%20jt%20-%20ain't%20no%20d_0.m3e&complet
ed=1&serialNo=K329030&entityID=_EID_&type=MUSICCLIP&file=game%20feat.%20jt%20-%20ain't%20no%20d_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:12 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/clips/mp3/pendulum%20-%20witchcraft.m3e&completed=1&serialNo=K
327530&entityID=_EID_&type=MUSICCLIP&file=pendulum%20-%20witchcraft.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:13 +0100] "GET /kvAutoUpdate/hawkSuccessful.php?id=210065&serialNo=K332330&success=NO&completed=NO&action=2&datestart=2
HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:13 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/Kaleidovision/SharedLocal/KvAutoUpdate/LatestImage-20100709062
946.1440.xml&completed=1&serialNo=K318430&entityID=_EID_&type=UNKNOWN&file= HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:13 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/clips/mp3/we%20are%20scientists%20-%20nice%20g_0.m3e&completed
=1&serialNo=K332830&entityID=_EID_&type=MUSICCLIP&file=we%20are%20scientists%20-%20nice%20g_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:13 +0100] "GET /kvAutoUpdate/hawkSuccessful.php?id=210160&serialNo=K329730&success=NO&completed=YES&action=1 HTTP/1.1" 2
00 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:13 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=D:/clips/mp3/kanye%20west%20-power%20(super%20c_0.m3e&completed=1
&serialNo=K214330&entityID=_EID_&type=MUSICCLIP&file=kanye%20west%20-power%20(super%20c_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:14 +0100] "GET /kvAutoUpdate/hawkSuccessful.php?id=210142&serialNo=K329630&success=NO&completed=YES&action=1 HTTP/1.1" 2
00 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:14 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=D:/clips/mp3/game%20feat.%20jt%20-%20ain't%20no%20d_0.m3e&complet
ed=1&serialNo=K318030&entityID=_EID_&type=MUSICCLIP&file=game%20feat.%20jt%20-%20ain't%20no%20d_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:14 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/clips/mp3/leddra%20chapman%20-%20summer%20song.m3e&completed=1
&serialNo=K333530&entityID=_EID_&type=MUSICCLIP&file=leddra%20chapman%20-%20summer%20song.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:14 +0100] "GET /kvAutoUpdate/hawkSuccessful.php?id=210271&serialNo=K294230&success=NO&completed=NO&action=2&datestart=2
HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:14 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/clips/mp3/everything%20everything%20-%20sc_0.m3e&completed=1&s
erialNo=K329030&entityID=_EID_&type=MUSICCLIP&file=everything%20everything%20-%20sc_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:15 +0100] "GET /kvAutoUpdate/hawkSuccessful.php?id=210425&serialNo=K327530&success=YES&completed=YES&action=1 HTTP/1.1"
200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:15 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/Kaleidovision/SharedLocal/KvAutoUpdate/LatestImage-20100710062
943.1252.xml&completed=1&serialNo=K332330&entityID=_EID_&type=UNKNOWN&file= HTTP/1.1" 200 4 "-" "-"
192.168.0.212 - - [11/Jul/2010:07:48:15 +0100] "GET /KvAutoPublish/KvAutoPublish.Rsync.FullList.php HTTP/1.1" 200 9900 "http://192.168.10.39/KvAutoPublish/Kv
AutoPublish.Rsync.Status.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6 (.NET CLR 3.5.30729)"
81.171.153.68 - - [11/Jul/2010:07:48:15 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=D:/clips/mp3/jack%20johnson%20-%20you%20and%20you_0.m3e&completed
=1&serialNo=K318430&entityID=_EID_&type=MUSICCLIP&file=jack%20johnson%20-%20you%20and%20you_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:15 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=D:/clips/mp3/marina%20_%20the%20diamonds%20-%20oh_0.m3e&completed
=1&serialNo=K214330&entityID=_EID_&type=MUSICCLIP&file=marina%20_%20the%20diamonds%20-%20oh_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:15 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=D:/clips/mp3/dan%20balan%20-%20chica%20bomb%20(uk_0.m3e&completed
=1&serialNo=K318030&entityID=_EID_&type=MUSICCLIP&file=dan%20balan%20-%20chica%20bomb%20(uk_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:16 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/clips/mp3/marina%20_%20the%20diamonds%20-%20oh_0.m3e&completed
=1&serialNo=K333530&entityID=_EID_&type=MUSICCLIP&file=marina%20_%20the%20diamonds%20-%20oh_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:16 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/Kaleidovision/SharedLocal/KvAutoUpdate/LatestImage-20100707062
950.1560.xml&completed=1&serialNo=K336030&entityID=_EID_&type=UNKNOWN&file= HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:16 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/Kaleidovision/SharedLocal/KvAutoUpdate/LatestImage-20100709063
011.1500.xml&completed=1&serialNo=K294230&entityID=_EID_&type=UNKNOWN&file= HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:16 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/clips/mp3/kate%20nash%20-%20kiss%20that%20grrrl.m3e&completed=
1&serialNo=K329030&entityID=_EID_&type=MUSICCLIP&file=kate%20nash%20-%20kiss%20that%20grrrl.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:17 +0100] "GET /kvAutoUpdate/hawkSuccessful.php?id=210558&serialNo=K329430&success=YES&completed=YES&action=1 HTTP/1.1"
200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:17 +0100] "GET /kvAutoUpdate/hawkSuccessful.php?id=210278&serialNo=K221430&success=YES&completed=YES&action=1 HTTP/1.1"
200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:17 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/clips/mp3/scouting%20for%20girls%20-%20famous.m3e&completed=1&
serialNo=K332330&entityID=_EID_&type=MUSICCLIP&file=scouting%20for%20girls%20-%20famous.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:17 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=D:/clips/mp3/30h_3%20-%20my%20first%20kiss-1.m3e&completed=1&seri
alNo=K318430&entityID=_EID_&type=MUSICCLIP&file=30h_3%20-%20my%20first%20kiss-1.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:17 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=D:/clips/mp3/scouting%20for%20girls%20-%20famous.m3e&completed=1&
:
Are you hosting all these music clips?
D:/clips/mp3/scouting%20fo
kate%20nash%20-%20kiss%20t
D:/clips/mp3/dan%20balan%2
All the requests appear to be coming from 81.171.153.68- is that one if your addresses?
D:/clips/mp3/scouting%20fo
kate%20nash%20-%20kiss%20t
D:/clips/mp3/dan%20balan%2
All the requests appear to be coming from 81.171.153.68- is that one if your addresses?
ASKER
We are hosting them.
81.171.153.68 is one of our clients. It's strange how the
tail -100 /var/log/apache2/access.lo
comand is only bringing up requests from 81.171.153.68. We have thousands of clients and checking netstat shows alot of them connecting. Why is the above IP the only one showing in the apache access log?
81.171.153.68 is one of our clients. It's strange how the
tail -100 /var/log/apache2/access.lo
comand is only bringing up requests from 81.171.153.68. We have thousands of clients and checking netstat shows alot of them connecting. Why is the above IP the only one showing in the apache access log?
tail -100 gives the last 100 lines
then the 'less' commands paginates the output, so it doesn't scroll past you.
perhaps the 60 lines above were all connections from the one server.
try again run
tail -50 /var/log/apache2/access.lo
a few time to see if you only see 1 IP in it.
What's in the /var/log/apache2/error.log
what's the output of
vmstat 1 10
if iostat is installed try
iostat -d
and
iostat -dx
then the 'less' commands paginates the output, so it doesn't scroll past you.
perhaps the 60 lines above were all connections from the one server.
try again run
tail -50 /var/log/apache2/access.lo
a few time to see if you only see 1 IP in it.
What's in the /var/log/apache2/error.log
what's the output of
vmstat 1 10
if iostat is installed try
iostat -d
and
iostat -dx
ASKER
same IP, also data seems 40mins old or more.?
error.log
[Sun Jul 11 06:25:08 2010] [notice] mod_python: Creating 8 session mutexes based on 250 max processes and 0 max threads.
[Sun Jul 11 06:25:08 2010] [notice] mod_python: using mutex_directory /tmp
[Sun Jul 11 06:25:09 2010] [notice] Apache/2.2.3 (Debian) mod_python/3.2.10 Python/2.4.4 PHP/5.2.0-8+etch13 mod_ssl/2.2.3 OpenSSL/0.9.8c mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Sun Jul 11 06:27:47 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting
* About to connect() to 192.168.0.51 port 80
* Trying 192.168.0.51... * connected
* Connected to 192.168.0.51 (192.168.0.51) port 80
> GET //dashboard_production/ext
Host: 192.168.0.51^M
Accept: */*^M
^M
< HTTP/1.1 200 OK^M
< Date: Sun, 11 Jul 2010 11:12:34 GMT^M
< Server: Apache/2.2.3 (Debian) mod_python/3.2.10 Python/2.4.4 PHP/5.2.0-8+etch16 mod_ssl/2.2.3 OpenSSL/0.9.8c mod_perl/2.0.2 Perl/v5.8.8^M
< X-Powered-By: PHP/5.2.0-8+etch16^M
< Cache-Control: no-store, no-cache, must-revalidate^M
< Cache-Control: post-check=0, pre-check=0^M
< Pragma: no-cache^M
< Content-Length: 104^M
< Content-Type: text/xml;charset=UTF-8^M
* Connection #0 to host 192.168.0.51 left intact
* Closing connection #0
[Sun Jul 11 12:24:47 2010] [error] [client 221.11.4.190] File does not exist: /var/www/htdocs/manager
[Sun Jul 11 12:51:31 2010] [error] [client 192.168.0.112] File does not exist: /var/www/htdocs/img/ajax-l
[Sun Jul 11 18:10:22 2010] [error] [client 92.240.68.153] File does not exist: /var/www/htdocs/data, referer: http://random.yahoo.com/fast/ryl
[Mon Jul 12 09:37:34 2010] [error] [client 192.168.0.112] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:28:49 2010] [error] [client 192.168.0.112] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:46:24 2010] [error] [client 192.168.0.150] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:46:28 2010] [error] [client 192.168.0.112] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:47:46 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:47:56 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:48:14 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:48:18 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:48:21 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:48:32 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:48:44 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:48:48 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:49:03 2010] [error] [client 192.168.0.150] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:57:21 2010] [notice] mod_python: Creating 8 session mutexes based on 250 max processes and 0 max threads.
[Mon Jul 12 10:57:21 2010] [notice] mod_python: using mutex_directory /tmp
[Mon Jul 12 10:57:21 2010] [notice] Apache/2.2.3 (Debian) mod_python/3.2.10 Python/2.4.4 PHP/5.2.0-8+etch13 mod_ssl/2.2.3 OpenSSL/0.9.8c mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Mon Jul 12 10:59:29 2010] [error] [client 192.168.0.150] File does not exist: /var/www/htdocs/img/ajax-l
vmstat 1 10
Mars:~# vmstat 1 10
procs -----------memory---------
r b swpd free buff cache si so bi bo in cs us sy id wa
5 0 0 1576156 54748 2180560 0 0 2 13 450 415 58 5 37 0
1 0 0 1575996 54748 2180580 0 0 0 0 2046 1472 68 5 27 0
4 0 0 1574368 54748 2181020 0 0 0 0 2253 1568 71 4 24 0
4 0 0 1573236 54748 2181020 0 0 0 0 1959 1275 73 4 23 0
10 0 0 1569888 54756 2181452 0 0 0 68 2283 1628 73 5 22 0
1 0 0 1569144 54764 2181924 0 0 0 56 2096 1738 63 6 31 0
5 0 0 1568904 54764 2181924 0 0 0 0 1804 1390 62 4 34 0
10 0 0 1567880 54764 2181964 0 0 0 0 1970 1456 64 5 31 0
8 0 0 1569276 54764 2181964 0 0 0 0 1844 1358 61 3 36 0
5 0 0 1569632 54772 2181956 0 0 0 148 2050 1549 69 5 27 0
Mars:~#
error.log
[Sun Jul 11 06:25:08 2010] [notice] mod_python: Creating 8 session mutexes based on 250 max processes and 0 max threads.
[Sun Jul 11 06:25:08 2010] [notice] mod_python: using mutex_directory /tmp
[Sun Jul 11 06:25:09 2010] [notice] Apache/2.2.3 (Debian) mod_python/3.2.10 Python/2.4.4 PHP/5.2.0-8+etch13 mod_ssl/2.2.3 OpenSSL/0.9.8c mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Sun Jul 11 06:27:47 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting
* About to connect() to 192.168.0.51 port 80
* Trying 192.168.0.51... * connected
* Connected to 192.168.0.51 (192.168.0.51) port 80
> GET //dashboard_production/ext
Host: 192.168.0.51^M
Accept: */*^M
^M
< HTTP/1.1 200 OK^M
< Date: Sun, 11 Jul 2010 11:12:34 GMT^M
< Server: Apache/2.2.3 (Debian) mod_python/3.2.10 Python/2.4.4 PHP/5.2.0-8+etch16 mod_ssl/2.2.3 OpenSSL/0.9.8c mod_perl/2.0.2 Perl/v5.8.8^M
< X-Powered-By: PHP/5.2.0-8+etch16^M
< Cache-Control: no-store, no-cache, must-revalidate^M
< Cache-Control: post-check=0, pre-check=0^M
< Pragma: no-cache^M
< Content-Length: 104^M
< Content-Type: text/xml;charset=UTF-8^M
* Connection #0 to host 192.168.0.51 left intact
* Closing connection #0
[Sun Jul 11 12:24:47 2010] [error] [client 221.11.4.190] File does not exist: /var/www/htdocs/manager
[Sun Jul 11 12:51:31 2010] [error] [client 192.168.0.112] File does not exist: /var/www/htdocs/img/ajax-l
[Sun Jul 11 18:10:22 2010] [error] [client 92.240.68.153] File does not exist: /var/www/htdocs/data, referer: http://random.yahoo.com/fast/ryl
[Mon Jul 12 09:37:34 2010] [error] [client 192.168.0.112] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:28:49 2010] [error] [client 192.168.0.112] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:46:24 2010] [error] [client 192.168.0.150] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:46:28 2010] [error] [client 192.168.0.112] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:47:46 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:47:56 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:48:14 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:48:18 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:48:21 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:48:32 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:48:44 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:48:48 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:49:03 2010] [error] [client 192.168.0.150] File does not exist: /var/www/htdocs/img/ajax-l
[Mon Jul 12 10:57:21 2010] [notice] mod_python: Creating 8 session mutexes based on 250 max processes and 0 max threads.
[Mon Jul 12 10:57:21 2010] [notice] mod_python: using mutex_directory /tmp
[Mon Jul 12 10:57:21 2010] [notice] Apache/2.2.3 (Debian) mod_python/3.2.10 Python/2.4.4 PHP/5.2.0-8+etch13 mod_ssl/2.2.3 OpenSSL/0.9.8c mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Mon Jul 12 10:59:29 2010] [error] [client 192.168.0.150] File does not exist: /var/www/htdocs/img/ajax-l
vmstat 1 10
Mars:~# vmstat 1 10
procs -----------memory---------
r b swpd free buff cache si so bi bo in cs us sy id wa
5 0 0 1576156 54748 2180560 0 0 2 13 450 415 58 5 37 0
1 0 0 1575996 54748 2180580 0 0 0 0 2046 1472 68 5 27 0
4 0 0 1574368 54748 2181020 0 0 0 0 2253 1568 71 4 24 0
4 0 0 1573236 54748 2181020 0 0 0 0 1959 1275 73 4 23 0
10 0 0 1569888 54756 2181452 0 0 0 68 2283 1628 73 5 22 0
1 0 0 1569144 54764 2181924 0 0 0 56 2096 1738 63 6 31 0
5 0 0 1568904 54764 2181924 0 0 0 0 1804 1390 62 4 34 0
10 0 0 1567880 54764 2181964 0 0 0 0 1970 1456 64 5 31 0
8 0 0 1569276 54764 2181964 0 0 0 0 1844 1358 61 3 36 0
5 0 0 1569632 54772 2181956 0 0 0 148 2050 1549 69 5 27 0
Mars:~#
Mars:~# tail -50 /var/log/apache2/access.log
81.171.153.68 - - [12/Jul/2010:14:25:44 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=D:/clips/mp3/eminem%20-%20love%20the%20way%20you%20_0.m3e&completed=1&seri alNo=K318430&entityID=_EID_&type=MUSICCLIP&file=eminem%20-%20love%20the%20way%20you %20_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:44 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=D:/clips/mp3/kylie%20-%20all%20the%20lovers-1.m3e&completed=1&serialNo=K31 8030&entityID=_EID_&type=MUSICCLIP&file=kylie%20-%20all%20the%20lovers-1.m3e HTTP/1 .1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:44 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=D:/clips/mp3/the%20drums%20-%20forever%20and%20ev_0.m3e&completed=1&serial No=K294230&entityID=_EID_&type=MUSICCLIP&file=the%20drums%20-%20forever%20and%20ev_ 0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:44 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/20575_pricing_thu_e%201750.mpe&completed=1&serialNo=K333530&entit yID=_EID_&type=GFXCLIP-BSM&file=20575_pricing_thu_e%201750.mpe HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:44 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=D:/clips/mp3/everything%20everything%20-%20sc_0.m3e&completed=1&serialNo=K 318430&entityID=_EID_&type=MUSICCLIP&file=everything%20everything%20-%20sc_0.m3e HT TP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:44 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/mp3/futureheads%20-%20i%20can%20do%20that.m3e&completed=1&serialN o=K328530&entityID=_EID_&type=MUSICCLIP&file=futureheads%20-%20i%20can%20do%20that. m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:44 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=D:/clips/mp3/m.i.a%20-%20xxxo%20-%20featuring%20j_0.m3e&completed=1&serial No=K318030&entityID=_EID_&type=MUSICCLIP&file=m.i.a%20-%20xxxo%20-%20featuring%20j_ 0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=D:/clips/mp3/usher%20-%20lil'%20freak%20(feat.%20_0.m3e&completed=1&serial No=K294230&entityID=_EID_&type=MUSICCLIP&file=usher%20-%20lil'%20freak%20(feat.%20_ 0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=D:/clips/mp3/futureheads%20-%20i%20can%20do%20that.m3e&completed=1&serialN o=K318430&entityID=_EID_&type=MUSICCLIP&file=futureheads%20-%20i%20can%20do%20that. m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/20575_pricing_tues_a%201250.mpe&completed=1&serialNo=K333530&enti tyID=_EID_&type=GFXCLIP-BSM&file=20575_pricing_tues_a%201250.mpe HTTP/1.1" 200 4 "- " "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=D:/clips/mp3/flo-rida%20feat.%20david%20guett_0.m3e&completed=1&serialNo=K 214330&entityID=_EID_&type=MUSICCLIP&file=flo-rida%20feat.%20david%20guett_0.m3e HT TP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/mp3/joshua%20radin%20-%20they%20bring%20_0.m3e&completed=1&serial No=K330930&entityID=_EID_&type=MUSICCLIP&file=joshua%20radin%20-%20they%20bring%20_ 0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/mp3/joshua%20radin%20-%20brand%20new%20d_0.m3e&completed=1&serial No=K328530&entityID=_EID_&type=MUSICCLIP&file=joshua%20radin%20-%20brand%20new%20d_ 0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=D:/clips/mp3/lcd%20soundsystem%20-%20i%20can%20ch_0.m3e&completed=1&serial No=K318030&entityID=_EID_&type=MUSICCLIP&file=lcd%20soundsystem%20-%20i%20can%20ch_ 0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=D:/clips/mp3/joshua%20radin%20-%20brand%20new%20d_0.m3e&completed=1&serial No=K318430&entityID=_EID_&type=MUSICCLIP&file=joshua%20radin%20-%20brand%20new%20d_ 0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/20575_pricing_tues_a%201750.mpe&completed=1&serialNo=K333530&enti tyID=_EID_&type=GFXCLIP-BSM&file=20575_pricing_tues_a%201750.mpe HTTP/1.1" 200 4 "- " "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/mp3/we%20are%20scientists%20-%20nice%20g_0.m3e&completed=1&serial No=K332830&entityID=_EID_&type=MUSICCLIP&file=we%20are%20scientists%20-%20nice%20g_ 0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=D:/clips/mp3/jls%20-%20the%20club%20is%20alive-1.m3e&completed=1&serialNo= K214330&entityID=_EID_&type=MUSICCLIP&file=jls%20-%20the%20club%20is%20alive-1.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/mp3/king%20blues%20-%20headbutt.m3e&completed=1&serialNo=K330930& entityID=_EID_&type=MUSICCLIP&file=king%20blues%20-%20headbutt.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/mp3/katy%20perry%20-%20california%20gu_0.m3e&completed=1&serialNo =K328530&entityID=_EID_&type=MUSICCLIP&file=katy%20perry%20-%20california%20gu_0.m3 e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=D:/clips/mp3/kate%20nash%20-%20kiss%20that%20grrrl.m3e&completed=1&serialN o=K318430&entityID=_EID_&type=MUSICCLIP&file=kate%20nash%20-%20kiss%20that%20grrrl. m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/20575_pricing_tues_e%201250.mpe&completed=1&serialNo=K333530&enti tyID=_EID_&type=GFXCLIP-BSM&file=20575_pricing_tues_e%201250.mpe HTTP/1.1" 200 4 "- " "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=D:/clips/mp3/kylie%20-%20all%20the%20lovers-1.m3e&completed=1&serialNo=K21 4330&entityID=_EID_&type=MUSICCLIP&file=kylie%20-%20all%20the%20lovers-1.m3e HTTP/1 .1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/mp3/1miley%20cyrus%20-%20can't%20be%20ta_0.m3e&completed=1&serial No=K339730&entityID=_EID_&type=MUSICCLIP&file=1miley%20cyrus%20-%20can't%20be%20ta_ 0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=D:/clips/mp3/katy%20perry%20-%20california%20gu_0.m3e&completed=1&serialNo =K318430&entityID=_EID_&type=MUSICCLIP&file=katy%20perry%20-%20california%20gu_0.m3 e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:46 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/20575_pricing_tues_e%201750.mpe&completed=1&serialNo=K333530&enti tyID=_EID_&type=GFXCLIP-BSM&file=20575_pricing_tues_e%201750.mpe HTTP/1.1" 200 4 "- " "-"
81.171.153.68 - - [12/Jul/2010:14:25:46 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=D:/clips/mp3/robbie%20rivera%20-%20let%20me%20sip_0.m3e&completed=1&serial No=K214330&entityID=_EID_&type=MUSICCLIP&file=robbie%20rivera%20-%20let%20me%20sip_ 0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:46 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/20575_pricing_wed_a%201250.mpe&completed=1&serialNo=K333530&entit yID=_EID_&type=GFXCLIP-BSM&file=20575_pricing_wed_a%201250.mpe HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:46 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=D:/clips/mp3/the%20coral%20-%201000%20years.m3e&completed=1&serialNo=K2143 30&entityID=_EID_&type=MUSICCLIP&file=the%20coral%20-%201000%20years.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:46 +0100] "GET /kvAutoUpdate/hawkSuccessful.ph p?id=210277&serialNo=K214330&success=YES&completed=YES&action=1 HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:47 +0100] "GET /kvAutoUpdate/manifest_hawk.php ?winversion=1.3.1.0&version=ServeTheServants&dated=2010-04-09-1511&serialNo=K334930 &entityID=2291 HTTP/1.0" 200 111 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:47 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/mp3/david%20guetta%20(fergie)%20-%20ge_0.m3e&completed=1&serialNo =K329030&entityID=_EID_&type=MUSICCLIP&file=david%20guetta%20(fergie)%20-%20ge_0.m3 e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:48 +0100] "GET /kvAutoUpdate/hawkSuccessful.ph p?id=210276&serialNo=K207830&success=YES&completed=YES&action=1 HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:48 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/mp3/erik%20hassle%20-%20isnt%20it%20obvi_0.m3e&completed=1&serial No=K339730&entityID=_EID_&type=MUSICCLIP&file=erik%20hassle%20-%20isnt%20it%20obvi_ 0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:48 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/mp3/everything%20everything%20-%20sc_0.m3e&completed=1&serialNo=K 329030&entityID=_EID_&type=MUSICCLIP&file=everything%20everything%20-%20sc_0.m3e HT TP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:48 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/mp3/kate%20nash%20-%20kiss%20that%20grrrl.m3e&completed=1&serialN o=K329030&entityID=_EID_&type=MUSICCLIP&file=kate%20nash%20-%20kiss%20that%20grrrl. m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:48 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/mp3/1lena%20-%20satellite.m3e&completed=1&serialNo=K339730&entity ID=_EID_&type=MUSICCLIP&file=1lena%20-%20satellite.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:48 +0100] "GET /kvAutoUpdate/hawkSuccessful.ph p?id=210577&serialNo=K203730&success=YES&completed=YES&action=1 HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:49 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/mp3/30h_3%20-%20my%20first%20kiss-1.m3e&completed=1&serialNo=K339 730&entityID=_EID_&type=MUSICCLIP&file=30h_3%20-%20my%20first%20kiss-1.m3e HTTP/1.1 " 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:49 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/mp3/lcd%20soundsystem%20-%20i%20can%20ch_0.m3e&completed=1&serial No=K329030&entityID=_EID_&type=MUSICCLIP&file=lcd%20soundsystem%20-%20i%20can%20ch_ 0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:49 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/mp3/i%20am%20arrows%20-%20green%20grass-1.m3e&completed=1&serialN o=K332730&entityID=_EID_&type=MUSICCLIP&file=i%20am%20arrows%20-%20green%20grass-1. m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:49 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/mp3/adam%20lambert%20-%20whataya%20wan_0.m3e&completed=1&serialNo =K339730&entityID=_EID_&type=MUSICCLIP&file=adam%20lambert%20-%20whataya%20wan_0.m3 e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:49 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/mp3/scissor%20-%20fire%20with%20fire%20(_0.m3e&completed=1&serial No=K329030&entityID=_EID_&type=MUSICCLIP&file=scissor%20-%20fire%20with%20fire%20(_ 0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:49 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/mp3/katy%20perry%20-%20california%20gu_0.m3e&completed=1&serialNo =K332730&entityID=_EID_&type=MUSICCLIP&file=katy%20perry%20-%20california%20gu_0.m3 e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:49 +0100] "GET /kvAutoUpdate/manifest_hawk.php ?winversion=1.3.1.0&version=ServeTheServants&dated=2010-04-09-1511&serialNo=K335930 &entityID=2323 HTTP/1.0" 200 111 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:49 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/mp3/angie%20stone%20-%20free-1.m3e&completed=1&serialNo=K339730&e ntityID=_EID_&type=MUSICCLIP&file=angie%20stone%20-%20free-1.m3e HTTP/1.1" 200 4 "- " "-"
81.171.153.68 - - [12/Jul/2010:14:25:49 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/mp3/two%20door%20cinema%20-%20somethin_0.m3e&completed=1&serialNo =K329030&entityID=_EID_&type=MUSICCLIP&file=two%20door%20cinema%20-%20somethin_0.m3 e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:49 +0100] "GET /kvAutoUpdate/downloadFile.php? destfile=C:/clips/mp3/robbie%20rivera%20-%20let%20me%20sip_0.m3e&completed=1&serial No=K332730&entityID=_EID_&type=MUSICCLIP&file=robbie%20rivera%20-%20let%20me%20sip_ 0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:50 +0100] "GET /kvAutoUpdate/hawkSuccessful.ph p?id=210426&serialNo=K332730&success=YES&completed=YES&action=1 HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:50 +0100] "GET /kvAutoUpdate/downloadFiMars:~#
The vmstat shows the run queue (r) greater than 1 - this is not good:
The runqueue value shows the number of tasks executing and waiting for CPU resources. When this number exceeds the number of CPUs on the server, a CPU bottleneck exists, and some tasks are waiting for execution.
You appear to be running out of a resource, probably disk or network, can you confirm your NIC is set to the correct speed and duplex for your system, sometimes if it's set to autonegotiate, it defaults down to a lower speed or duplex and you get network probs. (ethtool eth0, assuming your NIC is eth0)
I notice the error from your error.log
[error] server reached MaxClients setting, consider raising the MaxClients setting
* About to connect() to 192.168.0.51 port 80
What is your MaxClients set to? (in your apache httpd.conf file)
The runqueue value shows the number of tasks executing and waiting for CPU resources. When this number exceeds the number of CPUs on the server, a CPU bottleneck exists, and some tasks are waiting for execution.
You appear to be running out of a resource, probably disk or network, can you confirm your NIC is set to the correct speed and duplex for your system, sometimes if it's set to autonegotiate, it defaults down to a lower speed or duplex and you get network probs. (ethtool eth0, assuming your NIC is eth0)
I notice the error from your error.log
[error] server reached MaxClients setting, consider raising the MaxClients setting
* About to connect() to 192.168.0.51 port 80
What is your MaxClients set to? (in your apache httpd.conf file)
ASKER
Hi, Thank for this help.
Mars:~# ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Link detected: yes
MaxClients is set to 250 in apache2.conf (httpd.conf is empty?)
Mars:~# ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Link detected: yes
MaxClients is set to 250 in apache2.conf (httpd.conf is empty?)
ASKER
More information
Mars:~# iostat -d
Linux 2.6.18-6-amd64 (Mars) 12/07/10 _x86_64_
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 30.91 317.44 567.25 5749384 10273984
Mars:~# iostat -dx
Linux 2.6.18-6-amd64 (Mars) 12/07/10 _x86_64_
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.89 41.62 1.65 29.23 317.19 566.86 28.62 0.03 1.02 0.19 0.58
Mars:~#
You can try and up the MaxClient in your apache2.conf (then restart apache) to see if they clears the problem - or takes it longer to fail
I would always force NICs not to autoneg, but you should probably talk to your network people to see how they've got it. But it's reporting 1000fdx so looks fine
Can you shut down apache and re-run the iostat -dx when apache is down to see what difference there is, to help see if it's disk or network bottlenecking
I would always force NICs not to autoneg, but you should probably talk to your network people to see how they've got it. But it's reporting 1000fdx so looks fine
Can you shut down apache and re-run the iostat -dx when apache is down to see what difference there is, to help see if it's disk or network bottlenecking
ASKER
with apache2 stopped.
Mars:~# iostat -dx
Linux 2.6.18-6-amd64 (Mars) 12/07/10 _x86_64_
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.82 38.49 1.55 26.77 289.52 522.07 28.65 0.03 1.02 0.19 0.54
So looking at that it doesn't appear to be a disk I/o problem and as your running at 1000fdx, I doubt it's a network bottleneck, can you try upping the MaxClient
ASKER
upped MaxClient to 350 and got this upon apache2 restart.
Mars:/etc/apache2# /etc/init.d/apache2 restart
Forcing reload of web server (apache2)...WARNING: MaxClients of 350 exceeds ServerLimit value of 256 servers,
lowering MaxClients to 256. To increase, please see the ServerLimit
directive.
waiting WARNING: MaxClients of 350 exceeds ServerLimit value of 256 servers,
lowering MaxClients to 256. To increase, please see the ServerLimit
directive.
Yes, sorry added that in the next comment, up them both to 1000 and give it a go....
ASKER
added
ServerLimit 350
to apache2.conf but a restart spews out the same error?
ServerLimit 350
to apache2.conf but a restart spews out the same error?
did you restart apache or stop and start it? - there appears to be a gotcha (http://northernmost.org/blog/apache-gotcha-with-serverlimit/) a restart won't do it
ASKER
stopped it and started it.
Same error unfortunately.
Same error unfortunately.
Do you analyze your Apache log files (awstats / analog / webalizer), monitor your site through one of the Web analytic providers (Google Analytics / Omniture) or monitor system resources (cacti / nagios)?
If so have you noticed any change in the number of the requests, or the timing of the peek load over the last week?
If not, and you have a few weeks worth of log data, have a look at one of the log analyzers, and possibly setting up Cacti on a separate server / PC to monitor load, network throughput and Apache hits.
If so have you noticed any change in the number of the requests, or the timing of the peek load over the last week?
If not, and you have a few weeks worth of log data, have a look at one of the log analyzers, and possibly setting up Cacti on a separate server / PC to monitor load, network throughput and Apache hits.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Open in new window