We help IT Professionals succeed at work.

High load average all of a sudden - Apache2.

1,378 Views
Last Modified: 2013-12-06
Hello,

We have a Debian server running apache2. Our load average used to be below 1.0, however this week it is now at anywhere between 5-10.

It is causing real big problems. It is our front end clinet facing HTTP server. Does anyone have any advice? If I disable the apache2 service, the load average drops back down to an acceptable level, re-enabling it rockets the load average back up.

I'm not extremely well versed in linux so be gentle!

Thanks,
Comment
Watch Question

Author

Commented:
more info
top - 12:25:42 up  1:24,  2 users,  load average: 6.23, 6.38, 5.99
Tasks: 132 total,   2 running, 129 sleeping,   0 stopped,   1 zombie
Cpu(s): 48.9%us,  3.7%sy,  0.0%ni, 47.2%id,  0.0%wa,  0.1%hi,  0.2%si,  0.0%st
Mem:   4045104k total,   746116k used,  3298988k free,    21468k buffers
Swap:  1951888k total,        0k used,  1951888k free,   589548k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4762 www-data  16   0  117m  10m 3528 S    8  0.3   0:01.68 apache2
 5845 www-data  16   0  117m 9.9m 3372 S    8  0.3   0:00.60 apache2
30025 www-data  15   0  117m  12m 5316 S    8  0.3   0:09.91 apache2
 1758 www-data  15   0  117m  10m 3900 S    8  0.3   0:04.70 apache2
 3916 www-data  15   0  117m  10m 3600 S    8  0.3   0:02.85 apache2
  903 www-data  15   0  117m  10m 3852 S    8  0.3   0:02.53 apache2
27227 www-data  16   0  117m  11m 4796 S    7  0.3   0:11.85 apache2
27308 www-data  15   0  117m  10m 4024 S    7  0.3   0:12.12 apache2
 8208 www-data  15   0  117m  10m 4216 S    7  0.3   0:26.56 apache2
 4014 www-data  15   0  117m  10m 3776 S    7  0.3   0:02.52 apache2
 4197 www-data  15   0     0    0    0 Z    7  0.0   0:02.35 apache2 <defunct>
 5937 www-data  15   0  117m 9.9m 3344 S    7  0.2   0:00.41 apache2
22952 www-data  15   0  117m  12m 5364 S    7  0.3   0:15.06 apache2
 1593 www-data  15   0  116m  10m 3732 S    7  0.3   0:04.34 apache2
 3494 www-data  15   0  117m  10m 3760 S    7  0.3   0:02.88 apache2
28638 www-data  15   0  117m  12m 5304 S    6  0.3   0:10.16 apache2
30262 www-data  15   0  117m  10m 4152 S    6  0.3   0:07.92 apache2
 5843 www-data  15   0  116m 9.9m 3352 S    6  0.2   0:00.59 apache2
 1148 www-data  15   0  116m  10m 3912 S    6  0.3   0:05.84 apache2
 3416 www-data  16   0  116m  10m 3604 S    6  0.3   0:03.35 apache2
 5935 www-data  15   0  117m 9.9m 3352 R    6  0.2   0:00.40 apache2
 5950 www-data  15   0  116m 9.9m 3340 S    6  0.2   0:00.41 apache2
30622 www-data  15   0  117m  12m 5296 S    6  0.3   0:08.80 apache2
28334 www-data  16   0  117m  10m 4120 S    5  0.3   0:09.82 apache2
 3641 www-data  16   0  116m  10m 3564 S    5  0.3   0:02.70 apache2
14353 www-data  16   0  117m  10m 4172 S    5  0.3   0:24.37 apache2
27226 www-data  15   0  117m  10m 4212 S    5  0.3   0:12.14 apache2
25541 www-data  15   0  117m  12m 5504 S    4  0.3   0:14.05 apache2
 3495 www-data  15   0  117m  10m 3792 S    3  0.3   0:03.01 apache2
 5578 www-data  15   0  117m 9.9m 3392 S    2  0.3   0:00.72 apache2
 5764 www-data  15   0  116m 9.9m 3380 S    2  0.3   0:00.52 apache2
    1 root      15   0  6124  692  568 S    0  0.0   0:02.01 init
    2 root      RT   0     0    0    0 S    0  0.0   0:00.19 migration/0
    3 root      34  19     0    0    0 S    0  0.0   0:00.00 ksoftirqd/0
    4 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/0
    5 root      RT   0     0    0    0 S    0  0.0   0:00.09 migration/1
    6 root      34  19     0    0    0 S    0  0.0   0:00.00 ksoftirqd/1
    7 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/1
    8 root      RT   0     0    0    0 S    0  0.0   0:00.06 migration/2
    9 root      34  19     0    0    0 S    0  0.0   0:00.00 ksoftirqd/2
   10 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/2
   11 root      RT   0     0    0    0 S    0  0.0   0:00.12 migration/3
   12 root      34  19     0    0    0 S    0  0.0   0:00.00 ksoftirqd/3
   13 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/3
   14 root      10  -5     0    0    0 S    0  0.0   0:00.02 events/0
   15 root      10  -5     0    0    0 S    0  0.0   0:00.01 events/1
   16 root      10  -5     0    0    0 S    0  0.0   0:00.06 events/2
   17 root      10  -5     0    0    0 S    0  0.0   0:00.00 events/3
   18 root      10  -5     0    0    0 S    0  0.0   0:00.01 khelper
   19 root      11  -5     0    0    0 S    0  0.0   0:00.00 kthread
   26 root      10  -5     0    0    0 S    0  0.0   0:00.02 kblockd/0

Open in new window

CERTIFIED EXPERT
Top Expert 2007

Commented:
It's more than likely that Apache is getting hit relatively hard by someone (could be legitimate or malicious).

Do a

tail /var/log/apache2/error.log

to see if you can see anything that looks unusual.

Also do

tail -100 /var/log/apache2/access.log | less

to view the last 100 lines of the Apache access log to see if you can see anything unusual, eg: multiple requests all coming from the same IP or a particular bot hitting your site.

Author

Commented:
I did the tail -100.

And the following came back. Bizarre considering the debians system time is.

Mars:~# date
Mon Jul 12 13:21:47 BST 2010

Mars:~# tail -100 /var/log/apache2/access.log | less
81.171.153.68 - - [11/Jul/2010:07:48:11 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=D:/clips/mp3/flo-rida%20feat.%20david%20guett_0.m3e&completed=1&s
erialNo=K214330&entityID=_EID_&type=MUSICCLIP&file=flo-rida%20feat.%20david%20guett_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:12 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/clips/mp3/kylie%20-%20all%20the%20lovers-1.m3e&completed=1&ser
ialNo=K333530&entityID=_EID_&type=MUSICCLIP&file=kylie%20-%20all%20the%20lovers-1.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:12 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/clips/mp3/travie%20mccoy%20-billionaire%20_0.m3e&completed=1&s
erialNo=K332730&entityID=_EID_&type=MUSICCLIP&file=travie%20mccoy%20-billionaire%20_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:12 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/clips/mp3/game%20feat.%20jt%20-%20ain't%20no%20d_0.m3e&complet
ed=1&serialNo=K329030&entityID=_EID_&type=MUSICCLIP&file=game%20feat.%20jt%20-%20ain't%20no%20d_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:12 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/clips/mp3/pendulum%20-%20witchcraft.m3e&completed=1&serialNo=K
327530&entityID=_EID_&type=MUSICCLIP&file=pendulum%20-%20witchcraft.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:13 +0100] "GET /kvAutoUpdate/hawkSuccessful.php?id=210065&serialNo=K332330&success=NO&completed=NO&action=2&datestart=2
HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:13 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/Kaleidovision/SharedLocal/KvAutoUpdate/LatestImage-20100709062
946.1440.xml&completed=1&serialNo=K318430&entityID=_EID_&type=UNKNOWN&file= HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:13 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/clips/mp3/we%20are%20scientists%20-%20nice%20g_0.m3e&completed
=1&serialNo=K332830&entityID=_EID_&type=MUSICCLIP&file=we%20are%20scientists%20-%20nice%20g_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:13 +0100] "GET /kvAutoUpdate/hawkSuccessful.php?id=210160&serialNo=K329730&success=NO&completed=YES&action=1 HTTP/1.1" 2
00 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:13 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=D:/clips/mp3/kanye%20west%20-power%20(super%20c_0.m3e&completed=1
&serialNo=K214330&entityID=_EID_&type=MUSICCLIP&file=kanye%20west%20-power%20(super%20c_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:14 +0100] "GET /kvAutoUpdate/hawkSuccessful.php?id=210142&serialNo=K329630&success=NO&completed=YES&action=1 HTTP/1.1" 2
00 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:14 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=D:/clips/mp3/game%20feat.%20jt%20-%20ain't%20no%20d_0.m3e&complet
ed=1&serialNo=K318030&entityID=_EID_&type=MUSICCLIP&file=game%20feat.%20jt%20-%20ain't%20no%20d_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:14 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/clips/mp3/leddra%20chapman%20-%20summer%20song.m3e&completed=1
&serialNo=K333530&entityID=_EID_&type=MUSICCLIP&file=leddra%20chapman%20-%20summer%20song.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:14 +0100] "GET /kvAutoUpdate/hawkSuccessful.php?id=210271&serialNo=K294230&success=NO&completed=NO&action=2&datestart=2
HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:14 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/clips/mp3/everything%20everything%20-%20sc_0.m3e&completed=1&s
erialNo=K329030&entityID=_EID_&type=MUSICCLIP&file=everything%20everything%20-%20sc_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:15 +0100] "GET /kvAutoUpdate/hawkSuccessful.php?id=210425&serialNo=K327530&success=YES&completed=YES&action=1 HTTP/1.1"
200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:15 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/Kaleidovision/SharedLocal/KvAutoUpdate/LatestImage-20100710062
943.1252.xml&completed=1&serialNo=K332330&entityID=_EID_&type=UNKNOWN&file= HTTP/1.1" 200 4 "-" "-"
192.168.0.212 - - [11/Jul/2010:07:48:15 +0100] "GET /KvAutoPublish/KvAutoPublish.Rsync.FullList.php HTTP/1.1" 200 9900 "http://192.168.10.39/KvAutoPublish/Kv
AutoPublish.Rsync.Status.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6 (.NET CLR 3.5.30729)"
81.171.153.68 - - [11/Jul/2010:07:48:15 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=D:/clips/mp3/jack%20johnson%20-%20you%20and%20you_0.m3e&completed
=1&serialNo=K318430&entityID=_EID_&type=MUSICCLIP&file=jack%20johnson%20-%20you%20and%20you_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:15 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=D:/clips/mp3/marina%20_%20the%20diamonds%20-%20oh_0.m3e&completed
=1&serialNo=K214330&entityID=_EID_&type=MUSICCLIP&file=marina%20_%20the%20diamonds%20-%20oh_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:15 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=D:/clips/mp3/dan%20balan%20-%20chica%20bomb%20(uk_0.m3e&completed
=1&serialNo=K318030&entityID=_EID_&type=MUSICCLIP&file=dan%20balan%20-%20chica%20bomb%20(uk_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:16 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/clips/mp3/marina%20_%20the%20diamonds%20-%20oh_0.m3e&completed
=1&serialNo=K333530&entityID=_EID_&type=MUSICCLIP&file=marina%20_%20the%20diamonds%20-%20oh_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:16 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/Kaleidovision/SharedLocal/KvAutoUpdate/LatestImage-20100707062
950.1560.xml&completed=1&serialNo=K336030&entityID=_EID_&type=UNKNOWN&file= HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:16 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/Kaleidovision/SharedLocal/KvAutoUpdate/LatestImage-20100709063
011.1500.xml&completed=1&serialNo=K294230&entityID=_EID_&type=UNKNOWN&file= HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:16 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/clips/mp3/kate%20nash%20-%20kiss%20that%20grrrl.m3e&completed=
1&serialNo=K329030&entityID=_EID_&type=MUSICCLIP&file=kate%20nash%20-%20kiss%20that%20grrrl.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:17 +0100] "GET /kvAutoUpdate/hawkSuccessful.php?id=210558&serialNo=K329430&success=YES&completed=YES&action=1 HTTP/1.1"
200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:17 +0100] "GET /kvAutoUpdate/hawkSuccessful.php?id=210278&serialNo=K221430&success=YES&completed=YES&action=1 HTTP/1.1"
200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:17 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=C:/clips/mp3/scouting%20for%20girls%20-%20famous.m3e&completed=1&
serialNo=K332330&entityID=_EID_&type=MUSICCLIP&file=scouting%20for%20girls%20-%20famous.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:17 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=D:/clips/mp3/30h_3%20-%20my%20first%20kiss-1.m3e&completed=1&seri
alNo=K318430&entityID=_EID_&type=MUSICCLIP&file=30h_3%20-%20my%20first%20kiss-1.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [11/Jul/2010:07:48:17 +0100] "GET /kvAutoUpdate/downloadFile.php?destfile=D:/clips/mp3/scouting%20for%20girls%20-%20famous.m3e&completed=1&
:

Open in new window

Commented:
Are you hosting all these music clips?
D:/clips/mp3/scouting%20for%20girls%20-%20famous.m3
kate%20nash%20-%20kiss%20that%20grrrl.m3
D:/clips/mp3/dan%20balan%20-%20chica%20bomb%20(uk_0.m

All the requests appear to be coming from 81.171.153.68- is that one if your addresses?

Author

Commented:
We are hosting them.

81.171.153.68 is one of our clients. It's strange how the

tail -100 /var/log/apache2/access.log | less

comand is only bringing up requests from 81.171.153.68. We have thousands of clients and checking netstat shows alot of them connecting. Why is the above IP the only one showing in the apache access log?

Commented:
tail -100 gives the last 100 lines
then the 'less' commands paginates the output, so it doesn't scroll past you.
perhaps the 60 lines above were all connections from the one server.
try again run
tail -50 /var/log/apache2/access.log
a few time to see if you only see 1 IP in it.

What's in the /var/log/apache2/error.log?

what's the output of
vmstat 1 10

if iostat is installed try
iostat -d
and
 iostat -dx

Author

Commented:
same IP, also data seems 40mins old or more.?

error.log

[Sun Jul 11 06:25:08 2010] [notice] mod_python: Creating 8 session mutexes based on 250 max processes and 0 max threads.
[Sun Jul 11 06:25:08 2010] [notice] mod_python: using mutex_directory /tmp
[Sun Jul 11 06:25:09 2010] [notice] Apache/2.2.3 (Debian) mod_python/3.2.10 Python/2.4.4 PHP/5.2.0-8+etch13 mod_ssl/2.2.3 OpenSSL/0.9.8c mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Sun Jul 11 06:27:47 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting
* About to connect() to 192.168.0.51 port 80
*   Trying 192.168.0.51... * connected
* Connected to 192.168.0.51 (192.168.0.51) port 80
> GET //dashboard_production/external/unit/ext.unitOnline.php?ENTITY_ID=2286 HTTP/1.1^M
Host: 192.168.0.51^M
Accept: */*^M
^M
< HTTP/1.1 200 OK^M
< Date: Sun, 11 Jul 2010 11:12:34 GMT^M
< Server: Apache/2.2.3 (Debian) mod_python/3.2.10 Python/2.4.4 PHP/5.2.0-8+etch16 mod_ssl/2.2.3 OpenSSL/0.9.8c mod_perl/2.0.2 Perl/v5.8.8^M
< X-Powered-By: PHP/5.2.0-8+etch16^M
< Cache-Control: no-store, no-cache, must-revalidate^M
< Cache-Control: post-check=0, pre-check=0^M
< Pragma: no-cache^M
< Content-Length: 104^M
< Content-Type: text/xml;charset=UTF-8^M
* Connection #0 to host 192.168.0.51 left intact
* Closing connection #0
[Sun Jul 11 12:24:47 2010] [error] [client 221.11.4.190] File does not exist: /var/www/htdocs/manager
[Sun Jul 11 12:51:31 2010] [error] [client 192.168.0.112] File does not exist: /var/www/htdocs/img/ajax-loader.gif, referer: http://192.168.10.39/KvAutoPublish/KvAutoPublish.Rsync.Status.php
[Sun Jul 11 18:10:22 2010] [error] [client 92.240.68.153] File does not exist: /var/www/htdocs/data, referer: http://random.yahoo.com/fast/ryl
[Mon Jul 12 09:37:34 2010] [error] [client 192.168.0.112] File does not exist: /var/www/htdocs/img/ajax-loader.gif, referer: http://192.168.10.39/KvAutoPublish/KvAutoPublish.Rsync.Status.php
[Mon Jul 12 10:28:49 2010] [error] [client 192.168.0.112] File does not exist: /var/www/htdocs/img/ajax-loader.gif, referer: http://192.168.10.39/KvAutoPublish/KvAutoPublish.Rsync.Status.php
[Mon Jul 12 10:46:24 2010] [error] [client 192.168.0.150] File does not exist: /var/www/htdocs/img/ajax-loader.gif, referer: http://192.168.10.39/KvAutoPublish/KvAutoPublish.Rsync.Status.php
[Mon Jul 12 10:46:28 2010] [error] [client 192.168.0.112] File does not exist: /var/www/htdocs/img/ajax-loader.gif, referer: http://192.168.10.39/KvAutoPublish/KvAutoPublish.Rsync.Status.php
[Mon Jul 12 10:47:46 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-loader.gif, referer: http://192.168.10.39/KvAutoPublish/KvAutoPublish.Rsync.Status.php
[Mon Jul 12 10:47:56 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-loader.gif, referer: http://192.168.10.39/KvAutoPublish/KvAutoPublish.Rsync.Status.php
[Mon Jul 12 10:48:14 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-loader.gif, referer: http://192.168.10.39/KvAutoPublish/KvAutoPublish.Rsync.Status.php
[Mon Jul 12 10:48:18 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-loader.gif, referer: http://192.168.10.39/KvAutoPublish/KvAutoPublish.Rsync.Status.php
[Mon Jul 12 10:48:21 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-loader.gif, referer: http://192.168.10.39/KvAutoPublish/KvAutoPublish.Rsync.Status.php
[Mon Jul 12 10:48:32 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-loader.gif, referer: http://192.168.10.39/KvAutoPublish/KvAutoPublish.Rsync.Status.php
[Mon Jul 12 10:48:44 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-loader.gif, referer: http://192.168.10.39/KvAutoPublish/KvAutoPublish.Rsync.Status.php
[Mon Jul 12 10:48:48 2010] [error] [client 192.168.0.212] File does not exist: /var/www/htdocs/img/ajax-loader.gif, referer: http://192.168.10.39/KvAutoPublish/KvAutoPublish.Rsync.Status.php
[Mon Jul 12 10:49:03 2010] [error] [client 192.168.0.150] File does not exist: /var/www/htdocs/img/ajax-loader.g[Mon Jul 12 10:57:19 2010] [notice] caught SIGTERM, shutting down
[Mon Jul 12 10:57:21 2010] [notice] mod_python: Creating 8 session mutexes based on 250 max processes and 0 max threads.
[Mon Jul 12 10:57:21 2010] [notice] mod_python: using mutex_directory /tmp
[Mon Jul 12 10:57:21 2010] [notice] Apache/2.2.3 (Debian) mod_python/3.2.10 Python/2.4.4 PHP/5.2.0-8+etch13 mod_ssl/2.2.3 OpenSSL/0.9.8c mod_perl/2.0.2 Perl/v5.8.8 configured -- resuming normal operations
[Mon Jul 12 10:59:29 2010] [error] [client 192.168.0.150] File does not exist: /var/www/htdocs/img/ajax-loader.gif, referer: http://192.168.10.39/KvAutoPublish/KvAutoPublish.Rsync.Status.php

vmstat 1 10


Mars:~# vmstat 1 10
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 5  0      0 1576156  54748 2180560    0    0     2    13  450  415 58  5 37  0
 1  0      0 1575996  54748 2180580    0    0     0     0 2046 1472 68  5 27  0
 4  0      0 1574368  54748 2181020    0    0     0     0 2253 1568 71  4 24  0
 4  0      0 1573236  54748 2181020    0    0     0     0 1959 1275 73  4 23  0
10  0      0 1569888  54756 2181452    0    0     0    68 2283 1628 73  5 22  0
 1  0      0 1569144  54764 2181924    0    0     0    56 2096 1738 63  6 31  0
 5  0      0 1568904  54764 2181924    0    0     0     0 1804 1390 62  4 34  0
10  0      0 1567880  54764 2181964    0    0     0     0 1970 1456 64  5 31  0
 8  0      0 1569276  54764 2181964    0    0     0     0 1844 1358 61  3 36  0
 5  0      0 1569632  54772 2181956    0    0     0   148 2050 1549 69  5 27  0
Mars:~#


Mars:~# tail -50 /var/log/apache2/access.log
81.171.153.68 - - [12/Jul/2010:14:25:44 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=D:/clips/mp3/eminem%20-%20love%20the%20way%20you%20_0.m3e&completed=1&seri                                                                          alNo=K318430&entityID=_EID_&type=MUSICCLIP&file=eminem%20-%20love%20the%20way%20you                                                                          %20_0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:44 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=D:/clips/mp3/kylie%20-%20all%20the%20lovers-1.m3e&completed=1&serialNo=K31                                                                          8030&entityID=_EID_&type=MUSICCLIP&file=kylie%20-%20all%20the%20lovers-1.m3e HTTP/1                                                                          .1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:44 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=D:/clips/mp3/the%20drums%20-%20forever%20and%20ev_0.m3e&completed=1&serial                                                                          No=K294230&entityID=_EID_&type=MUSICCLIP&file=the%20drums%20-%20forever%20and%20ev_                                                                          0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:44 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/20575_pricing_thu_e%201750.mpe&completed=1&serialNo=K333530&entit                                                                          yID=_EID_&type=GFXCLIP-BSM&file=20575_pricing_thu_e%201750.mpe HTTP/1.1" 200 4 "-"                                                                           "-"
81.171.153.68 - - [12/Jul/2010:14:25:44 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=D:/clips/mp3/everything%20everything%20-%20sc_0.m3e&completed=1&serialNo=K                                                                          318430&entityID=_EID_&type=MUSICCLIP&file=everything%20everything%20-%20sc_0.m3e HT                                                                          TP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:44 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/mp3/futureheads%20-%20i%20can%20do%20that.m3e&completed=1&serialN                                                                          o=K328530&entityID=_EID_&type=MUSICCLIP&file=futureheads%20-%20i%20can%20do%20that.                                                                          m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:44 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=D:/clips/mp3/m.i.a%20-%20xxxo%20-%20featuring%20j_0.m3e&completed=1&serial                                                                          No=K318030&entityID=_EID_&type=MUSICCLIP&file=m.i.a%20-%20xxxo%20-%20featuring%20j_                                                                          0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=D:/clips/mp3/usher%20-%20lil'%20freak%20(feat.%20_0.m3e&completed=1&serial                                                                          No=K294230&entityID=_EID_&type=MUSICCLIP&file=usher%20-%20lil'%20freak%20(feat.%20_                                                                          0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=D:/clips/mp3/futureheads%20-%20i%20can%20do%20that.m3e&completed=1&serialN                                                                          o=K318430&entityID=_EID_&type=MUSICCLIP&file=futureheads%20-%20i%20can%20do%20that.                                                                          m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/20575_pricing_tues_a%201250.mpe&completed=1&serialNo=K333530&enti                                                                          tyID=_EID_&type=GFXCLIP-BSM&file=20575_pricing_tues_a%201250.mpe HTTP/1.1" 200 4 "-                                                                          " "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=D:/clips/mp3/flo-rida%20feat.%20david%20guett_0.m3e&completed=1&serialNo=K                                                                          214330&entityID=_EID_&type=MUSICCLIP&file=flo-rida%20feat.%20david%20guett_0.m3e HT                                                                          TP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/mp3/joshua%20radin%20-%20they%20bring%20_0.m3e&completed=1&serial                                                                          No=K330930&entityID=_EID_&type=MUSICCLIP&file=joshua%20radin%20-%20they%20bring%20_                                                                          0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/mp3/joshua%20radin%20-%20brand%20new%20d_0.m3e&completed=1&serial                                                                          No=K328530&entityID=_EID_&type=MUSICCLIP&file=joshua%20radin%20-%20brand%20new%20d_                                                                          0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=D:/clips/mp3/lcd%20soundsystem%20-%20i%20can%20ch_0.m3e&completed=1&serial                                                                          No=K318030&entityID=_EID_&type=MUSICCLIP&file=lcd%20soundsystem%20-%20i%20can%20ch_                                                                          0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=D:/clips/mp3/joshua%20radin%20-%20brand%20new%20d_0.m3e&completed=1&serial                                                                          No=K318430&entityID=_EID_&type=MUSICCLIP&file=joshua%20radin%20-%20brand%20new%20d_                                                                          0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/20575_pricing_tues_a%201750.mpe&completed=1&serialNo=K333530&enti                                                                          tyID=_EID_&type=GFXCLIP-BSM&file=20575_pricing_tues_a%201750.mpe HTTP/1.1" 200 4 "-                                                                          " "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/mp3/we%20are%20scientists%20-%20nice%20g_0.m3e&completed=1&serial                                                                          No=K332830&entityID=_EID_&type=MUSICCLIP&file=we%20are%20scientists%20-%20nice%20g_                                                                          0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=D:/clips/mp3/jls%20-%20the%20club%20is%20alive-1.m3e&completed=1&serialNo=                                                                          K214330&entityID=_EID_&type=MUSICCLIP&file=jls%20-%20the%20club%20is%20alive-1.m3e                                                                           HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/mp3/king%20blues%20-%20headbutt.m3e&completed=1&serialNo=K330930&                                                                          entityID=_EID_&type=MUSICCLIP&file=king%20blues%20-%20headbutt.m3e HTTP/1.1" 200 4                                                                           "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/mp3/katy%20perry%20-%20california%20gu_0.m3e&completed=1&serialNo                                                                          =K328530&entityID=_EID_&type=MUSICCLIP&file=katy%20perry%20-%20california%20gu_0.m3                                                                          e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=D:/clips/mp3/kate%20nash%20-%20kiss%20that%20grrrl.m3e&completed=1&serialN                                                                          o=K318430&entityID=_EID_&type=MUSICCLIP&file=kate%20nash%20-%20kiss%20that%20grrrl.                                                                          m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/20575_pricing_tues_e%201250.mpe&completed=1&serialNo=K333530&enti                                                                          tyID=_EID_&type=GFXCLIP-BSM&file=20575_pricing_tues_e%201250.mpe HTTP/1.1" 200 4 "-                                                                          " "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=D:/clips/mp3/kylie%20-%20all%20the%20lovers-1.m3e&completed=1&serialNo=K21                                                                          4330&entityID=_EID_&type=MUSICCLIP&file=kylie%20-%20all%20the%20lovers-1.m3e HTTP/1                                                                          .1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/mp3/1miley%20cyrus%20-%20can't%20be%20ta_0.m3e&completed=1&serial                                                                          No=K339730&entityID=_EID_&type=MUSICCLIP&file=1miley%20cyrus%20-%20can't%20be%20ta_                                                                          0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:45 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=D:/clips/mp3/katy%20perry%20-%20california%20gu_0.m3e&completed=1&serialNo                                                                          =K318430&entityID=_EID_&type=MUSICCLIP&file=katy%20perry%20-%20california%20gu_0.m3                                                                          e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:46 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/20575_pricing_tues_e%201750.mpe&completed=1&serialNo=K333530&enti                                                                          tyID=_EID_&type=GFXCLIP-BSM&file=20575_pricing_tues_e%201750.mpe HTTP/1.1" 200 4 "-                                                                          " "-"
81.171.153.68 - - [12/Jul/2010:14:25:46 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=D:/clips/mp3/robbie%20rivera%20-%20let%20me%20sip_0.m3e&completed=1&serial                                                                          No=K214330&entityID=_EID_&type=MUSICCLIP&file=robbie%20rivera%20-%20let%20me%20sip_                                                                          0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:46 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/20575_pricing_wed_a%201250.mpe&completed=1&serialNo=K333530&entit                                                                          yID=_EID_&type=GFXCLIP-BSM&file=20575_pricing_wed_a%201250.mpe HTTP/1.1" 200 4 "-"                                                                           "-"
81.171.153.68 - - [12/Jul/2010:14:25:46 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=D:/clips/mp3/the%20coral%20-%201000%20years.m3e&completed=1&serialNo=K2143                                                                          30&entityID=_EID_&type=MUSICCLIP&file=the%20coral%20-%201000%20years.m3e HTTP/1.1"                                                                           200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:46 +0100] "GET /kvAutoUpdate/hawkSuccessful.ph                                                                          p?id=210277&serialNo=K214330&success=YES&completed=YES&action=1 HTTP/1.1" 200 4 "-"                                                                           "-"
81.171.153.68 - - [12/Jul/2010:14:25:47 +0100] "GET /kvAutoUpdate/manifest_hawk.php                                                                          ?winversion=1.3.1.0&version=ServeTheServants&dated=2010-04-09-1511&serialNo=K334930                                                                          &entityID=2291 HTTP/1.0" 200 111 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:47 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/mp3/david%20guetta%20(fergie)%20-%20ge_0.m3e&completed=1&serialNo                                                                          =K329030&entityID=_EID_&type=MUSICCLIP&file=david%20guetta%20(fergie)%20-%20ge_0.m3                                                                          e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:48 +0100] "GET /kvAutoUpdate/hawkSuccessful.ph                                                                          p?id=210276&serialNo=K207830&success=YES&completed=YES&action=1 HTTP/1.1" 200 4 "-"                                                                           "-"
81.171.153.68 - - [12/Jul/2010:14:25:48 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/mp3/erik%20hassle%20-%20isnt%20it%20obvi_0.m3e&completed=1&serial                                                                          No=K339730&entityID=_EID_&type=MUSICCLIP&file=erik%20hassle%20-%20isnt%20it%20obvi_                                                                          0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:48 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/mp3/everything%20everything%20-%20sc_0.m3e&completed=1&serialNo=K                                                                          329030&entityID=_EID_&type=MUSICCLIP&file=everything%20everything%20-%20sc_0.m3e HT                                                                          TP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:48 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/mp3/kate%20nash%20-%20kiss%20that%20grrrl.m3e&completed=1&serialN                                                                          o=K329030&entityID=_EID_&type=MUSICCLIP&file=kate%20nash%20-%20kiss%20that%20grrrl.                                                                          m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:48 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/mp3/1lena%20-%20satellite.m3e&completed=1&serialNo=K339730&entity                                                                          ID=_EID_&type=MUSICCLIP&file=1lena%20-%20satellite.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:48 +0100] "GET /kvAutoUpdate/hawkSuccessful.ph                                                                          p?id=210577&serialNo=K203730&success=YES&completed=YES&action=1 HTTP/1.1" 200 4 "-"                                                                           "-"
81.171.153.68 - - [12/Jul/2010:14:25:49 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/mp3/30h_3%20-%20my%20first%20kiss-1.m3e&completed=1&serialNo=K339                                                                          730&entityID=_EID_&type=MUSICCLIP&file=30h_3%20-%20my%20first%20kiss-1.m3e HTTP/1.1                                                                          " 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:49 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/mp3/lcd%20soundsystem%20-%20i%20can%20ch_0.m3e&completed=1&serial                                                                          No=K329030&entityID=_EID_&type=MUSICCLIP&file=lcd%20soundsystem%20-%20i%20can%20ch_                                                                          0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:49 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/mp3/i%20am%20arrows%20-%20green%20grass-1.m3e&completed=1&serialN                                                                          o=K332730&entityID=_EID_&type=MUSICCLIP&file=i%20am%20arrows%20-%20green%20grass-1.                                                                          m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:49 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/mp3/adam%20lambert%20-%20whataya%20wan_0.m3e&completed=1&serialNo                                                                          =K339730&entityID=_EID_&type=MUSICCLIP&file=adam%20lambert%20-%20whataya%20wan_0.m3                                                                          e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:49 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/mp3/scissor%20-%20fire%20with%20fire%20(_0.m3e&completed=1&serial                                                                          No=K329030&entityID=_EID_&type=MUSICCLIP&file=scissor%20-%20fire%20with%20fire%20(_                                                                          0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:49 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/mp3/katy%20perry%20-%20california%20gu_0.m3e&completed=1&serialNo                                                                          =K332730&entityID=_EID_&type=MUSICCLIP&file=katy%20perry%20-%20california%20gu_0.m3                                                                          e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:49 +0100] "GET /kvAutoUpdate/manifest_hawk.php                                                                          ?winversion=1.3.1.0&version=ServeTheServants&dated=2010-04-09-1511&serialNo=K335930                                                                          &entityID=2323 HTTP/1.0" 200 111 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:49 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/mp3/angie%20stone%20-%20free-1.m3e&completed=1&serialNo=K339730&e                                                                          ntityID=_EID_&type=MUSICCLIP&file=angie%20stone%20-%20free-1.m3e HTTP/1.1" 200 4 "-                                                                          " "-"
81.171.153.68 - - [12/Jul/2010:14:25:49 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/mp3/two%20door%20cinema%20-%20somethin_0.m3e&completed=1&serialNo                                                                          =K329030&entityID=_EID_&type=MUSICCLIP&file=two%20door%20cinema%20-%20somethin_0.m3                                                                          e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:49 +0100] "GET /kvAutoUpdate/downloadFile.php?                                                                          destfile=C:/clips/mp3/robbie%20rivera%20-%20let%20me%20sip_0.m3e&completed=1&serial                                                                          No=K332730&entityID=_EID_&type=MUSICCLIP&file=robbie%20rivera%20-%20let%20me%20sip_                                                                          0.m3e HTTP/1.1" 200 4 "-" "-"
81.171.153.68 - - [12/Jul/2010:14:25:50 +0100] "GET /kvAutoUpdate/hawkSuccessful.ph                                                                          p?id=210426&serialNo=K332730&success=YES&completed=YES&action=1 HTTP/1.1" 200 4 "-"                                                                           "-"
81.171.153.68 - - [12/Jul/2010:14:25:50 +0100] "GET /kvAutoUpdate/downloadFiMars:~#

Open in new window

Commented:
The vmstat shows the run queue (r) greater than 1 - this is not good:

The runqueue value shows the number of tasks executing and waiting for CPU resources. When this number exceeds the number of CPUs on the server, a CPU bottleneck exists, and some tasks are waiting for execution.

You appear to be running out of a resource, probably disk or network, can you confirm your NIC is set to the correct speed and duplex for your system, sometimes if it's set to autonegotiate, it defaults down to a lower speed or duplex and you get network probs.  (ethtool eth0, assuming your NIC is eth0)

I notice the error from your error.log
[error] server reached MaxClients setting, consider raising the MaxClients setting
* About to connect() to 192.168.0.51 port 80
What is your MaxClients set to? (in your apache httpd.conf file)

Author

Commented:
Hi, Thank for this help.

Mars:~# ethtool eth0
Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: g
        Wake-on: d
        Link detected: yes

MaxClients is set to 250 in apache2.conf (httpd.conf is empty?)

Author

Commented:
More information
Mars:~# iostat -d
Linux 2.6.18-6-amd64 (Mars)     12/07/10        _x86_64_

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              30.91       317.44       567.25    5749384   10273984

Mars:~# iostat -dx
Linux 2.6.18-6-amd64 (Mars)     12/07/10        _x86_64_

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.89    41.62    1.65   29.23   317.19   566.86    28.62     0.03    1.02   0.19   0.58

Mars:~#

Open in new window

Commented:
You can try and up the MaxClient in your apache2.conf (then restart apache) to see if they clears the problem - or takes it longer to fail

I would always force NICs not to autoneg, but you should probably talk to your network people to see how they've got it.  But it's reporting 1000fdx so looks fine

Can you shut down apache and re-run the iostat -dx when apache is down to see what difference there is, to help see if it's disk or network bottlenecking

Author

Commented:
with apache2 stopped.
Mars:~# iostat -dx
Linux 2.6.18-6-amd64 (Mars)     12/07/10        _x86_64_

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.82    38.49    1.55   26.77   289.52   522.07    28.65     0.03    1.02   0.19   0.54

Open in new window

Commented:
So looking at that it doesn't appear to be a disk I/o problem and as your running at 1000fdx, I doubt it's a network bottleneck, can you try upping the MaxClient

Commented:
You will also have to raise ServerLimit
(http://www.webhostingtalk.com/showthread.php?t=661905)

Author

Commented:
upped MaxClient to 350 and got this upon apache2 restart.
Mars:/etc/apache2# /etc/init.d/apache2 restart
Forcing reload of web server (apache2)...WARNING: MaxClients of 350 exceeds ServerLimit value of 256 servers,
 lowering MaxClients to 256.  To increase, please see the ServerLimit
 directive.
 waiting WARNING: MaxClients of 350 exceeds ServerLimit value of 256 servers,
 lowering MaxClients to 256.  To increase, please see the ServerLimit
 directive.

Open in new window

Commented:
Yes, sorry added that in the next comment, up them both to 1000 and give it a go....

Author

Commented:
added

ServerLimit   350

to apache2.conf but a restart spews out the same error?

Commented:
did you restart apache or stop and start it? - there appears to be a gotcha (http://northernmost.org/blog/apache-gotcha-with-serverlimit/) a restart won't do it

Author

Commented:
stopped it and started it.

Same error unfortunately.
CERTIFIED EXPERT
Top Expert 2010

Commented:
Do you analyze your Apache log files (awstats / analog / webalizer), monitor your site through one of the Web analytic providers (Google Analytics / Omniture) or monitor system resources (cacti / nagios)?

If so have you noticed any change in the number of the requests, or the timing of the peek load over the last week?

If not, and you have a few weeks worth of log data, have a look at one of the log analyzers, and possibly setting up Cacti on a separate server / PC to monitor load, network throughput and Apache hits.
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION

Gain unlimited access to on-demand training courses with an Experts Exchange subscription.

Get Access
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Empower Your Career
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.