Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 5107
  • Last Modified:

too many sending reply apache status URGENT

I have a problem with apache , I am also install nginx as reverse proxy but  I still get this problem
I hope to help me with this problem
its occurring  some time ever 1 or 2 hour

I hope to help me

Server Version: apache
Server Built: Apr 23 2011 12:02:28

Current Time: Sunday, 24-Apr-2011 01:00:51 WEST
Restart Time: Saturday, 23-Apr-2011 23:44:23 WEST
Parent Server Generation: 0
Server uptime: 1 hour 16 minutes 28 seconds
Total accesses: 94055 - Total Traffic: 394.1 MB
CPU Usage: u6687.8 s176.51 cu.35 cs0 - 150% CPU load
20.5 requests/sec - 88.0 kB/second - 4393 B/request
507 requests currently being processed, 5 idle workers

WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW_WWWWW_W_WWW
_WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW_WWWWWWWWWWWWWWWWWWWWWW

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

Srv    PID    Acc    M    CPU     SS    Req    Conn    Child    Slot    Client    VHost    Request
0-0    17206    0/1806/2069    W     158.62    170    0    0.0    5.17    6.96     2.89.87.40    vb.domain.com    GET /t70067-34559.html HTTP/1.0
1-0    17207    0/1814/2093    W     158.71    166    0    0.0    6.49    7.96     86.60.32.140    vb.domain.com    POST /newreply.php?do=postreply&t=498050 HTTP/1.0
2-0    17208    0/1812/2106    W     158.94    167    0    0.0    5.62    7.97     188.49.101.100    vb.domain.com    POST /reputation.php?do=addreputation&p=16485798 HTTP/1.0
3-0    21233    0/2/272    W     0.22    172    0    0.0    0.00    2.35     46.240.30.43    vb.domain.com    POST /newreply.php?do=postreply&t=292602 HTTP/1.0
4-0    21234    0/2/256    W     0.10    171    0    0.0    0.00    1.07     90.148.236.110    vb.domain.com    POST /newreply.php?do=postreply&t=506900 HTTP/1.0
5-0    17211    0/1812/2071    W     152.40    177    0    0.0    5.94    6.48     77.30.2.55    vb.domain.com    GET /t512228-new-post/ HTTP/1.0
6-0    17212    0/1805/2041    W     151.52    175    0    0.0    5.29    7.74     188.49.58.198    vb.domain.com    POST /editpost.php?do=updatepost&p=16485753 HTTP/1.0
7-0    21239    0/12/278    W     0.82    164    0    0.0    0.04    0.87     77.30.91.194    vb.domain.com    GET /t438661/ HTTP/1.0
8-0    21240    0/1/283    W     0.00    170    0    0.0    0.00    0.98     65.52.110.90    vb.domain.com    GET /t322050-1833.html HTTP/1.0
9-0    21274    0/9/260    W     0.15    165    0    0.0    0.00    1.91     188.49.101.100    vb.domain.com    POST /reputation.php?do=addreputation&p=16485798 HTTP/1.0
10-0    21275    0/3/242    W     0.38    166    0    0.0    0.03    1.61     188.54.96.124    vb.domain.com    GET /t488143/ HTTP/1.0
11-0    21276    0/0/243    W     23.29    170    0    0.0    0.00    1.46     84.235.75.93    vb.domain.com    POST /newreply.php?do=postreply&t=237061 HTTP/1.0
12-0    21277    0/9/272    W     0.50    164    0    0.0    0.00    0.76     188.249.78.252    vb.domain.com    GET /t510223/ HTTP/1.0
13-0    21278    0/15/271    W     0.78    159    0    0.0    0.01    1.47     188.51.8.3    vb.domain.com    POST /newreply.php?do=postreply&t=507993 HTTP/1.0
14-0    21279    0/1/257    W     0.00    168    0    0.0    0.00    2.07     65.52.109.60    vb.domain.com    GET /t499296/ HTTP/1.0
15-0    21280    0/2/296    W     0.01    167    0    0.0    0.00    1.38     87.109.147.185    vb.domain.com    GET /t471717/ HTTP/1.0
16-0    21281    0/6/299    W     0.06    165    0    0.0    0.00    1.83     2.91.245.79    vb.domain.com    GET /t362187-947.html HTTP/1.0
17-0    21282    0/3/262    W     0.09    166    0    0.0    0.00    0.58     41.100.208.235    vb.domain.com    GET /member.php?u=302285 HTTP/1.0
18-0    21283    0/1/196    W     0.18    167    0    0.0    0.00    0.46     65.52.110.36    vb.domain.com    GET /t433599/ HTTP/1.0
19-0    21284    0/4/259    W     0.07    166    0    0.0    0.00    0.45     77.31.101.61    vb.domain.com    POST /ajax.php?do=vbseoui HTTP/1.0
20-0    21285    0/15/166    W     0.29    159    0    0.0    0.01    0.58     72.30.161.235    vb.domain.com    GET /t506509/ HTTP/1.0
21-0    21290    0/3/23    W     0.02    165    0    0.0    0.00    0.15     188.49.101.100    vb.domain.com    POST /reputation.php?do=addreputation&p=16485798 HTTP/1.0
22-0    21291    0/0/15    W     1.13    164    0    0.0    0.00    0.01     86.51.122.222    vb.domain.com    GET /f21/ HTTP/1.0
23-0    21292    0/2/32    W     0.09    163    0    0.0    0.00    0.06     77.31.101.61    vb.domain.com    POST /ajax.php?do=vbseoui HTTP/1.0
24-0    21293    0/17/47    W     0.65    155    0    0.0    0.00    0.17     77.64.124.75    vb.domain.com    GET /t418435-991.html HTTP/1.0
25-0    21294    0/1/6    W     0.00    163    0    0.0    0.00    0.00     94.96.3.57    vb.domain.com    POST /newreply.php?do=postreply&t=490738 HTTP/1.0
26-0    21295    0/7/12    W     0.36    159    0    0.0    0.00    0.01     220.181.108.165    vb.domain.com    GET /t512230/ HTTP/1.0
27-0    21296    0/4/19    W     0.16    160    0    0.0    0.01    0.05     77.31.101.61    vb.domain.com    POST /ajax.php?do=vbseoui HTTP/1.0
28-0    21297    0/8/21    W     0.50    158    0    0.0    0.01    0.02     87.109.181.151    vb.domain.com    GET /t283011/ HTTP/1.0
29-0    21298    0/2/47    W     0.16    160    0    0.0    0.00    0.03     188.248.35.32    vb.domain.com    GET /t414140/ HTTP/1.0
30-0    21299    0/0/29    W     2.79    162    0    0.0    0.00    0.07     87.109.169.49    vb.domain.com    GET /t510259-3.html HTTP/1.0
31-0    21300    0/0/31    W     2.89    162    0    0.0    0.00    0.11     94.96.21.7    vb.domain.com    GET /t22317-6.html HTTP/1.0
32-0    21301    0/2/23    W     0.01    160    0    0.0    0.00    0.42     87.109.181.151    vb.domain.com    GET /t283011/ HTTP/1.0
33-0    21302    0/1/53    W     0.01    161    0    0.0    0.00    0.19     94.99.48.124    vb.domain.com    GET /search.php?do=finduser&u=262363 HTTP/1.0
34-0    21303    0/0/40    W     3.67    161    0    0.0    0.00    0.04     220.181.108.170    vb.domain.com    GET /t512229/ HTTP/1.0
35-0    21304    0/25/83    W     1.79    143    0    0.0    0.36    0.49     188.249.36.51    vb.domain.com    GET /t70067-34559.html HTTP/1.0
36-0    13217    0/3346/3346    W     290.70    177    0    0.0    15.62    15.62     66.249.71.12    vb.domain.com    GET /showpost.php?p=3881831&postcount=1 HTTP/1.0
37-0    21306    0/2/13    W     0.15    158    0    0.0    0.12    0.47

Open in new window

0
xserverx
Asked:
xserverx
  • 5
  • 4
4 Solutions
 
arnoldCommented:
What are you looking to do?  This seems to be normal requests/replies from apache?  If the performance of the system allows, you can increase the number of child process apache can spawn to answer queries.

150% CPU load

This means that your demand on the system is higher than the system can provide.
0
 
xserverxAuthor Commented:
please help me
I don't understand what I can do
0
 
xserverxAuthor Commented:

system info :
dual X5670 (24 cores)
24Gb Ram
2 x 1TB sata
1 x 32Gb ssd for mysql
1000mbps

KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 2
HostNameLookups Off
<IfModule prefork.c>
StartServers 5
MinSpareServers 10
MaxSpareServers 35
ServerLimit 512
MaxClients 512
MaxRequestsPerChild 6000
</IfModule>
now I changed to worker but it still same problem
0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 
arnoldCommented:
What is the issue?
Your mysql might be the issue and not apache, install phpmyadmin from http://www.phpmyadmin.net/home_page/index.php
And see what it is reporting about the performance of the mysql server and whether that can be improved.

Increase the startservers to 30
MinSpareServers 40
MaxSpareServers 140
Reduce the
MaxRequestsPerChild 600

Note the stats you have are reflecting 35 servers sending replies.

0
 
xserverxAuthor Commented:
can you tell me why mysql is the problem not apache ?
0
 
arnoldCommented:
I am not saying it is, I am saying you should consider the possibility and use the tool I referenced (phpmyadmin) to see whether it is and if it is there will be information there on how to improve its performance. i.e. increasenig tmp_table_space, etc.

What about adjusting the apache configuration entries?
can you post top -n 1 during the time period you seeing as a problem??
0
 
xserverxAuthor Commented:
right now I have the problem quit normal but time to time I have taht happened again and apache stop for some time befor go normal again
right now I have switched to mpm worker with this setting :
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 2
HostNameLookups Off

<IfModule mpm_worker_module>
StartServers 10
MaxClients 400
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 10000
</IfModule>

but I am still having same issue :(

please see  if I understand correct :
if this  really  related to mysql , so when I have all worker in sending replay status "W"   that mean that apache try to get results from php/mysql , I have this W status because I have db tables are locked and apache try to get result from them  ?

top - 14:27:43 up 2 days,  3:03,  1 user,  load average: 5.55, 6.12, 5.87
Tasks: 431 total,   4 running, 427 sleeping,   0 stopped,   0 zombie
Cpu(s): 10.5%us,  2.2%sy,  0.0%ni, 84.3%id,  2.8%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:  24675964k total, 13484684k used, 11191280k free,   317652k buffers
Swap:  2096440k total,      136k used,  2096304k free,  8447980k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
32033 mysql      5 -10 5360m 3.3g 4600 S 110.3 14.2 798:37.85 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/
22188 vb    17   0  292m  44m  19m R 29.0  0.2   0:00.15 /usr/bin/php /home/vb/public_html/vbseo.php
22189 vb    17   0  277m  19m  10m R  5.8  0.1   0:00.03 /usr/bin/php /home/vb/public_html/vbseo.php
22190 vb    17   0  277m  19m  10m R  5.8  0.1   0:00.03 /usr/bin/php /home/vb/public_html/vbseo.php
11452 nobody    19   0  364m  10m 2256 S  1.9  0.0   0:09.20 /usr/local/apache/bin/httpd -k start
22184 root      15   0 13024 1268  728 R  1.9  0.0   0:00.02 top c -n 1
    1 root      15   0 10372  696  580 S  0.0  0.0   0:05.77 init [3]
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.56 [migration/0]
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/0]
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/0]
    5 root      RT  -5     0    0    0 S  0.0  0.0   0:01.49 [migration/1]
    6 root      34  19     0    0    0 S  0.0  0.0   0:00.01 [ksoftirqd/1]
    7 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/1]
    8 root      RT  -5     0    0    0 S  0.0  0.0   0:00.31 [migration/2]
    9 root      34  19     0    0    0 S  0.0  0.0   0:00.01 [ksoftirqd/2]
   10 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/2]
   11 root      RT  -5     0    0    0 S  0.0  0.0   0:00.68 [migration/3]
   12 root      34  19     0    0    0 S  0.0  0.0   0:00.01 [ksoftirqd/3]
   13 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/3]
   14 root      RT  -5     0    0    0 S  0.0  0.0   0:03.15 [migration/4]
   15 root      34  19     0    0    0 S  0.0  0.0   0:00.12 [ksoftirqd/4]
   16 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/4]
   17 root      RT  -5     0    0    0 S  0.0  0.0   0:01.49 [migration/5]
   18 root      34  19     0    0    0 S  0.0  0.0   0:00.01 [ksoftirqd/5]
   19 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/5]
   20 root      RT  -5     0    0    0 S  0.0  0.0   0:00.95 [migration/6]
   21 root      34  19     0    0    0 S  0.0  0.0   0:00.01 [ksoftirqd/6]
   22 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/6]
   23 root      RT  -5     0    0    0 S  0.0  0.0   0:01.09 [migration/7]
   24 root      34  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/7]
   25 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/7]
   26 root      RT  -5     0    0    0 S  0.0  0.0   0:01.46 [migration/8]
   27 root      34  19     0    0    0 S  0.0  0.0   0:00.01 [ksoftirqd/8]
   28 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/8]
   29 root      RT  -5     0    0    0 S  0.0  0.0   0:01.86 [migration/9]
   30 root      34  19     0    0    0 S  0.0  0.0   0:00.02 [ksoftirqd/9]
   31 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/9]
   32 root      RT  -5     0    0    0 S  0.0  0.0   0:01.49 [migration/10]
   33 root      34  19     0    0    0 S  0.0  0.0   0:00.01 [ksoftirqd/10]
   34 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/10]
   35 root      RT  -5     0    0    0 S  0.0  0.0   0:01.09 [migration/11]
   36 root      34  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/11]
   37 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/11]

Open in new window

0
 
arnoldCommented:
The more requests you allow per client the more likely it is that you will have few clients generating many requests on the same connection while others wiating.  Forcing the client to reconnect lets others an opportunity to connect and get their data.

You basically have setup a buffet where the first clients can requests and monopolize all the apache resources you've allocated while others will stand in line.

Swithing from one to the other has be based on something .

Your configuration also does not seem to make sense:
You have max of 400 clients with max of 25 threads per client while you have 25 min sparethreads and 75 maxsparethreads.

0
 
Gabriel OrozcoSolution ArchitectCommented:
I believe you need to do a performance tuning on your server.

which configuration are you using for mysql? are you using settings for a small amount of users when you need hundreds of threads running?

change the my.cnf to the medium or huge config if you have no idea of what I am saying

Also you need to check your logs to see if all these connections are coming from the same users, and specially if you are being used as a proxy to send spam.
0
 
xserverxAuthor Commented:
Thanks
0

Featured Post

Evaluating UTMs? Here's what you need to know!

Evaluating a UTM appliance and vendor can prove to be an overwhelming exercise.  How can you make sure that you're getting the security that your organization needs without breaking the bank? Check out our UTM Buyer's Guide for more information on what you should be looking for!

  • 5
  • 4
Tackle projects and never again get stuck behind a technical roadblock.
Join Now