Solved

Apache 2.2 > ApacheBench > Optimization > Data transfer rate?

Posted on 2008-06-23
15
4,445 Views
Last Modified: 2013-12-06
Experts,

I recently bench marked a client's RedHat Apache 2.2.3 web server from a remote server (see Test #1 below) and on the web server itself (see Test #2 below).

The results from the remote host are weird because the "Transfer rate" is only 616.36 [Kbytes/sec] - which is about 5 Mbps.  However when I do an FTP speed test from the server, I can move a 38 MB file in 25 seconds which is about 250 Mbps.  So yes, we have a fast Internet connection.

Is the "Transfer rate" really an issue?  And what do I need to do to get my server to respond faster?
### Test #1 (From Remote Host) ####
 

windows> ab -kc 100 -t 60 http://www.myclient.com/
 

This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0

Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/

Copyright 2006 The Apache Software Foundation, http://www.apache.org/
 

Benchmarking www.myclient.com (be patient)

Finished 739 requests
 
 

Server Software:        Apache/2.2.3

Server Hostname:        www.myclient.com

Server Port:            80
 

Document Path:          /

Document Length:        55094 bytes
 

Concurrency Level:      100

Time taken for tests:   65.437500 seconds

Complete requests:      739

Failed requests:        0

Write errors:           0

Keep-Alive requests:    0

Total transferred:      41301913 bytes

HTML transferred:       41087878 bytes

Requests per second:    11.29 [#/sec] (mean)

Time per request:       8854.871 [ms] (mean)

Time per request:       88.549 [ms] (mean, across all concurrent requests)

Transfer rate:          616.36 [Kbytes/sec] received
 

Connection Times (ms)

              min  mean[+/-sd] median   max

Connect:       15   76  48.0     78     515

Processing:  3515 7864 1609.6   7484   17110

Waiting:       78 3545 2419.5   3406   11078

Total:       3578 7941 1610.1   7562   17125
 

Percentage of the requests served within a certain time (ms)

  50%   7562

  66%   7687

  75%   8109

  80%   8265

  90%  10093

  95%  11531

  98%  11937

  99%  12046

 100%  17125 (longest request)
 
 

### Test #2 (On localhost) ###

[linux /]# ab -kc 100 -t 60 http://www.myclient.com/
 

This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0

Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/

Copyright 2006 The Apache Software Foundation, http://www.apache.org/
 

Benchmarking www.myclient.com (be patient)

Completed 5000 requests

Completed 10000 requests

Completed 15000 requests

Completed 20000 requests

Completed 25000 requests

Completed 30000 requests

Completed 35000 requests

Completed 40000 requests

Completed 45000 requests

Finished 50000 requests
 
 

Server Software:        Apache/2.2.3

Server Hostname:        www.myclient.com

Server Port:            80
 

Document Path:          /

Document Length:        55094 bytes
 

Concurrency Level:      100

Time taken for tests:   50.354540 seconds

Complete requests:      50000

Failed requests:        0

Write errors:           0

Keep-Alive requests:    0

Total transferred:      2769060758 bytes

HTML transferred:       2754810188 bytes

Requests per second:    992.96 [#/sec] (mean)

Time per request:       100.709 [ms] (mean)

Time per request:       1.007 [ms] (mean, across all concurrent requests)

Transfer rate:          53702.41 [Kbytes/sec] received
 

Connection Times (ms)

              min  mean[+/-sd] median   max

Connect:        0    4   6.9      1     100

Processing:    17   95  21.3     95     249

Waiting:        1   55  36.6     52     244

Total:         19   99  21.9     98     249
 

Percentage of the requests served within a certain time (ms)

  50%     98

  66%    104

  75%    108

  80%    111

  90%    123

  95%    138

  98%    154

  99%    173

 100%    249 (longest request)
 

### Prefork Config ###

<IfModule prefork.c>

StartServers       8

MinSpareServers    5

MaxSpareServers   15

ServerLimit      150

MaxClients       150

MaxRequestsPerChild  2000

SendBufferSize 16384

</IfModule>

Open in new window

0
Comment
Question by:Geoff Millikan
  • 8
  • 7
15 Comments
 
LVL 57

Expert Comment

by:giltjr
ID: 21861685
What is your Internet connection?

When looking at throughput you have to look at multiple things.   The first thing is overhead

FTP: 1 tranfer of 38 MB, which means basically 1 TCP  connection.

HTTP: 739 tranfers totaling roughly 41 MB for an avg. of 55K per transfer.  This also means, most likley 739 TCP connections.

Each TCP connection requries 3 packets to start (3-way handshake) and two to end.

TCP uses what is called a "slow start" mechanism.  So that it does not flood the network with traffic TCP will send out two packets and see how long it takes to get back the ACK's.  It slowly starts to increase the number of packets it sends out up to the remote sides TCP window size.

So for  EACH HTTP transfer TCP will go through this slow start, or 739 times.

If you want to compare the Apache test to ftp, try doing 739 ftp transfers of a file that is about 55K.

Then create one HTML document that is 38MB and do one HTTP transfer.

 
0
 

Author Comment

by:Geoff Millikan
ID: 21887102
giltjr: The client's download capacity just tested at 15590 kbps (1948.8 KB/sec transfer rate).  And the host should be capable of at least 100Mbps since it's in a fast collocation center. So we'd expect the "Transfer Rate" to be near the speed of the client, 1948.8 KB/sec - but it's not.

Are you saying that the difference between the reported transfer rate (616.36 Kbytes/sec) and the client's capacity (1948.8 KB/sec) is due to slow start and overhead?  (The client is running Windows XP so maybe the TCP/IP receive buffer is overloaded?)

P.S.  I attached another test below, this time from a different client location on a T1 (1.5 Mbps) Internet connection.


FreeBSD> ab -kc 100 -t 60 http://www.myclient.com/

This is ApacheBench, Version 1.3d <$Revision: 1.73 $> apache-1.3

Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/

Copyright (c) 2006 The Apache Software Foundation, http://www.apache.org/
 

Benchmarking www.myclient.com (be patient)

Finished 2278 requests

Server Software:        Apache/2.2.3

Server Hostname:        www.myclient.com

Server Port:            80
 

Document Path:          /

Document Length:        55094 bytes
 

Concurrency Level:      100

Time taken for tests:   60.006 seconds

Complete requests:      2278

Failed requests:        0

Broken pipe errors:     0

Keep-Alive requests:    0

Total transferred:      128735146 bytes

HTML transferred:       128061121 bytes

Requests per second:    37.96 [#/sec] (mean)

Time per request:       2634.15 [ms] (mean)

Time per request:       26.34 [ms] (mean, across all concurrent requests)

Transfer rate:          2145.37 [Kbytes/sec] received
 

Connnection Times (ms)

              min  mean[+/-sd] median   max

Connect:       64   123  412.5     66  3143

Processing:   326  2429 1294.2   2258 12003

Waiting:      261  2428 1294.2   2258 12003

Total:        326  2552 1355.9   2346 12067
 

Percentage of the requests served within a certain time (ms)

  50%   2346

  66%   2848

  75%   3198

  80%   3466

  90%   4254

  95%   5148

  98%   6190

  99%   6960

 100%  12067 (last request)

Open in new window

0
 
LVL 57

Expert Comment

by:giltjr
ID: 21887555
--> Are you saying that the difference between the reported transfer rate(616.36 Kbytes/sec) and the client's capacity (1948.8 KB/sec) is due toslow start and overhead?  (The client is running Windows XP so maybethe TCP/IP receive buffer is overloaded?)<br><br>There are a few things that come into play here and TCP slow start is one of them.  <br><br>1) The link capacity. The slowest link 'wins'.  Meaning you can't send data any faster that the slowest link in the path.<br>2) The amount of available capacity on all  links in the path at the time of transfer.  The customer may have a 2MB/sec link and that may be the slow link.  However, if your side, their side, or someplace in the middle is at running at 99% capacity, you will may not get the throughput you expect.<br>3) The amount of data in the transfer.  Link utilization increases up to the maximum available bandwidth the larger the file is.  This is because just to start and stop a TCP connection takes 5 packets at a minimum.  If you have 1 packet worth of data, you will never see through put equal to the avalaible bandwidth.  Your test only has about 55K Bytes, or about 38 packets of data.<br>4) Latency/RTT.  You could have a 100Mbps connection and they could have a 100Mbps connection, however if you are 1/2 around the world from each other you would also have something in the range of 300-500 ms RTT.  This means that the 3-way TCP hand shake takes 900 to 1500 ms, so you are about 1 second in to the process before you send byte 1 of your data.  Then when you add the TCP slow start you could be about done with a 55,000 byte transfer by the time you get to sending a full window size. <br><br>You payload is only 38 packets of data.  The speed test that they did is most like a file that is somewher between 1MB and 5MB of data.<br><br>What I would suggest is that you create a 1MB, 5MB, and a 10 MB file to test with and see what you get with those file sizes.<br><br>You may also want to use something like wireshark (http://www.wireshark.org) to run a packet capture and see if you can find any thing.  It is possible that somewhere between your site and their's there is (was) congestion and there were a lot of packet drops. <br>
0
 

Author Comment

by:Geoff Millikan
ID: 21888111
>1) The link capacity. The slowest link 'wins'. Meaning you can't send data any faster that the slowest link in the path.

Understood.  I understand RFC 2001 pretty well so no need to dive into network topology basics.  

2) The amount of available capacity on all links in the path at the time of transfer.

Capacity was addressed above - we can move a lot of data very quickly.  On both ends of the pipe.

3) The amount of data in the transfer.  Your test only has about 55K Bytes, or about 38 packets of data.
4) Latency/RTT.

Ok.  So let's say that of the 1948.8 KB/sec transfer speed available, we allocate a whopping 40% of it to issues like normal handshaking, a extra slow start, jitter, network latency and some congestion on both sides (per pathping below, we can see that latency is about 12ms with a little bit of packet loss on one hop).  So with our 40% bandwidth reserved, we'd expect to see the transfer rate at 1169.28 KB/sec but it's not, it's at 616.36 [Kbytes/sec].

I think this is what you're saying, "It's normal to expect 70% of bandwidth to be eaten up in TCP/IP handshaking and TCP/IP overhead when loading a 55K web page 100 times a second."

Am I understanding you right?

>What I would suggest is that you create a 1MB, 5MB, and a 10 MB file to test with and see what you get with those file sizes.

Sounds good, I'll do this.

>It is possible that somewhere between your site and their's there is (was) congestion and there were a lot of packet drops.

Yes, I guess it's possible that packets are being mangled but that's pretty rare and hard to debug.  I'll save packet inspection for last if that's ok?

Any other thoughts?
windows>pathping www.myclient.com
 

Tracing route to www.myclient.com [208.64.252.230]

over a maximum of 30 hops:

  0  mycomputer [192.168.0.2]

  1  192.168.0.1

  2  L300.VFTTP-35.LSANCA.verizon-gni.net [71.177.216.1]

  3  P5-3.LCR-05.LSANCA.verizon-gni.net [130.81.58.240]

  4  so-6-3-0-0.BB-RTR1.LAX01.verizon-gni.net [130.81.29.69]

  5  0.so-6-2-0.XT1.LAX7.ALTER.NET [152.63.112.45]

  6  0.so-6-0-0.BR1.LAX7.ALTER.NET [152.63.112.41]

  7  204.255.169.38

  8  sl-crs2-ana-0-12-3-0.sprintlink.net [144.232.20.68]

  9  sl-crs1-ana-0-6-5-0.sprintlink.net [144.232.0.52]

 10  sl-st25-la-0-0-0.sprintlink.net [144.232.9.20]

 11  sl-intel9-171208-0.sprintlink.net [144.223.149.182]

 12  204.9.204.10.uscolo.com [204.9.204.10]

 13  18th02layer3.uscolo.com [204.9.204.30]

 14  www.myclient.com [208.64.252.230]
 

Computing statistics for 350 seconds...

            Source to Here   This Node/Link

Hop  RTT    Lost/Sent = Pct  Lost/Sent = Pct  Address

  0                                           mycomputer [192.168.0.2]

                                0/ 100 =  0%   |

  1    0ms     0/ 100 =  0%     0/ 100 =  0%  192.168.0.1

                                0/ 100 =  0%   |

  2    5ms     0/ 100 =  0%     0/ 100 =  0%  L300.VFTTP-35.LSANCA.verizon-gni.net [71.177.216.1]

                                0/ 100 =  0%   |

  3    6ms     0/ 100 =  0%     0/ 100 =  0%  P5-3.LCR-05.LSANCA.verizon-gni.net [130.81.58.240]

                                0/ 100 =  0%   |

  4    7ms     0/ 100 =  0%     0/ 100 =  0%  so-6-3-0-0.BB-RTR1.LAX01.verizon-gni.net [130.81.29.69]

                                0/ 100 =  0%   |

  5   10ms     0/ 100 =  0%     0/ 100 =  0%  0.so-6-2-0.XT1.LAX7.ALTER.NET [152.63.112.45]

                                0/ 100 =  0%   |

  6   10ms     0/ 100 =  0%     0/ 100 =  0%  0.so-6-0-0.BR1.LAX7.ALTER.NET [152.63.112.41]

                                0/ 100 =  0%   |

  7    7ms     0/ 100 =  0%     0/ 100 =  0%  204.255.169.38

                                0/ 100 =  0%   |

  8   10ms     0/ 100 =  0%     0/ 100 =  0%  sl-crs2-ana-0-12-3-0.sprintlink.net [144.232.20.68]

                                0/ 100 =  0%   |

  9   10ms     0/ 100 =  0%     0/ 100 =  0%  sl-crs1-ana-0-6-5-0.sprintlink.net [144.232.0.52]

                                0/ 100 =  0%   |

 10   12ms     0/ 100 =  0%     0/ 100 =  0%  sl-st25-la-0-0-0.sprintlink.net [144.232.9.20]

                                0/ 100 =  0%   |

 11   11ms     2/ 100 =  2%     2/ 100 =  2%  sl-intel9-171208-0.sprintlink.net [144.223.149.182]

                                0/ 100 =  0%   |

 12   13ms     0/ 100 =  0%     0/ 100 =  0%  204.9.204.10.uscolo.com [204.9.204.10]

                                0/ 100 =  0%   |

 13   10ms     0/ 100 =  0%     0/ 100 =  0%  18th02layer3.uscolo.com [204.9.204.30]

                                0/ 100 =  0%   |

 14   12ms     0/ 100 =  0%     0/ 100 =  0%  www.myclient.com [208.64.252.230]
 

Trace complete.

Open in new window

0
 

Author Comment

by:Geoff Millikan
ID: 21888386
So I did the test with the 10MB, 5MB and 1MB files (all text files).  Of course, this stresses the throughput on the web server, not its ability to serve pages rapidly to multiple clients which is what I'm trying to test.

During the tests I confirmed that the web server was moving data off at very fast rates (10 to 20 Mbps) using MTRG graphs.

I got lots of interesting errors during the 10MB test like, "apr_socket_recv: An existing connection was forcibly closed by the remote host.   (730054) \n Total of 109 requests completed" and "The entire post RQ could not be transmitted to the socket."  Doing any more than about 50 connections on the 10MB test resulted in errors.

The results below show that the transfer speed is faster (1896 Kbytes/sec) when benchmarking larger files however the size of the file didn't really seem to impact the transfer speed - the 1, 5, and 10 MB all came in at about the same transfer rate.

These results give some support to the theory that a portion of the bandwidth is taken up in ACK's/NAK's on the TCP/IP overhead.  However, I don't think it's possible that 70% of it is used - especially when Apache's Keep Alive is being used (which I would suspect would decrease some of the handshaking overhead).

So I've verified the web server can move data off at high speeds when a large file is involved but that really wasn't the question.  The question was:

Why does my web server respond slowly when doing 100 concurrent requests on a normal sized file?


## 1 MB Test ##

windows> ab -k -c 3 -v 2 -n 12  http://www.the-web-server.com/1MB.html
 

Document Path:          /1MB.html

Document Length:        992916 bytes
 

Concurrency Level:      3

Time taken for tests:   6.734375 seconds

Complete requests:      12

Failed requests:        0

Write errors:           0

Keep-Alive requests:    12

Total transferred:      12810356 bytes

HTML transferred:       12805195 bytes

Requests per second:    1.78 [#/sec] (mean)

Time per request:       1683.594 [ms] (mean)

Time per request:       561.198 [ms] (mean, across all concurrent requests)

Transfer rate:          1857.63 [Kbytes/sec] received
 

Connection Times (ms)

              min  mean[+/-sd] median   max

Connect:        0    2   5.9      0      15

Processing:   875 1037 147.7   1000    1390

Waiting:       15   35  16.7     31      62

Total:        875 1040 148.0   1015    1390
 

Percentage of the requests served within a certain time (ms)

  50%   1015

  66%   1031

  75%   1140

  80%   1140

  90%   1187

  95%   1390

  98%   1390

  99%   1390

 100%   1390 (longest request)
 

## 5MB Test ##############################

Document Path:          /5MB.html

Document Length:        4976371 bytes
 

Concurrency Level:      3

Time taken for tests:   33.234375 seconds

Complete requests:      12

Failed requests:        0

Write errors:           0

Keep-Alive requests:    12

Total transferred:      64323148 bytes

HTML transferred:       64317974 bytes

Requests per second:    0.36 [#/sec] (mean)

Time per request:       8308.594 [ms] (mean)

Time per request:       2769.531 [ms] (mean, across all concurrent requests)

Transfer rate:          1890.06 [Kbytes/sec] received
 

Connection Times (ms)

              min  mean[+/-sd] median   max

Connect:        0    5  10.0      0      31

Processing:  4390 7537 2682.1   6812   14281

Waiting:       15   52  17.0     62      78

Total:       4390 7542 2683.1   6812   14281
 

Percentage of the requests served within a certain time (ms)

  50%   6812

  66%   8359

  75%   8593

  80%   8593

  90%  10093

  95%  14281

  98%  14281

  99%  14281

 100%  14281 (longest request)
 

## 10MB Test ##############################

Document Path:          /10MB.html

Document Length:        9952742 bytes
 

Concurrency Level:      3

Time taken for tests:   63.125000 seconds

Complete requests:      12

Failed requests:        0

Write errors:           0

Keep-Alive requests:    12

Total transferred:      122602960 bytes

HTML transferred:       122597388 bytes

Requests per second:    0.19 [#/sec] (mean)

Time per request:       15781.250 [ms] (mean)

Time per request:       5260.417 [ms] (mean, across all concurrent requests)

Transfer rate:          1896.70 [Kbytes/sec] received
 

Connection Times (ms)

              min  mean[+/-sd] median   max

Connect:        0    3   6.8      0      15

Processing: 13015 15245 1494.9  14968   18750

Waiting:       15   62  34.6     62     109

Total:      13015 15249 1495.1  14968   18750
 

Percentage of the requests served within a certain time (ms)

  50%  14968

  66%  15312

  75%  16218

  80%  16218

  90%  16562

  95%  18750

  98%  18750

  99%  18750

 100%  18750 (longest request)

Open in new window

0
 
LVL 57

Accepted Solution

by:
giltjr earned 500 total points
ID: 21888627
With 12ms RTT latency is not going to be a real problem.

The second run was from a site with only a T1?  That is weird as it had a throughput of 2145 KB/sec, a T1 can only do about 190 KB/sec.  So I have no clue how you could get more throughput than a T1 capacity can provide.  Unless something local was doing caching.

O.K. I have never seen or heard of the Apache Bench mark tool before. I have tried to research it a bit as some of the stats don't make sense.

One thing I would try is reducing the concurrency level when you are running it remotely.  I would start with say 10 or 20 and see if that changes anything.

Since your computer shows a private IP address, there is something that is doing NAT.  If I understand the concurrency option, it means that ab will try and do 100 requests at once.  Which could be an issue with whatever is doing NAT for you.  I don't that for sure, but it is a possibility.

On the first remote run I would think that you definitely has some lost packets or there some  other type of network congestion.  It took 17+ second to process one request.  The only way 55K would have taken that long is:

   drop packet
   bad congestion
   or possibly something throttling bandwidth
0
 
LVL 57

Assisted Solution

by:giltjr
giltjr earned 500 total points
ID: 21888877
O.K. Some how I missed your 7:30 EDT post, so you can ignore most of my 9:03 post.

As for you 1, 5, and 10 MB test.   What this shows, it what I explained earlier, TCP slow start does not allow small files to take full advantage of bandwidth.  My guess is if you had a 500K, or even a 100K file you would also get close to the limit of the available bandwidth.  Its TCP slow start.

I've been playing with ab a little.  To show you how file size affect throughput I did a test and only got 2MB/sec throughput.  This was running ab on the web server.  Why so slow?  Easy the web page was only 3K.  So even though I was running ab on the same computer as the web server, because the page was so small the overhead of initiating and terminating the connection was so high that I only got 2MB/sec.  Running the same test but with a 200K page I got 72 MB/sec.  So 3K = 2MB/sec and 200K = 72MB/sec.

About the only way you could increase the throughput is to disable slow start.  The problem with doing this, is you could flood the network with data and cause dropped packets, which would cause more problems than slow start may be causing you.
0
Windows Server 2016: All you need to know

Learn about Hyper-V features that increase functionality and usability of Microsoft Windows Server 2016. Also, throughout this eBook, you’ll find some basic PowerShell examples that will help you leverage the scripts in your environments!

 

Author Comment

by:Geoff Millikan
ID: 21891154
Ok, that sounds good.

So the issue of network throughput has been tested.  I guess the question is now, "What Apache configuration is needed to produce the best web page response times using a 50K web page size?"  But that's a question for another thread.

This has been a very interesting thread, albeit time consuming.   Thanks so much for your help!
0
 
LVL 57

Expert Comment

by:giltjr
ID: 21891411
Glad to have been help.  This really was not all that time consuming, compare to other thread that may go on for weeks or even months.  Those are time consmuing,  a couple of days is nothing.

There not a whole lot you can do.  The issue is not with Apache.  If it was, then when you ran ab locally on the Apache server you would get terrible response time also.

You could tweak the IP stack so that there were certain things it would not do, but this could lead to other issues.

Which Linux distribution are you running Apache on?
0
 

Author Comment

by:Geoff Millikan
ID: 21891513
RedHat 5 - I believe typically the newer Linux kernels auto-adjust windows and buffer sizes so tuning them is ually not a good thing from what I read.

Yes, it seems that testing Apache locally returns pretty amazing results - it looks like it can handle 1000 requests per second pretty well returning pages in an average time of 181ms and a transfer rate of 56,600 KB/sec.

I think some of this is due to how Apache is configured - but it has a lot to do with hardware too.  Our prefork module is configured like this:

StartServers       8
MinSpareServers    6
MaxSpareServers   16
ServerLimit      500
MaxClients       500
MaxRequestsPerChild  4000
SendBufferSize 16384
[root@www /]# ab -c 190 -n 50000 http://www.our-web-site.com/

This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0

Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/

Copyright 2006 The Apache Software Foundation, http://www.apache.org/
 

Benchmarking www.our-web-site.com (be patient)

Completed 5000 requests

Completed 10000 requests

Completed 15000 requests

Completed 20000 requests

Completed 25000 requests

Completed 30000 requests

Completed 35000 requests

Completed 40000 requests

Completed 45000 requests

Finished 50000 requests
 
 

Server Software:        Apache/2.2.3

Server Hostname:        www.our-web-site.com

Server Port:            80
 

Document Path:          /

Document Length:        55094 bytes
 

Concurrency Level:      190

Time taken for tests:   47.777347 seconds

Complete requests:      50000

Failed requests:        0

Write errors:           0

Total transferred:      2769602094 bytes

HTML transferred:       2755348674 bytes

Requests per second:    1046.52 [#/sec] (mean)

Time per request:       181.554 [ms] (mean)

Time per request:       0.956 [ms] (mean, across all concurrent requests)

Transfer rate:          56610.28 [Kbytes/sec] received
 

Connection Times (ms)

              min  mean[+/-sd] median   max

Connect:        0   31 365.5      4   21000

Processing:    23  148  89.8    141    5883

Waiting:        2   75  93.7     60    5770

Total:         24  180 382.6    148   21133
 

Percentage of the requests served within a certain time (ms)

  50%    148

  66%    168

  75%    179

  80%    185

  90%    199

  95%    211

  98%    226

  99%    638

 100%  21133 (longest request)

Open in new window

0
 
LVL 57

Expert Comment

by:giltjr
ID: 21892105
Well, how many hits per second do you plan to have typically?  Based on what you have posted, my opinion is that the problem is networking, not Apache.  Why?  Easy, if the problem was Apace then when you ran the test on the localhost you would not have achieved over 50MB/sec throughput, you would have had performance problems there.

However, obviously you think you have a problem. So:

How much traffic do you expect to get?  What hardware are you running Apache on?   If you expect to have a high hit count with basically the same static information, then you need multiple processors.  Any of the following: multiple single/dual/quad core, single dual/quad core, or even single hyper-threading.  

How much static content are you going to have?  The more static content, the more RAM you want so that Apache can cache more in memory.  Starting with 2GB is s a good starting point, but I have run Apache on computers with 512MB with good performance, but I only averaged 10,000 hits a day.

If you have dynamic content, then you definitely need multiple fast processors.  If dynamic content are you running the database on the same box as Apache is running?  If so, the as many processors as you can put in the box and as much RAM as you can put in the box.


Explanation of what the parameters mean.

--> StartServers       8

Start 8 httpd daemons

--> MinSpareServers    6

Have at least 6 unused httpd daemons.  So if you get 3 requests right away, Apache will start a 9th daemon.

--> MaxSpareServers   16

If there are more than 16 unused httpd daemons, stop them.

--> ServerLimit      500

Do now allow more than 500 httpd daemons to be started at any point in time.

--> MaxClients       500

Maximum number of simultaneous requests Apache will handle.   ServerLimit needs to be equal to or greater than this.

--> MaxRequestsPerChild  4000

After a http daemon has handled 4,000 requests, stop it.  If needed start a new one.

0
 

Author Comment

by:Geoff Millikan
ID: 21919230
>How much traffic do you expect to get?

 We want to optimize our web site to expect an average of 5 requests per second (13MM hits a month) with a peak of 20 requests per second (50MM hits/month).

> What hardware are you running Apache on?

RAID 10, 10K RPM drives, 8 GB DDR3200 RAM, 64 Bit RedHat5, Two single core Opteron processors.  The server is pretty beefy.

If dynamic content are you running the database on the same box as Apache is running?  Yes, so there's no network overhead to get data from the database.   Also, I'm using mod_mem_cache so most of the site is getting servered out of memory instead of off the disk.

I've tested this box from a nearly identical server on a 45 Mbps connection and it still tests out at about 40 requests per second.  So I'm confused by when I test locally, I get 1000 reqs/second and when I test remote, I get 40 reqs/sec.  It seems to point to networking issues but I cannot isolate it - both sides run at over 45 Mbps which seems like enough to hold 1000 reqs/sec.

Thoughts?
0
 
LVL 57

Expert Comment

by:giltjr
ID: 21920743
I don't think you could get 1,000 hits per second with 55K per hit over a DS3. If I do my math right:

    1000 * 55000 * 8 = 440,000,000 bits per second.

Which is about ten times what a DS3 can do.  In fact its more that 100 Mbps Ethernet can do, and more than Gigabit ethernet can do with 1500 frames.  And that is just the raw data, that does not include the overhead of HTTP, TCP and IP.

At about 55K per page the best you could do about 90-95 hits per second (taking into account http, tcp, and ip network overhead)

Although running the database on the same box as Apache will cut down on network traffic, the problem is that you now have Apache and the database fighting for CPU, memory, and disk I/O.

This is why most people run them on separate boxes  using an isolated Gb Ethernet network using jumbo frames.  Now you can run them on the same box and if you have enough CPU, memory, and DISK I/O capacity it may not be a problem.

What I would suggest, based on 20 hits per second max is to change your pre-fork to:

StartServers       10
MinSpareServers    5
MaxSpareServers  10
ServerLimit      100
MaxClients       100
MaxRequestsPerChild  4000
SendBufferSize 16384

This should start 10, add new servers when ever you get below 5 unused servers, leave you at least 10 spare/unused servers, allow no more than 100, way more than you think you need.

You may want to analyze your file size (html pages, gifs, jpgs, js, css, whatever) It would be best if the majority of your files fit in one buffer.  So if the majority of your pages are 16K or less, then 16K is good for sendbuffersize.  However if the majority of your files are say 40K, then a larger sendbuffersize is better, fewer buffer gets.  But just increase it.  You don't want to be getter 32K or 64K of memory if 90% of your files are only 4K.
0
 

Author Comment

by:Geoff Millikan
ID: 21921933
Yep, I think you're right on the network traffic.  I ran the math and came out with needing a 438 Mbps connection to read a 55K file at 1000 times a second.  That's a lot.  I'm off to buy my Gig-E connection!  Now we know why Google owns their own network.  :-)

To get the TCP overhead, I think we can subtract the 10240000 bytes (HTML transferred) from the 13913347 bytes (Total transferred) which is 3,673,347 bytes.  That's 36% overhead!

Anyway, to confirm that it's the network limiting the requests/sec, I did the stress test on a 1KB file web page on the same 45 Mbps connection and I was able to run at 1339.56 requests per second.  So there you have it, a DS-3 isn't fast enough.

We've used your config with the exception of upping the MaxRequestsPerChild  to 10000.  Also,we're using "AcceptFilter http data" and "AcceptFilter https data" to queue incoming TCP.

So now if we get slashdotted, we're kind of prepared.  ;-)

http://en.wikipedia.org/wiki/Slashdot_effect

Thank you again!

Server Software:        Apache/2.2.3

Server Hostname:        www.our-web-site.com

Server Port:            80

 

Document Path:          /1KB.html

Document Length:        1024 bytes

 

Concurrency Level:      50

Time taken for tests:   7.465159 seconds

Complete requests:      10000

Failed requests:        0

Write errors:           0

Total transferred:      13913347 bytes

HTML transferred:       10240000 bytes

Requests per second:    1339.56 [#/sec] (mean)

Time per request:       37.326 [ms] (mean)

Time per request:       0.747 [ms] (mean, across all concurrent requests)

Transfer rate:          1820.06 [Kbytes/sec] received

 

Connection Times (ms)

              min  mean[+/-sd] median   max

Connect:        5   11  84.9      9    3013

Processing:     7   23  32.9     19     655

Waiting:        7   22  31.4     18     654

Total:         14   35  91.4     31    3232

 

Percentage of the requests served within a certain time (ms)

  50%     31

  66%     35

  75%     36

  80%     36

  90%     38

  95%     39

  98%     49

  99%    235

 100%   3232 (longest request)

Open in new window

0
 
LVL 57

Expert Comment

by:giltjr
ID: 21922082
Yep, overhead on a small page kills you.  I need to play with ab some more.  I am not sure that it actually includes ALL of the overhead, so there could be even more.


The only issue with maxrequestperchild is if you are doing scripting and you have a "bug" that leaks memory.  Ten thousands request could eat some memory.  But you know your site and so you are in a better position to make that determination.
0

Featured Post

What is SQL Server and how does it work?

The purpose of this paper is to provide you background on SQL Server. It’s your self-study guide for learning fundamentals. It includes both the history of SQL and its technical basics. Concepts and definitions will form the solid foundation of your future DBA expertise.

Join & Write a Comment

This document is written for Red Hat Enterprise Linux AS release 4 and ORACLE 10g.  Earlier releases can be installed using this document as well however there are some additional steps for packages to be installed see Metalink. Disclaimer: I hav…
Hi, in this article I'm going to teach you how to run your own site, and how to let people in (without IP). I'll talk about and explain each step... :) By the way, everything in this Tutorial is completely free and legal. This article is for …
Internet Business Fax to Email Made Easy - With eFax Corporate (http://www.enterprise.efax.com), you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, fr…
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now