Apache 2.2 > ApacheBench > Optimization > Data transfer rate?

What is your Internet connection?

When looking at throughput you have to look at multiple things.   The first thing is overhead

FTP: 1 tranfer of 38 MB, which means basically 1 TCP  connection.

HTTP: 739 tranfers totaling roughly 41 MB for an avg. of 55K per transfer.  This also means, most likley 739 TCP connections.

Each TCP connection requries 3 packets to start (3-way handshake) and two to end.

TCP uses what is called a "slow start" mechanism.  So that it does not flood the network with traffic TCP will send out two packets and see how long it takes to get back the ACK's.  It slowly starts to increase the number of packets it sends out up to the remote sides TCP window size.

So for  EACH HTTP transfer TCP will go through this slow start, or 739 times.

If you want to compare the Apache test to ftp, try doing 739 ftp transfers of a file that is about 55K.

Then create one HTML document that is 38MB and do one HTTP transfer.

 

giltjr: The client's download capacity just tested at 15590 kbps (1948.8 KB/sec transfer rate).  And the host should be capable of at least 100Mbps since it's in a fast collocation center. So we'd expect the "Transfer Rate" to be near the speed of the client, 1948.8 KB/sec - but it's not.

Are you saying that the difference between the reported transfer rate (616.36 Kbytes/sec) and the client's capacity (1948.8 KB/sec) is due to slow start and overhead?  (The client is running Windows XP so maybe the TCP/IP receive buffer is overloaded?)

P.S.  I attached another test below, this time from a different client location on a T1 (1.5 Mbps) Internet connection.

FreeBSD> ab -kc 100 -t 60 http://www.myclient.com/
This is ApacheBench, Version 1.3d <$Revision: 1.73 $> apache-1.3
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 2006 The Apache Software Foundation, http://www.apache.org/
 
Benchmarking www.myclient.com (be patient)
Finished 2278 requests
Server Software:        Apache/2.2.3
Server Hostname:        www.myclient.com
Server Port:            80
 
Document Path:          /
Document Length:        55094 bytes
 
Concurrency Level:      100
Time taken for tests:   60.006 seconds
Complete requests:      2278
Failed requests:        0
Broken pipe errors:     0
Keep-Alive requests:    0
Total transferred:      128735146 bytes
HTML transferred:       128061121 bytes
Requests per second:    37.96 [#/sec] (mean)
Time per request:       2634.15 [ms] (mean)
Time per request:       26.34 [ms] (mean, across all concurrent requests)
Transfer rate:          2145.37 [Kbytes/sec] received
 
Connnection Times (ms)
              min  mean[+/-sd] median   max
Connect:       64   123  412.5     66  3143
Processing:   326  2429 1294.2   2258 12003
Waiting:      261  2428 1294.2   2258 12003
Total:        326  2552 1355.9   2346 12067
 
Percentage of the requests served within a certain time (ms)
  50%   2346
  66%   2848
  75%   3198
  80%   3466
  90%   4254
  95%   5148
  98%   6190
  99%   6960
 100%  12067 (last request)

Select allOpen in new window

--> Are you saying that the difference between the reported transfer rate(616.36 Kbytes/sec) and the client's capacity (1948.8 KB/sec) is due toslow start and overhead?  (The client is running Windows XP so maybethe TCP/IP receive buffer is overloaded?)<br><br>There are a few things that come into play here and TCP slow start is one of them.  <br><br>1) The link capacity. The slowest link 'wins'.  Meaning you can't send data any faster that the slowest link in the path.<br>2) The amount of available capacity on all  links in the path at the time of transfer.  The customer may have a 2MB/sec link and that may be the slow link.  However, if your side, their side, or someplace in the middle is at running at 99% capacity, you will may not get the throughput you expect.<br>3) The amount of data in the transfer.  Link utilization increases up to the maximum available bandwidth the larger the file is.  This is because just to start and stop a TCP connection takes 5 packets at a minimum.  If you have 1 packet worth of data, you will never see through put equal to the avalaible bandwidth.  Your test only has about 55K Bytes, or about 38 packets of data.<br>4) Latency/RTT.  You could have a 100Mbps connection and they could have a 100Mbps connection, however if you are 1/2 around the world from each other you would also have something in the range of 300-500 ms RTT.  This means that the 3-way TCP hand shake takes 900 to 1500 ms, so you are about 1 second in to the process before you send byte 1 of your data.  Then when you add the TCP slow start you could be about done with a 55,000 byte transfer by the time you get to sending a full window size. <br><br>You payload is only 38 packets of data.  The speed test that they did is most like a file that is somewher between 1MB and 5MB of data.<br><br>What I would suggest is that you create a 1MB, 5MB, and a 10 MB file to test with and see what you get with those file sizes.<br><br>You may also want to use something like wireshark (http://www.wireshark.org) to run a packet capture and see if you can find any thing.  It is possible that somewhere between your site and their's there is (was) congestion and there were a lot of packet drops. <br>

>1) The link capacity. The slowest link 'wins'. Meaning you can't send data any faster that the slowest link in the path.

Understood.  I understand RFC 2001 pretty well so no need to dive into network topology basics.  

2) The amount of available capacity on all links in the path at the time of transfer.

Capacity was addressed above - we can move a lot of data very quickly.  On both ends of the pipe.

3) The amount of data in the transfer.  Your test only has about 55K Bytes, or about 38 packets of data.
4) Latency/RTT.

Ok.  So let's say that of the 1948.8 KB/sec transfer speed available, we allocate a whopping 40% of it to issues like normal handshaking, a extra slow start, jitter, network latency and some congestion on both sides (per pathping below, we can see that latency is about 12ms with a little bit of packet loss on one hop).  So with our 40% bandwidth reserved, we'd expect to see the transfer rate at 1169.28 KB/sec but it's not, it's at 616.36 [Kbytes/sec].

I think this is what you're saying, "It's normal to expect 70% of bandwidth to be eaten up in TCP/IP handshaking and TCP/IP overhead when loading a 55K web page 100 times a second."

Am I understanding you right?

>What I would suggest is that you create a 1MB, 5MB, and a 10 MB file to test with and see what you get with those file sizes.

Sounds good, I'll do this.

>It is possible that somewhere between your site and their's there is (was) congestion and there were a lot of packet drops.

Yes, I guess it's possible that packets are being mangled but that's pretty rare and hard to debug.  I'll save packet inspection for last if that's ok?

Any other thoughts?

windows>pathping www.myclient.com
 
Tracing route to www.myclient.com [208.64.252.230]
over a maximum of 30 hops:
  0  mycomputer [192.168.0.2]
  1  192.168.0.1
  2  L300.VFTTP-35.LSANCA.verizon-gni.net [71.177.216.1]
  3  P5-3.LCR-05.LSANCA.verizon-gni.net [130.81.58.240]
  4  so-6-3-0-0.BB-RTR1.LAX01.verizon-gni.net [130.81.29.69]
  5  0.so-6-2-0.XT1.LAX7.ALTER.NET [152.63.112.45]
  6  0.so-6-0-0.BR1.LAX7.ALTER.NET [152.63.112.41]
  7  204.255.169.38
  8  sl-crs2-ana-0-12-3-0.sprintlink.net [144.232.20.68]
  9  sl-crs1-ana-0-6-5-0.sprintlink.net [144.232.0.52]
 10  sl-st25-la-0-0-0.sprintlink.net [144.232.9.20]
 11  sl-intel9-171208-0.sprintlink.net [144.223.149.182]
 12  204.9.204.10.uscolo.com [204.9.204.10]
 13  18th02layer3.uscolo.com [204.9.204.30]
 14  www.myclient.com [208.64.252.230]
 
Computing statistics for 350 seconds...
            Source to Here   This Node/Link
Hop  RTT    Lost/Sent = Pct  Lost/Sent = Pct  Address
  0                                           mycomputer [192.168.0.2]
                                0/ 100 =  0%   |
  1    0ms     0/ 100 =  0%     0/ 100 =  0%  192.168.0.1
                                0/ 100 =  0%   |
  2    5ms     0/ 100 =  0%     0/ 100 =  0%  L300.VFTTP-35.LSANCA.verizon-gni.net [71.177.216.1]
                                0/ 100 =  0%   |
  3    6ms     0/ 100 =  0%     0/ 100 =  0%  P5-3.LCR-05.LSANCA.verizon-gni.net [130.81.58.240]
                                0/ 100 =  0%   |
  4    7ms     0/ 100 =  0%     0/ 100 =  0%  so-6-3-0-0.BB-RTR1.LAX01.verizon-gni.net [130.81.29.69]
                                0/ 100 =  0%   |
  5   10ms     0/ 100 =  0%     0/ 100 =  0%  0.so-6-2-0.XT1.LAX7.ALTER.NET [152.63.112.45]
                                0/ 100 =  0%   |
  6   10ms     0/ 100 =  0%     0/ 100 =  0%  0.so-6-0-0.BR1.LAX7.ALTER.NET [152.63.112.41]
                                0/ 100 =  0%   |
  7    7ms     0/ 100 =  0%     0/ 100 =  0%  204.255.169.38
                                0/ 100 =  0%   |
  8   10ms     0/ 100 =  0%     0/ 100 =  0%  sl-crs2-ana-0-12-3-0.sprintlink.net [144.232.20.68]
                                0/ 100 =  0%   |
  9   10ms     0/ 100 =  0%     0/ 100 =  0%  sl-crs1-ana-0-6-5-0.sprintlink.net [144.232.0.52]
                                0/ 100 =  0%   |
 10   12ms     0/ 100 =  0%     0/ 100 =  0%  sl-st25-la-0-0-0.sprintlink.net [144.232.9.20]
                                0/ 100 =  0%   |
 11   11ms     2/ 100 =  2%     2/ 100 =  2%  sl-intel9-171208-0.sprintlink.net [144.223.149.182]
                                0/ 100 =  0%   |
 12   13ms     0/ 100 =  0%     0/ 100 =  0%  204.9.204.10.uscolo.com [204.9.204.10]
                                0/ 100 =  0%   |
 13   10ms     0/ 100 =  0%     0/ 100 =  0%  18th02layer3.uscolo.com [204.9.204.30]
                                0/ 100 =  0%   |
 14   12ms     0/ 100 =  0%     0/ 100 =  0%  www.myclient.com [208.64.252.230]
 
Trace complete.

Select allOpen in new window

So I did the test with the 10MB, 5MB and 1MB files (all text files).  Of course, this stresses the throughput on the web server, not its ability to serve pages rapidly to multiple clients which is what I'm trying to test.

During the tests I confirmed that the web server was moving data off at very fast rates (10 to 20 Mbps) using MTRG graphs.

I got lots of interesting errors during the 10MB test like, "apr_socket_recv: An existing connection was forcibly closed by the remote host.   (730054) \n Total of 109 requests completed" and "The entire post RQ could not be transmitted to the socket."  Doing any more than about 50 connections on the 10MB test resulted in errors.

The results below show that the transfer speed is faster (1896 Kbytes/sec) when benchmarking larger files however the size of the file didn't really seem to impact the transfer speed - the 1, 5, and 10 MB all came in at about the same transfer rate.

These results give some support to the theory that a portion of the bandwidth is taken up in ACK's/NAK's on the TCP/IP overhead.  However, I don't think it's possible that 70% of it is used - especially when Apache's Keep Alive is being used (which I would suspect would decrease some of the handshaking overhead).

So I've verified the web server can move data off at high speeds when a large file is involved but that really wasn't the question.  The question was:

Why does my web server respond slowly when doing 100 concurrent requests on a normal sized file?

## 1 MB Test ##
windows> ab -k -c 3 -v 2 -n 12  http://www.the-web-server.com/1MB.html
 
Document Path:          /1MB.html
Document Length:        992916 bytes
 
Concurrency Level:      3
Time taken for tests:   6.734375 seconds
Complete requests:      12
Failed requests:        0
Write errors:           0
Keep-Alive requests:    12
Total transferred:      12810356 bytes
HTML transferred:       12805195 bytes
Requests per second:    1.78 [#/sec] (mean)
Time per request:       1683.594 [ms] (mean)
Time per request:       561.198 [ms] (mean, across all concurrent requests)
Transfer rate:          1857.63 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2   5.9      0      15
Processing:   875 1037 147.7   1000    1390
Waiting:       15   35  16.7     31      62
Total:        875 1040 148.0   1015    1390
 
Percentage of the requests served within a certain time (ms)
  50%   1015
  66%   1031
  75%   1140
  80%   1140
  90%   1187
  95%   1390
  98%   1390
  99%   1390
 100%   1390 (longest request)
 
## 5MB Test ##############################
Document Path:          /5MB.html
Document Length:        4976371 bytes
 
Concurrency Level:      3
Time taken for tests:   33.234375 seconds
Complete requests:      12
Failed requests:        0
Write errors:           0
Keep-Alive requests:    12
Total transferred:      64323148 bytes
HTML transferred:       64317974 bytes
Requests per second:    0.36 [#/sec] (mean)
Time per request:       8308.594 [ms] (mean)
Time per request:       2769.531 [ms] (mean, across all concurrent requests)
Transfer rate:          1890.06 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    5  10.0      0      31
Processing:  4390 7537 2682.1   6812   14281
Waiting:       15   52  17.0     62      78
Total:       4390 7542 2683.1   6812   14281
 
Percentage of the requests served within a certain time (ms)
  50%   6812
  66%   8359
  75%   8593
  80%   8593
  90%  10093
  95%  14281
  98%  14281
  99%  14281
 100%  14281 (longest request)
 
## 10MB Test ##############################
Document Path:          /10MB.html
Document Length:        9952742 bytes
 
Concurrency Level:      3
Time taken for tests:   63.125000 seconds
Complete requests:      12
Failed requests:        0
Write errors:           0
Keep-Alive requests:    12
Total transferred:      122602960 bytes
HTML transferred:       122597388 bytes
Requests per second:    0.19 [#/sec] (mean)
Time per request:       15781.250 [ms] (mean)
Time per request:       5260.417 [ms] (mean, across all concurrent requests)
Transfer rate:          1896.70 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    3   6.8      0      15
Processing: 13015 15245 1494.9  14968   18750
Waiting:       15   62  34.6     62     109
Total:      13015 15249 1495.1  14968   18750
 
Percentage of the requests served within a certain time (ms)
  50%  14968
  66%  15312
  75%  16218
  80%  16218
  90%  16562
  95%  18750
  98%  18750
  99%  18750
 100%  18750 (longest request)

Select allOpen in new window

Ok, that sounds good.

So the issue of network throughput has been tested.  I guess the question is now, "What Apache configuration is needed to produce the best web page response times using a 50K web page size?"  But that's a question for another thread.

This has been a very interesting thread, albeit time consuming.   Thanks so much for your help!

Glad to have been help.  This really was not all that time consuming, compare to other thread that may go on for weeks or even months.  Those are time consmuing,  a couple of days is nothing.

There not a whole lot you can do.  The issue is not with Apache.  If it was, then when you ran ab locally on the Apache server you would get terrible response time also.

You could tweak the IP stack so that there were certain things it would not do, but this could lead to other issues.

Which Linux distribution are you running Apache on?

RedHat 5 - I believe typically the newer Linux kernels auto-adjust windows and buffer sizes so tuning them is ually not a good thing from what I read.

Yes, it seems that testing Apache locally returns pretty amazing results - it looks like it can handle 1000 requests per second pretty well returning pages in an average time of 181ms and a transfer rate of 56,600 KB/sec.

I think some of this is due to how Apache is configured - but it has a lot to do with hardware too.  Our prefork module is configured like this:

StartServers       8
MinSpareServers    6
MaxSpareServers   16
ServerLimit      500
MaxClients       500
MaxRequestsPerChild  4000
SendBufferSize 16384

[root@www /]# ab -c 190 -n 50000 http://www.our-web-site.com/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
 
Benchmarking www.our-web-site.com (be patient)
Completed 5000 requests
Completed 10000 requests
Completed 15000 requests
Completed 20000 requests
Completed 25000 requests
Completed 30000 requests
Completed 35000 requests
Completed 40000 requests
Completed 45000 requests
Finished 50000 requests
 
 
Server Software:        Apache/2.2.3
Server Hostname:        www.our-web-site.com
Server Port:            80
 
Document Path:          /
Document Length:        55094 bytes
 
Concurrency Level:      190
Time taken for tests:   47.777347 seconds
Complete requests:      50000
Failed requests:        0
Write errors:           0
Total transferred:      2769602094 bytes
HTML transferred:       2755348674 bytes
Requests per second:    1046.52 [#/sec] (mean)
Time per request:       181.554 [ms] (mean)
Time per request:       0.956 [ms] (mean, across all concurrent requests)
Transfer rate:          56610.28 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   31 365.5      4   21000
Processing:    23  148  89.8    141    5883
Waiting:        2   75  93.7     60    5770
Total:         24  180 382.6    148   21133
 
Percentage of the requests served within a certain time (ms)
  50%    148
  66%    168
  75%    179
  80%    185
  90%    199
  95%    211
  98%    226
  99%    638
 100%  21133 (longest request)

Select allOpen in new window

Well, how many hits per second do you plan to have typically?  Based on what you have posted, my opinion is that the problem is networking, not Apache.  Why?  Easy, if the problem was Apace then when you ran the test on the localhost you would not have achieved over 50MB/sec throughput, you would have had performance problems there.

However, obviously you think you have a problem. So:

How much traffic do you expect to get?  What hardware are you running Apache on?   If you expect to have a high hit count with basically the same static information, then you need multiple processors.  Any of the following: multiple single/dual/quad core, single dual/quad core, or even single hyper-threading.  

How much static content are you going to have?  The more static content, the more RAM you want so that Apache can cache more in memory.  Starting with 2GB is s a good starting point, but I have run Apache on computers with 512MB with good performance, but I only averaged 10,000 hits a day.

If you have dynamic content, then you definitely need multiple fast processors.  If dynamic content are you running the database on the same box as Apache is running?  If so, the as many processors as you can put in the box and as much RAM as you can put in the box.


Explanation of what the parameters mean.

--> StartServers       8

Start 8 httpd daemons

--> MinSpareServers    6

Have at least 6 unused httpd daemons.  So if you get 3 requests right away, Apache will start a 9th daemon.

--> MaxSpareServers   16

If there are more than 16 unused httpd daemons, stop them.

--> ServerLimit      500

Do now allow more than 500 httpd daemons to be started at any point in time.

--> MaxClients       500

Maximum number of simultaneous requests Apache will handle.   ServerLimit needs to be equal to or greater than this.

--> MaxRequestsPerChild  4000

After a http daemon has handled 4,000 requests, stop it.  If needed start a new one.

>How much traffic do you expect to get?

 We want to optimize our web site to expect an average of 5 requests per second (13MM hits a month) with a peak of 20 requests per second (50MM hits/month).

> What hardware are you running Apache on?

RAID 10, 10K RPM drives, 8 GB DDR3200 RAM, 64 Bit RedHat5, Two single core Opteron processors.  The server is pretty beefy.

If dynamic content are you running the database on the same box as Apache is running?  Yes, so there's no network overhead to get data from the database.   Also, I'm using mod_mem_cache so most of the site is getting servered out of memory instead of off the disk.

I've tested this box from a nearly identical server on a 45 Mbps connection and it still tests out at about 40 requests per second.  So I'm confused by when I test locally, I get 1000 reqs/second and when I test remote, I get 40 reqs/sec.  It seems to point to networking issues but I cannot isolate it - both sides run at over 45 Mbps which seems like enough to hold 1000 reqs/sec.

Thoughts?

I don't think you could get 1,000 hits per second with 55K per hit over a DS3. If I do my math right:

    1000 * 55000 * 8 = 440,000,000 bits per second.

Which is about ten times what a DS3 can do.  In fact its more that 100 Mbps Ethernet can do, and more than Gigabit ethernet can do with 1500 frames.  And that is just the raw data, that does not include the overhead of HTTP, TCP and IP.

At about 55K per page the best you could do about 90-95 hits per second (taking into account http, tcp, and ip network overhead)

Although running the database on the same box as Apache will cut down on network traffic, the problem is that you now have Apache and the database fighting for CPU, memory, and disk I/O.

This is why most people run them on separate boxes  using an isolated Gb Ethernet network using jumbo frames.  Now you can run them on the same box and if you have enough CPU, memory, and DISK I/O capacity it may not be a problem.

What I would suggest, based on 20 hits per second max is to change your pre-fork to:

StartServers       10
MinSpareServers    5
MaxSpareServers  10
ServerLimit      100
MaxClients       100
MaxRequestsPerChild  4000
SendBufferSize 16384

This should start 10, add new servers when ever you get below 5 unused servers, leave you at least 10 spare/unused servers, allow no more than 100, way more than you think you need.

You may want to analyze your file size (html pages, gifs, jpgs, js, css, whatever) It would be best if the majority of your files fit in one buffer.  So if the majority of your pages are 16K or less, then 16K is good for sendbuffersize.  However if the majority of your files are say 40K, then a larger sendbuffersize is better, fewer buffer gets.  But just increase it.  You don't want to be getter 32K or 64K of memory if 90% of your files are only 4K.

Yep, I think you're right on the network traffic.  I ran the math and came out with needing a 438 Mbps connection to read a 55K file at 1000 times a second.  That's a lot.  I'm off to buy my Gig-E connection!  Now we know why Google owns their own network.  :-)

To get the TCP overhead, I think we can subtract the 10240000 bytes (HTML transferred) from the 13913347 bytes (Total transferred) which is 3,673,347 bytes.  That's 36% overhead!

Anyway, to confirm that it's the network limiting the requests/sec, I did the stress test on a 1KB file web page on the same 45 Mbps connection and I was able to run at 1339.56 requests per second.  So there you have it, a DS-3 isn't fast enough.

We've used your config with the exception of upping the MaxRequestsPerChild  to 10000.  Also,we're using "AcceptFilter http data" and "AcceptFilter https data" to queue incoming TCP.

So now if we get slashdotted, we're kind of prepared.  ;-)

http://en.wikipedia.org/wiki/Slashdot_effect

Thank you again!

Server Software:        Apache/2.2.3
Server Hostname:        www.our-web-site.com
Server Port:            80
 
Document Path:          /1KB.html
Document Length:        1024 bytes
 
Concurrency Level:      50
Time taken for tests:   7.465159 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      13913347 bytes
HTML transferred:       10240000 bytes
Requests per second:    1339.56 [#/sec] (mean)
Time per request:       37.326 [ms] (mean)
Time per request:       0.747 [ms] (mean, across all concurrent requests)
Transfer rate:          1820.06 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        5   11  84.9      9    3013
Processing:     7   23  32.9     19     655
Waiting:        7   22  31.4     18     654
Total:         14   35  91.4     31    3232
 
Percentage of the requests served within a certain time (ms)
  50%     31
  66%     35
  75%     36
  80%     36
  90%     38
  95%     39
  98%     49
  99%    235
 100%   3232 (longest request)

Select allOpen in new window

Yep, overhead on a small page kills you.  I need to play with ab some more.  I am not sure that it actually includes ALL of the overhead, so there could be even more.


The only issue with maxrequestperchild is if you are doing scripting and you have a "bug" that leaks memory.  Ten thousands request could eat some memory.  But you know your site and so you are in a better position to make that determination.