• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 630
  • Last Modified:

2 http apache servers, 1 is slow, 1 is fast

I have 2 severs, server S (for slow) and server F (for fast). Both run Linux and apache. When downloading the same 50MB file containing random binary data from a 3rd host ~200ms (latency) away using wget, I get substantially different results. When downloading from server F, the transfer will run at 40-90mbps whereas when I download the same file from server S, the transfer will run at 1-3mbps.

Alternatively, I can download the file from my ADSL connection (20ms latency) and saturate the link. My ADSL connection is ~16mbps.

I can also download the file at gigabit speed from both hosts when on the local network.

Both servers were idle, with very little IO or network activity when the tests were performed.

Some information about the 2 servers:
- Both are on the same network
- Both servers are virtualised xen hosts
- Both run apache 2.2.11
- Both servers have the same setting for /proc/sys/net/ipv4/tcp_wmem (4096  16384  4194304)
- Both servers have the same setting for /proc/sys/net/core/wmem_max (131071)
- Both servers have syncookies on - changing this makes no difference
- Both servers have tcp timestamps off - changing this makes no difference
- Both servers have window scaling turned on (tcp_window_scaling=1). In addition to this, using tcpdump shows wscale=7 during the TCP options negotiation for connections to both servers
- Server F runs a 2.6.27 kernel and server S runs a 2.6.28 kernel. I have tried running server S on the same 2.6.27 kernel but it makes no difference.
- Server F has keepalive off, while server S has it on, but turning it off makes no difference
- Server S also handles SSL and PHP/FCGID, but disabling both of these makes no difference
- Server S has 4 cpus  and 4GB of ram (Server F has 1 cpu and 256MB ram), but cutting server S down to 1 cpu and 256MB ram makes no difference
- Running both virtualised servers on the same physical host makes no difference
- Server S has higher apache MPM settings, but changing these to the same as that on server F makes no difference.
- When the file was being downloaded from server F, netstat shows a Send-Q of 2-3MB, while on server S it is only about 100-300KB.

In summary, the issue appears to only affect high latency connections (in this case it's trans-continental). The file is downloaded at 40-90mbps from server F while only at 1-3mbps from server S.

Ideas, thoughts, questions?
  • 16
  • 14
1 Solution
Can you do another capture using tcpdump and use something like Wireshark to look at it and see if you can tell an obvious difference?

Such as server "F" is filling the TCP Window size, and server "S" is not filling the Windows size.

Have you tried running server "S" on the same OS as server "F"?
bplantAuthor Commented:
I have tried running server S with the same kernel as server F. All other packages are the same version.

I used tcpdump to capture all the packet headers for the 50MB file download. I found the file was downloaded from server F using 51824 packets, while server S needed 55223. Using wireshark's TCP analysis, server S received 4373 duplicate ACK's while server F only received 2670.

I also noticed that the majority of duplicate ACK's for received by server F were in 2 large groups occurring at around packet number 10,000 and 30,000. The duplicate ACK's received by server S on the other hand were spread out into much smaller groups spread throughout the entire stream.

I'm not sure if the duplicate ACK's information is relevant. I've tried hosting the both virtual hosts on the same physical host meaning that packets would take the exact same route so any network congestion etc should be the same.

The receive window grew to in excess of 3MB when downloading from both machines. As for the size of the transmit window (bytes in transit), I looked and difference between the TCP sequence number at the ACK number at some random intervals listed below.

~packet #7163
Server S: ~35KB
Server F: ~2.45MB

~packet #23782
Server S: ~17.5KB
Server F: ~2.17MB

~packet #40681
Server S: ~67KB
Server F: ~1.73MB

Quite a clear difference and is in line with the observed Send-Q values seen on both servers.
I'm not sure what you mean buy running it with the same kernel.  Do you mean that you took the httpd.conf files from server 'S' and put on server 'F'?  If not, that is what I am suggesting that you do.

If possible can you setup FTP servers on both of the Linux systems and do some test with FTP?  This will help point to either something weird with Apache, or something in the Linux TCP configuration.

Receiving a lot of duplicate ACK's indicates that the receiver received the packet and expected the next one and did not receive if within "x" amount of time so it assumes that the original ACK got lost.

Now, one other thing could be that with the way the firewalls are set up, some how server "S" is having its bandwidth throttled and server "F" is not.  The easiest way to throttle bandwidth is to drop packets, which would account for a increased number of duplicate ACK's.  So you may want to check and make sure there are no firewalls (or any other devices) that could be throttling bandwidth.
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

bplantAuthor Commented:
The kernel is what boots the computer and what's responsible for the accessing disks, network and handling the TCP communication.

I have also tried using the apache configuration from server F on server S.

When both virtual machines were running on the same physical host, they were protected by the same firewall with the same rules for each machine.

I will check out FTP and report back.
I understand what the kernel is.

However, on a single Linux system you can have multiple kernels that you choose to boot from.

Since you said you ran "S" under the same kernel level as "F" I could not tell if you copied "S"'s config to the same server that "F" was running on, or if you installed a second kernel on the server that "S" is running on and just booted that linux system under a different kernel.

The reason I asked is just because you are running the same kernel level, does not mean that all of the system parameters are the same.

However, since you did copy "S" config to "F"'s sever that should elimate both Linux system configuration options and firewall issues on the server side (unless you are using a different port for "S" and the firewall is doing this by port).

bplantAuthor Commented:
Re the kernel I was meaning that I booted server S with the same kernel bzImage as what was used to boot server F. In addition to this, I also tried coping the apache configuration and sysctl settings. I did not copy any other configuration options.

I tried transferring the file using FTP on server S and the speed was similar to HTTP on server S. A few runs of netstat -nt also showed Send-Q never got larger that 98KB.

Re the firewall, there is no bandwidth throttling present and the firewall is external to the virtual machines.
Can you try coping just the Apache configuration and not the sysctl?  If you do FTP on the "F" server is it fast?

With the send-q not getting large that indicates that some how the "S" server is not attempting to fill the TCP window size.  Since this is only occurring on the link with high latency it indicates that there are delays in the "S" server receiving the ACK's.

Does the packet capture show delays on the ACK's?
"coping" should be "copying"
bplantAuthor Commented:
I already had FTP setup on server S, but not on server F so I haven't tested FTP on server F. I've no doubt it will be fast though.

I checked the RTT from the packet dumps and it's ~180ms in both cases.

I agree that server S is not attempting to fill the TCP window. It's like it's being limited by /proc/sys/net/ipv4/tcp_wmem, but both servers specify a window scaling factor of 7 when sending the SYNACK in the TCP handshake.
Do both systems have the same value for  /proc/sys/net/ipv4/tcp_mem?

Well, /proc/sys/net/ipv4/tcp_wmem is only one way that the transmit window size id decided.

Is it possible for you to run a packet capture on the client side?

What is the bandwidth to the Internet on the server side and on the client side?
bplantAuthor Commented:
The 2 servers have different values for tcp_mem because server S has 4GB of RAM campared to 256MB for server F.
Server F: 24576      32768      49152
Server S: 393216      524288      786432

When did try booting server S with only 256MB of RAM at one stage so tcp_mem should have been the same then.

Both servers have the following for /proc/sys/net/ipv4/tcp_wmem: 4096      16384      4194304

The servers have a 1gbps connection and the client has a 100mbps connection. Before you ask, this is bandwidth to the Internet, not just the interface speed.

I could run a packet capture on the client side. What would I be looking for though?
Delay's in packets arriving or lost packets.

In an early post you stated that server "S" got a about 4,000 duplicate acks.  This indicates that the client was missing the next incoming packet.

If the receiver is sending out duplicate acks, then the sender will not fill the window size because it (the sender) believes that it is flooding something on the network and that packets are getting dropped.

Which so far everything you have described for "S" indicates that packets are getting dropped, the duplicate ACKs as well as the the much lower transmit queue size.

When you copied the "S" configuration to "F", did "S" IP address move with it?  If so, can you flip "S" and "F"'s IP address?
bplantAuthor Commented:
The only configuration I copied from server F to S (not S to F) was the apache configuration (/etc/apache2) and the sysctl settings (/etc/sysctl.conf). I NEVER copied anything else.

I cannot change any IP addresses as these servers are in production.

Out of curiosity I looked at the Send-Q value reported by netstat when downloading from both a local machine and to my 16mbps ADSL connection. For the local gigabit connected host Send-Q was around the 2.5MB mark while for my ADSL connection it was 250-350KB. So server S _is_ capable of filling the window.

I will do a packet capture now on the client side and see if it shows anything.
What all are the differences between the two sysctl.conf files?

bplantAuthor Commented:
Server S is on the left, server F is on the right.

I have tried changing min_free_kbytes on server S to 8192 (same as server F), but it made no difference.

< net.netfilter.nf_conntrack_max = 65536
< # Increase the udp timeout as some DNS queries take longer than 30 seconds
< # (the default conntrack timeout) when the target nameserver cannot be reached
< net.ipv4.netfilter.ip_conntrack_udp_timeout = 60

< # Decrease tcp established timeout to 24 hours (from 5 days)
< net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 86400

< vm.min_free_kbytes = 131072
> vm.min_free_kbytes = 8192
Some how the post got messed up, so I'm not sure about some of the parms.

The only one that would be nice to know is the conntrack_max.  I really can't tell what it is on either box.

I did notice that you increased the udp timeout from30 to 60 seconds and your comments mentions DNS lookups.  Did you change the timeout in /etc/resolver.conf also?  By default it is 5 seconds.

min_free_kbytes should not make any difference, especially with 4GB  of RAM, unless by chance this box is very memory constrained.
bplantAuthor Commented:
There isn't meant to be 2 boxes in the previous post. Lines preceded with a "<" are from the left side (server S) and lines preceded by a ">" are from the right side (server F).

Server F has it's default conntrack_max of 8192 while server S has been increased to 65536.

None of my comments mention DNS. I have been accessing both hosts by IP address so DNS isn't even being used.

The box is not under any memory pressure. The box currently has over 2GB free and is under low load.
You may want re-read your post with the sysctl differences.  What you posted says:

"< # Increase the udp timeout as some DNS queries take longer than 30 seconds
< # (the default conntrack timeout) when the target nameserver cannot be reached"

I am assuming they are your comments.  My comment is more of a F.Y.I.  Changing the upd timeout does not change the timeout for DSN requests if it is defaulted to 5 seconds or is specified in /etc/resolve.conf and it is 30 seconds or less.  

What distribution of Linux are you using?  Typically the default for conntrack_max is 32,768.
bplantAuthor Commented:
That was a comment in sysctl.conf. It showed up because it was in sysctl.conf on server S, but not on server F. Rest assured that DNS is not used in this instance.

I am using Gentoo Linux.

I believe the default nf_conntrack_max is calculated at boot time based on the amount of RAM available to the host.

I've taken packet captures on the client side and I've been looking at them in wireshark. Like last time, the download from server F only had duplicate ACK's sent in 1 of 2 large groups. When this happened I can see based on the sequence numbers that there were some missing packets.

Also like last time, the duplicate ACK's were much more spread out through the download from server S. Unlike the download from server F however, I cannot determine the reason for the duplicate ACK's being sent to server S. The sequence numbers seem to be in order but the client is indicating via SACK that a packet is missing.

Here's an attempt to describe what wireshark is showing:

Server S sends packet with bytes  80301 -> 81761
Client sends ACK 81761
Server S sends packet with bytes 81761 -> 83221
Server S sends packet with bytes 83221 -> 84681
Client sends ACK 81761, SLE=83221 SRE=84681

SLE and SRE is the left edge and right edge of the SACK.

I'm not sure why the client missed bytes 81761->83221 and they were clearly received. I guess there could be a CRC error but tcpdump only captured the first 64 bytes of each packet. I'll increase the capture size to confirm.

FYI, I got access to another box located ~230ms away and it shows the same results. I.e. can download quickly from server F, but slowly from server S.
bplantAuthor Commented:
Ok I've verified that the duplicate ACKs are being caused by random corrupted TCP payloads. The packets that I capture leaving the physical host (according to tcpdump) on the sending end however are correct so the corruption is happening along the way somewhere.

The big question now is why don't the packets sent by server F get randomly corrupted? Even when the virtual machines are hosted on the same physical host.

More strangeness - scping the file from the physical host currently hosting server S runs at ~45mbps while scping the file from the physical host currently hosting server F runs at ~1mbps. Yes it's backwards!

Both physical hosts are connected to the same switch and have the same default gateway.
Is the MTU the same on both servers?

On the receiving side are packets fragmented when coming from one and not fragmented when coming from the other?

O.K., just to make sure I got everything straight.

You have run "S" config on the "F" box and it was still slow?
bplantAuthor Commented:
All physical and virtual servers have a MTU of 1500. I saw no fragmented packets from either host on the receiving side. All the received packets had a 1460 byte TCP payload.

I don't understand your last question. I have 2 physical boxes that both use xen, we'll call them box1 and box2. The hardware, kernel and configuration is identical on these 2 boxes except for of course the IP address and hostname.

Server S, a virtual server (VPS), runs on box1 while server F, also a virtual server, runs on box2. Given this you'd expect sending a file from box1 (physical host) to be slow and from box2 (physical host) to be fast. But it's not, sending from box2 is slow and sending from box1 is fast.

Looking at the TCP payload in wireshark, it looks like a 4 bytes sequence is getting changed to the previous 4 bytes. For example, if the sender sent

aa bb cc dd ee ff gg hh ii jj kk ll mm nn oo pp

then the receiver might occasionally end up with

aa bb cc dd ee ff gg hh ee ff gg hh mm nn oo pp
To clarify my last question.

I wanted to verify that you copied the apache.conf file from virtual server "S" to the virtual server "F" and run Apache on server "F" with that configuration.  When did that it was still slow.  Is that correct?

Why I am asking is that it makes no sense to start the you have 2 Apache servers (S and F) servers that both send data from the same site and that the data is leaving the site without any problems, but data from one of the two servers arrives corrupted.

It make even less sense that when you run Apache "F" using the configuration file from "S"  that all of a sudden the data coming from server "F" also get corrupted.  Especially when it is the same exact data.

Now dealing with the:

aa bb cc dd ee ff gg hh ii jj kk ll mm nn oo pp

then the receiver might occasionally end up with

aa bb cc dd ee ff gg hh ee ff gg hh mm nn oo pp

This would be valid if the fisrt time the receiver received ee it was corrupted and dd was the last packet that the received sent a ACK for.   If ee was corrupted, then the receiver should have sent a second ACK for dd, which tells the sender, dd is the last packet I received cleanly.  The sender will then start re-sending from ee.
bplantAuthor Commented:
I never tried copied the apache config for server S onto server F. Based on the comparison between packets leaving the virtual hosts and packets arriving at the client, it would seem to be independent of the application being used to send the data. This is reinforced by the fact that the issue also affects FTP and SSH/SCP.

With the "aa bb ... pp", these are individual nibbles shown in hexadecimal. I.e. the "aa bb .... pp" was a 16 bytes extract from one of the corrupted TCP packets.
bplantAuthor Commented:
Based on the data corruption I described, if all the bytes in the TCP payload were the same, all zeros for example, then the transfer should operate at full speed. I created a file containing all zeros from /dev/zero and sure enough, downloading this file is at least 10 times faster, usually more.
Some where I thought you had run "S" conf file on the same server that "F" normally ran prior to thinking it was a issue with the physical box.

If you do a trace route from server "S" and server "F" do they both take the same path?

Since you have noticed that data from "S" gets corrupted and FTP also preforms poorly I'm thinking that something is treating the traffic  from "S" IP address differently from "F".  The only way this should occur is if something is doing source IP routing.

Is there a way you can do a packet capture at the last point in the building where "S" is located and at the 1st point in the building where the client is installed?
bplantAuthor Commented:
Despite all the servers having the same default gateway, there seems to be some sort of source based routing going on when the packets cross the pacific ocean. Servers that are able to send data quickly all go via 1 route while the servers that transmit much more slowly go via another route.

I'll follow it up with the data centre and let you know.
Now, that is the 1st thing that has made since.  

One path is "clean" and the other path is not so clean?
bplantAuthor Commented:
It is the first thing that has made sense. The data centre took a bit of convincing that something was wrong, they're looking into it now. I will post back when I have something.
bplantAuthor Commented:
Thanks for your help debugging this one giltjr.

Featured Post

Hire Technology Freelancers with Gigs

Work with freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely, and get projects done right.

  • 16
  • 14
Tackle projects and never again get stuck behind a technical roadblock.
Join Now