I have 2 severs, server S (for slow) and server F (for fast). Both run Linux and apache. When downloading the same 50MB file containing random binary data from a 3rd host ~200ms (latency) away using wget, I get substantially different results. When downloading from server F, the transfer will run at 40-90mbps whereas when I download the same file from server S, the transfer will run at 1-3mbps.
Alternatively, I can download the file from my ADSL connection (20ms latency) and saturate the link. My ADSL connection is ~16mbps.
I can also download the file at gigabit speed from both hosts when on the local network.
Both servers were idle, with very little IO or network activity when the tests were performed.
Some information about the 2 servers:
- Both are on the same network
- Both servers are virtualised xen hosts
- Both run apache 2.2.11
- Both servers have the same setting for /proc/sys/net/ipv4/tcp_wmem (4096 16384 4194304)
- Both servers have the same setting for /proc/sys/net/core/wmem_max (131071)
- Both servers have syncookies on - changing this makes no difference
- Both servers have tcp timestamps off - changing this makes no difference
- Both servers have window scaling turned on (tcp_window_scaling=1). In addition to this, using tcpdump shows wscale=7 during the TCP options negotiation for connections to both servers
- Server F runs a 2.6.27 kernel and server S runs a 2.6.28 kernel. I have tried running server S on the same 2.6.27 kernel but it makes no difference.
- Server F has keepalive off, while server S has it on, but turning it off makes no difference
- Server S also handles SSL and PHP/FCGID, but disabling both of these makes no difference
- Server S has 4 cpus and 4GB of ram (Server F has 1 cpu and 256MB ram), but cutting server S down to 1 cpu and 256MB ram makes no difference
- Running both virtualised servers on the same physical host makes no difference
- Server S has higher apache MPM settings, but changing these to the same as that on server F makes no difference.
- When the file was being downloaded from server F, netstat shows a Send-Q of 2-3MB, while on server S it is only about 100-300KB.
In summary, the issue appears to only affect high latency connections (in this case it's trans-continental). The file is downloaded at 40-90mbps from server F while only at 1-3mbps from server S.
Ideas, thoughts, questions?