• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 4561
  • Last Modified:

Windows file transfers over high latency, high bandwidth link

I'm trying to transfer large files (100+ gigabyte db backups in this case) between our Windows servers in the US and the UK.  I'm running into terrible transfer speeds regardless of the protocol I use, whether it be regular Windows copy (smb) or scp.  This is a very high bandwidth link with about 100ms of latency.  These are Windows 2008 servers in the US and 2008R2 servers in the UK.

My best consistent transfer speed so far has been about 512 kbytes/sec, which is terrible.  Using Linux-to-Linux transfers from servers in these networks (using scp), the transfer rates can get as high as 35 mbytes/sec.  Linux-to-Windows transfers suffer the same as Windows-to-Windows transfers.

I'm starting to exhaust my ideas, so I'm desperately hoping you folks here might have some new ones for me.  I've tried:

Lowering the MTU as low as 1300 (to account for any possible vpn overhead, issues, etc).
netsh interface tcp set global autotuninglevel=disabled
netsh interface tcp set global rss=disabled
HKLM\system\CurrentControlSet\Services\lanmanworkstation\parameters\DisableBandwidthThrottling = 1
HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\EnableWsd = 0
xcopy /j
robocopy
TeraCopy (which is surprisingly consistently the fastest option)
scp in Cygwin
pscp (which is consistently faster than Cygwin's scp, but still slow)

Does anyone have any further recommendations?  We need to transfer hundreds of gigabytes of data between these locations, and right now we have to use a roundabout method:

Copy from US Windows server to US Linux server in the same  network.
Copy from US Linux server to UK Linux server.
Copy from UK Linux server to UK Windows server in the same network.

Surely there is some sort of tuning in Windows 2008 that causes these issues over high latency connections.
0
ZMEadmin
Asked:
ZMEadmin
  • 6
  • 4
1 Solution
 
Networking_EnthusiastCommented:
Hello,

  Have you tried disabling SMB 2.0 support on your 2008 servers? Also, are you using Symantec anit-virus? In the past Symantec has needed to be disabled for file transfers to speed up across links..

Further reading on the issue; read the third and fourth paragraph in the poster's question:

http://www.symantec.com/connect/forums/cannot-copy-files-using-smb-20-protocol-sep-11-ru6-and-ru6a

0
 
ZMEadminAuthor Commented:
We actually are not running anti-virus on these servers at this time, nor have I tried disabling SMB 2.0.  I can try that, but this problem occurs even when using SCP, which doesn't use smb at all.  My gut is telling me there is something deeper in the Windows tcp/ip stack causing this issue.
0
 
kevinhsiehCommented:
Have you tried TCP Optimizer? http://www.speedguide.net/downloads.php

I would think that SMB 2.0 and and the autotuning would both help. By default, TCP can't fill a fast high latency pipe. Adjustments need to be made in the TCP receive window. See
http://www.speedguide.net/articles/advanced-tweaking-156
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
ZMEadminAuthor Commented:
I've seen a few other references to TCP Optimizer as well, so I checked it out.  Nice utility, but it doesn't fix the issue.  Most of the settings it seems to tweak are things I've already tried manually.  I'll keep that utility around for an easier way of changing things in the future, though.

I'm also not confident in changing the SMB 2.0 to 1.0 change.  We're not even talking SMB at this point.  I went ahead and installed the ftp service on one of the servers, and I get the same poor transfer rates with that.  Usually around 500 kbytes/sec average.  So I've now tried smb, scp, and ftp between these servers, and I get roughly the same transfer rates with all of them.  Like I mentioned earlier, Linux servers in these same networks can get 20-30 megabytes/sec between them, but Windows can only get 500 kbytes/sec.
0
 
kevinhsiehCommented:
Some things to try, from http://www.speedguide.net/articles/windows-7-vista-2008-tweaks-2574

1. Set TCP-autotuning to experimental
2. Turn on Compound TCP


TCP Auto-Tuning
 
To turn off the default RWIN auto tuning behavior, (in elevated command prompt) type:
 

netsh int tcp set global autotuninglevel=disabled
 
The default auto-tuning level is "normal", and the possible settings for the above command are:
 

disabled: uses a fixed value for the tcp receive window. Limits it to 64KB (limited at 65535).
highlyrestricted: allows the receive window to grow beyond its default value, very conservatively
restricted: somewhat restricted growth of the tcp receive window beyond its default value
normal: default value, allows the receive window to grow to accommodate most conditions
experimental: allows the receive window to grow to accommodate extreme scenarios (not recommended, it can degrade performance in common scenarios, only intended for research purposes. It enables RWIN values of over 16 MB)
 
Our recommendation: normal  (unless you're experiencing problems).
 
If you're experiencing problems with your NAT router or SPI firewall, try the "restricted", "highlyrestricted", or even "disabled" state.
 
Notes:
- Reportedly, some older residential NAT routers with a SPI firewall may have problems with enabled tcp auto-tuning in it's "normal" state, resulting in slow speeds, packet loss, reduced network performance in general.
- auto-tuning also causes problems with really old routers that do not support TCP Windows scaling. See MSKB 935400
- netsh set commands take effect immediately after executing, there is no need to reboot.
- sometimes when using "normal" mode and long lasting connections (p2p software / torrents), tcp windows can get very large and consume too much resources, if you're experiencing problems try a more conservative (restricted) setting.
 
If you're experiencing problems with Auto-Tuning, see also:
MSKB 835400 - email issues
MSKB 934430 - network connectivity behind firewall problems
MSKB 940646 - 3G WWAN throughput issues
MSKB 929868 - web browsing issues
MSKB 932170 - slow network file transfer
 
 
 
Compound TCP - Improve throughput
Add-On Congestion Control Provider
 
The traditional slow-start and congestion avoidance algorithms in TCP help avoid network congestion by gradually increasing the TCP window at the beginning of transfers until the TCP Receive Window boundary is reached, or packet loss occurs. For broadband internet connections that combine high TCP Window with higher latency (high BDP), these algorithms do not increase the TCP windows fast enough to fully utilize the bandwidth of the connection.
 
Compound TCP (CTCP) is a newer method, available in Vista and Server 2008 (there is also a hotfix available for XP x64 and 2003 Server - MSKB 949316). CTCP increases the TCP send window more aggressively for broadband connections (with large RWIN and BDP). CTCP attempts to maximize throughput by monitoring delay variations and packet loss. It also ensures that its behavior does not impact other TCP connections negatively.
 
By default, Vista and Windows 7 have CTCP turned off, it is only on by default under Server 2008. Turning this option on can significantly increase throughput and packet loss recovery.
 
To enable CTCP, in elevated command prompt type:
 

netsh int tcp set global congestionprovider=ctcp
 
To disable CTCP:
 

netsh int tcp set global congestionprovider=none
 
Possible options are:  ctcp, none, default (restores the system default value).
Recommended setting: ctcp
 
It is better to use this newer generation CTCP congestion control algorithm for most broadband connections, we highly recommend it being turned on.
0
 
ZMEadminAuthor Commented:
No luck, either.  The TCP Optimizer tool actually enabled CTCP, but I hadn't tried the "experimental" auto tuning setting.  Unfortunately, that didn't do the trick.  Transfer rate averages are staying the same whether I choose disabled, restricted, normal, or experimental.  Experimental does seem to do *something*, though, as transfer rates appear jump around a bit more.  Regardless, the averaged end result is the same, either way.
0
 
kevinhsiehCommented:
Can you try with 2008 R2 or Windows 7 in the US? Just wondering if there are improvements in the Windows 7/2008 R2 code over the Vista/2008 code.

I assume that you have bandwidth to burn, or you would be looking at Riverbed or Silver Peak to minimize the amount of traffic you put on the WAN. They also do TCP optimization for long fat networks (LFN) like yours.
0
 
ZMEadminAuthor Commented:
Bandwidth is thankfully not a problem, so we definitely have it to burn in this case.  And I do indeed have access to some 2008R2 machines in the US environment, so I did try those.  Things actually do perform between in 2008R2-to-2008R2 transfers, but still slow.  I'm getting about 1.0 to 1.5 megabytes/sec there.  Quick, but nowhere near the 20-30 megabytes/sec we see from the Linux-to-Linux transfers.

As far as companies like Riverbed and Silver Peak, I think those offerings are a bit out of the scope of my particular need.  We have some very large database backups that we need to copy from the US to the UK.  These copies will only happen a few times, and then likely never again.  We have a "plan B" of doing the copies from Windows-to-Linux in the US datacenter, then Linux-to-Linux from the US to the UK, then Linux-to-Windows in the UK.  It works since we won't have to do it more than a few times, but it seems ridiculous we should have to resort to such a measure.
0
 
ZMEadminAuthor Commented:
I made a bit of a typo in my last post.  I meant to say things do indeed perform better in 2008R2-to-2008R2 transfers, but they are still slower than they should be.
0
 
kevinhsiehCommented:
I don't have an answer for you. :(
If you really want to get to the bottom of it, you can request attention to try to get help from additional experts, post to the Microsoft TechNet forums, or open up a support case with Microsoft. Otherwise, you may want to do the whole linux to linux transfer for when you need to do this.
0
 
ZMEadminAuthor Commented:
No problem, thanks for your suggestions.  In my digging through the web, I did indeed find many people with this exact problem all over the TechNet forums, but unfortunately none of them were able to figure it out.  Most of them ended up giving up and using other methods for moving their files.  (Linux, shipping tapes, etc.)
0

Featured Post

Prepare for your VMware VCP6-DCV exam.

Josh Coen and Jason Langer have prepared the latest edition of VCP study guide. Both authors have been working in the IT field for more than a decade, and both hold VMware certifications. This 163-page guide covers all 10 of the exam blueprint sections.

  • 6
  • 4
Tackle projects and never again get stuck behind a technical roadblock.
Join Now