RGNCM Expert
asked on
How to overcome VPN slowness due to Verizon/Cogent bottleneck
I have a VPN running between 2 sites.
Local site is Cogent (100M/100M), remote is Verizon FiOS (75M/35M). Both sites have SonicWall NSA class firewalls capable of >100M VPN throughput.
Fios and Verizon have a little peering problem that started a few months ago, and the round trip ping time varies from 40Ms to 90Ms.
My VPN performance for standard windows file copying is inversely related to the ping times, due to turn around time of each small packet. (I use XXCopy for my copy utility between 2 Netware 6.5 servers through a Windows XP workstation)
However, iPerf tests between the two indicate that if I use a LARGE window size, and 3 parallel transfers, I can get expected transfer speeds.
Testing with a 64K window:
iperf -c remote.host -l 64K -P 3
--------------------------
Client connecting to remote.host TCP port 5001
TCP window size: 8.00 KByte (default)
--------------------------
[ ID] Interval Transfer Bandwidth
[1912] 0.0-10.1 sec 13.9 MBytes 11.6 Mbits/sec
[1896] 0.0-10.1 sec 13.9 MBytes 11.6 Mbits/sec
[1880] 0.0-10.1 sec 13.9 MBytes 11.5 Mbits/sec
[SUM] 0.0-10.1 sec 41.8 MBytes 34.7 Mbits/sec
~35 Mb/S = very nice speed (11Mb/Sec is a great speed for a single thread)
Testing with a 8K window:
iperf -c remote.host -l 8K -P 3
--------------------------
Client connecting to 192.168.165.43, TCP port 5001
TCP window size: 8.00 KByte (default)
--------------------------
[ ID] Interval Transfer Bandwidth
[1880] 0.0-10.1 sec 3.13 MBytes 2.61 Mbits/sec
[1912] 0.0-10.1 sec 3.13 MBytes 2.60 Mbits/sec
[1896] 0.0-10.1 sec 3.13 MBytes 2.61 Mbits/sec
[SUM] 0.0-10.1 sec 9.39 MBytes 7.82 Mbits/sec
~8 Mb/S = Crappy speeds (this is what i see using XXCOPY)
I have been running speed tests every 15 minutes and found that one late night when the ping times droped to 5Ms, I got 35 Mb/S on the 8K window size! So it is possible to get good performance with a normal copy, just not when the lines are 'long'.
I know that I can do nothing to improve the peering problem with Verizon and Cogent (every time I talk to one side, they blame the other)
Now my actual QUESTION:
Is there any way that I can force windows/xxcopy to make use of a larger TCP window (Like the iperf -l 64K parameter)?
Or is there some other way that I can negate the effect the r/t ping time has on my throughput?
Thanks
Ken (RGNCMExpert)
Local site is Cogent (100M/100M), remote is Verizon FiOS (75M/35M). Both sites have SonicWall NSA class firewalls capable of >100M VPN throughput.
Fios and Verizon have a little peering problem that started a few months ago, and the round trip ping time varies from 40Ms to 90Ms.
My VPN performance for standard windows file copying is inversely related to the ping times, due to turn around time of each small packet. (I use XXCopy for my copy utility between 2 Netware 6.5 servers through a Windows XP workstation)
However, iPerf tests between the two indicate that if I use a LARGE window size, and 3 parallel transfers, I can get expected transfer speeds.
Testing with a 64K window:
iperf -c remote.host -l 64K -P 3
--------------------------
Client connecting to remote.host TCP port 5001
TCP window size: 8.00 KByte (default)
--------------------------
[ ID] Interval Transfer Bandwidth
[1912] 0.0-10.1 sec 13.9 MBytes 11.6 Mbits/sec
[1896] 0.0-10.1 sec 13.9 MBytes 11.6 Mbits/sec
[1880] 0.0-10.1 sec 13.9 MBytes 11.5 Mbits/sec
[SUM] 0.0-10.1 sec 41.8 MBytes 34.7 Mbits/sec
~35 Mb/S = very nice speed (11Mb/Sec is a great speed for a single thread)
Testing with a 8K window:
iperf -c remote.host -l 8K -P 3
--------------------------
Client connecting to 192.168.165.43, TCP port 5001
TCP window size: 8.00 KByte (default)
--------------------------
[ ID] Interval Transfer Bandwidth
[1880] 0.0-10.1 sec 3.13 MBytes 2.61 Mbits/sec
[1912] 0.0-10.1 sec 3.13 MBytes 2.60 Mbits/sec
[1896] 0.0-10.1 sec 3.13 MBytes 2.61 Mbits/sec
[SUM] 0.0-10.1 sec 9.39 MBytes 7.82 Mbits/sec
~8 Mb/S = Crappy speeds (this is what i see using XXCOPY)
I have been running speed tests every 15 minutes and found that one late night when the ping times droped to 5Ms, I got 35 Mb/S on the 8K window size! So it is possible to get good performance with a normal copy, just not when the lines are 'long'.
I know that I can do nothing to improve the peering problem with Verizon and Cogent (every time I talk to one side, they blame the other)
Now my actual QUESTION:
Is there any way that I can force windows/xxcopy to make use of a larger TCP window (Like the iperf -l 64K parameter)?
Or is there some other way that I can negate the effect the r/t ping time has on my throughput?
Thanks
Ken (RGNCMExpert)
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
You need to be off Windows XP in about 6 months. NetWare 6.5 is already end of life. It looks to me that you need to be making some architectural changes anyway.
ASKER
It's become clearer now that the problem is related to the Netware Client.
NW seems to be taking a huge hit because of the long round-trip time on the WAN.
I have seen suggestions to increase the "ncp tcp receive window" to 65535.
But that did not have any effect.
Does anyone have any idea on how to make the NCP protocol work more efficiently over a slow VPN?
K
NW seems to be taking a huge hit because of the long round-trip time on the WAN.
I have seen suggestions to increase the "ncp tcp receive window" to 65535.
But that did not have any effect.
Does anyone have any idea on how to make the NCP protocol work more efficiently over a slow VPN?
K
Netware was designed long ago, when WANs didn't exist. It is inherently slow on a high latency WAN and this can not be "fixed".
WAN optimizers can bring some relief, but these are *very* expensive.
As stated before, a protocol change is your best bet. You can do this using two intermediate workstations at both ends, limiting netware to the LAN at each side.
WAN optimizers can bring some relief, but these are *very* expensive.
As stated before, a protocol change is your best bet. You can do this using two intermediate workstations at both ends, limiting netware to the LAN at each side.
ASKER
Thank you all for your input!
We are working on moving our remote backup servers off NW and onto Linux, which works better over the VPN.
We are working on moving our remote backup servers off NW and onto Linux, which works better over the VPN.
It could be Cogent that is the issue. I had 3 separate issues on my Cogent 100x100. Ipef tests come back fine but real world download speeds were not what i expected.
run a tracert and see if you get any slow times on the Cogent side. that was the problem number 1. They had to change some equipment at the Dmarc to correct the issue. The onsite tech also noticed i had high light levels on my fiber line. 3rd issue Level 2 tech support noticed i had a lot of dropped packets.
run a tracert and see if you get any slow times on the Cogent side. that was the problem number 1. They had to change some equipment at the Dmarc to correct the issue. The onsite tech also noticed i had high light levels on my fiber line. 3rd issue Level 2 tech support noticed i had a lot of dropped packets.
ASKER
Going to Win7 on the computers that initiate the copy has been tried, and there is only a slight improvement in speed. I believe this is because we are still using the NetWare client.
You may be on the right track though, because when I transfer files from Win7 to Ubuntu, I do see a full 40Mb/s transfer speed, but only 10Mb/s with XP to Ubuntu.
I will explore our options on re-configuring the Netware IP client. But I am not optimistic...
I will also look into Silverpeak.
I will update this issue shortly.
Ken