Solved

Ethernet Private Line

Posted on 2011-03-01
12
1,107 Views
Last Modified: 2012-05-11
We recently had a 100Mbps Ethernet Private Line installed between our two offices. We are seeing speeds using iPerf of 100Mbps with UDP. However, We are only getting max speeds of 30Mbps with TCP. Does the windows packets need to be adjusted from the default size of 64Kbps, and if so, how is that done.
0
Comment
Question by:wadehood
  • 5
  • 2
  • 2
  • +2
12 Comments
 
LVL 17

Accepted Solution

by:
Kvistofta earned 500 total points
ID: 35014838
What do you mean with "default size of 64Kbps"? The size of a packet is defined in bytes, not bytes per second (bps). The normal maximum packet size for ethernet ie 1500 bytes and that can be raised if you know that all devices in transit can handle larger packets, which is not always the case.

What you suffer is probably not related to packet sizes but rather the tcp windows size in the end host. That value controls how much tcp data to send before getting an acknowledgements. If you have a fast network with relative high response time you have what is commonly named LFN or Long Fat network. What needs to be done then is to raise the tcp windows size in the computers in both ends in order to send larger chunks of data before requiring an acknowledge of that data chunk.

I have personal experience of a 500Mbps ethernet-type wan-connection where the backup system could only utilize 30Mbps out of the 500Mbps available. When changing a registry setting in the two backup-devices (running Windows 2003s) we instantly made it possible to pass 480Mbps of data thru that link.

More info about LFN:

http://en.wikipedia.org/wiki/Bandwidth-delay_product

Best regards
Kvistofta
0
 
LVL 34

Expert Comment

by:Istvan Kalmar
ID: 35015064
Hi,

You need to test with MTU 1500, because it is the 'default size of ETH packet'
You never reach 100M with 64 bytes packets on this line:

http://www.faqs.org/rfcs/rfc2544.html
0
 
LVL 17

Expert Comment

by:Kvistofta
ID: 35015850
ikalmar: MTU is an ethernet-setting. With UDP he gets 100Mbps, the issue is with TCP.

/Kvistofta
0
 
LVL 34

Expert Comment

by:Istvan Kalmar
ID: 35015912
TCP transmission depends on MTU and delay... So we need more info about the problem

0
 
LVL 57

Expert Comment

by:giltjr
ID: 35017470
What OS(s) are you using?

Do they have current fixes on them?

Most current OS's do not have a fixed TCP Window size of 64KB any more, they now use sliding windows and thus they adjust on the fly based on various things they see during the transfer.

What is the latency? ping a host on the other side and see what the RTT is.
0
 

Author Comment

by:wadehood
ID: 35018141
Thanks for all your responses. I stand corrected...I should have said default size of 64KB. We will play with the TCP WIndows sizes. In addition, I understand that at the AT&T side they are moving our lines from an OC 3 to an OC 12. This may help as well. Will keep all posted.
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 17

Expert Comment

by:Kvistofta
ID: 35018298
wadehood: You are right, the 64KB is the tcp windows size I mention above, and that is the value you need to raise in order to utilize full bandwidth. Please keep us updated on the progress.

ikalmar: Yes but the MTU and delay affects the UDP-thruput aswell and since the author gets full thruput with UDP the issue is related to TCP itself, not any underlaying protocol. Please read the posted links about LFN:s ("elephants" :-) )

giltjr: You are right about the sliding window for TCP. What I write is the maximum value for the sliding window that needs to be raised.

Best regards
Kvistofta
0
 
LVL 32

Expert Comment

by:harbor235
ID: 35018395


Most modern IP stacks support auto scaling of the TCP window, unless you have hard coded a partiuclar size window this is negoitated to the max window availble. I agree with ikalmar, throughput is a function of several things, mtu, delay, window size, link reliability, congestion, PMTUD, etc .......

I doubt your problem is the window size alone, there are additional factors.

harbor235 ;}
0
 
LVL 17

Expert Comment

by:Kvistofta
ID: 35018433
Windows server 2003 has sliding window support, up to a maximum of 64KB. I got 6% thruput in a 500Mbps-connection before raising the tcp maximum window size.

Go figure.

/Kvistofta
0
 
LVL 17

Expert Comment

by:Kvistofta
ID: 35018555
And just to clarify: Of course the delay is an important factor. The Maximum window size is depending upon the Bandwith-delay-product which obviously is based on available bandwidth and delay (RTT).

/Kvistofta
0
 
LVL 57

Expert Comment

by:giltjr
ID: 35018715
If you can get near 100 Mbps with UDP then moving from OC3 to OC12 will not change a thing.

UDP just throws data over it shoulder and goes.  It has no pacing or ack's or guarteened delivery.  If it gets there it gets there, if it does not the application needs to recover.

TCP has pacing (window sizes) and ACKs to prevent flooding the nettwork with data and has recovery if a packet is dropped.

Kvistofta, I don't think you are using sliding window.  Based on what you are saying I think you are still using a fixed window size.

Sidling window sizes allows for windows larger than 65K.  The 65K is the limit for fixed window sizes.

Windows 2003 does have sliding window support, but it does NOT use it by default. So by default it will use a FIXED Window size and that size is based on a few other things.  If you want 2003 to use sliding windows you need to set the Tcp1323Opts registry value to 1.

When using sliding windows, there is another field that is used as a multiplier.  The multiplier is used in conjuction with the TcpWindowSize to calulate the true window size.
0
 
LVL 32

Expert Comment

by:harbor235
ID: 35018789

Sliding window and auto scaling are not the same, In addition, other things to consider are your udp/tcp send and receive buffers, they impact greatly perfromance.

Not to mention that MS (Vista for sure) IP stacks are known to have issues with TCp window scaling and network equipment

harbor235 ;}
0

Featured Post

Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

Join & Write a Comment

Creating an OSPF network that automatically (dynamically) reroutes network traffic over other connections to prevent network downtime.
Configuring network clients can be a chore, especially if there are a large number of them or a lot of itinerant users.  DHCP dynamically manages this process, much to the relief of users and administrators alike!
This video discusses moving either the default database or any database to a new volume.
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

706 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now