Solved

WAN Optimization techniques

Posted on 2010-08-15
19
1,085 Views
Last Modified: 2012-05-10
Hello,
Our network consists of a 300Mbps Fiber Uplink connected to a Cisco Catalyst 4560 (Our Gateway to the provider) and about 150 servers on the network.
Our network utilization usually sits around 150Mbps to 220Mbps (Peak hours)
The issue we are experiencing is extremely poor file download speeds (Download Link or FTP)
At most these days what I'm able to get in 1 file download is about 600KBps (I'm testing with a 25Mbps connection I have at home and a few 100Mbps connections we have in branch offices. At most we'll get is about 1MBps.
I spoke with the ISP about this and we did an Iperf test and by spawning several connection we are able to aggregate 60+ Mbps but what I'm looking for is better performance for that 1 file download (HTTP) and so forth.
I'm just wondering if there is anything I can do with my cisco catalyst to improve performance (QOS) and any other options I may have.
Any help would be greatly appreciated (Please be as specific as possible) I'm a little lost right now.
I looked over WAN optimization appliances but it seems these need to be setup on 2 end-points over a wan in order to work (not going to work for us).

Hope you can help,
0
Comment
Question by:ntmyflt
  • 9
  • 9
19 Comments
 
LVL 24

Expert Comment

by:rfc1180
Comment Utility
These 2 end-points, are they separated by the Internet or point to point dedicated connections (Are the transfers via VPN?, anything else you can tell us about the connection[s])?
How large are the files that you are transferring?
What is the end to end latency (Forward path and return path)?


Billy
0
 

Author Comment

by:ntmyflt
Comment Utility
Hello,
Yes the endpoints are separated by the internet. there is NO vpn connection.

1 Server (Connected Directly to Internet) --- (Internet) -- Other Server

ICMP Delay from Datacenter to Home PC = 32ms (average)
ICMP Delay from Home to Datacenter = 32ms (average)

Traceroute is 10 Hops with the maximum hop being 37ms. From PC to Datacenter
Traceroute is 12 Hops with the maximum hop being 47ms From DC to PC


Hope this helps,



0
 

Author Comment

by:ntmyflt
Comment Utility
I do have to mention that this isn't an issue isolated to my home connection.
It doesn't matter where I test this from or which ISP I try this from, the results are always the same. (I've tested this from several different locations and ISPs)
This seems to be an issue on the DataCenter side.

0
 
LVL 24

Expert Comment

by:rfc1180
Comment Utility
The iperf test that you performed was between a location that has issues with the file transfer as well?
What is your commit for your bandwidth at the data center?
I understand FTP file transfers, but I do not know what you mean by "Download Link", URL link by HTTP?
The latency seems not to be an issue; what about any packet loss statistics on the network at the data center, packet loss will dramatically decrease performance on TCP applications.
>This seems to be an issue on the DataCenter side.
I agree; it appears that your bandwidth utilization (220Mbps [peak]) is pretty good data, again, what is the commit and/or burstable?

Billy
0
 

Author Comment

by:ntmyflt
Comment Utility
Hi Billy,
Yes when I mention HTTP I mean a file sitting on my Windows IIS server and downloading it from there.

I reviewed the Catalyst 4560 logs already yesterday and absolutely no packet errors at all.

How can I figure out the burstable and commit??

0
 
LVL 24

Expert Comment

by:rfc1180
Comment Utility
>How can I figure out the burstable and commit??
You will need to contact the NOC/Support Engineers or possibly your account manager.
You might want to access if they have a rate-limit applied to your upstream interface.
Also, is this only one server on your network that is experiencing the issue, or is it others?
Do you have any that result in good bandwidth (Also take note that when using iperf), it is using
raw memory, so anything else related to hardware/software could be the bottleneck [I/O, TCP Stack, etc])
What is the operating systems in your network that are having the issue?

Billy
0
 

Author Comment

by:ntmyflt
Comment Utility
Its a gigabit circuit that is rate limited at 300mbps up and down the ispsside.
i'm currently testing from an iis box but that box used to be a centos apache server but still the same issues :( just really crappy speeds in a single threaded connection.
0
 

Author Comment

by:ntmyflt
Comment Utility
what should a person be expecting as a download speed when there is about 100Mbps available on the network and it's just a single file download to a circuit that is also 100Mbps and the ping reply is roughly 35ms.. I would think about at least 70Mbps not 500 or 600K going maybe up to 1MB.

0
 
LVL 24

Expert Comment

by:rfc1180
Comment Utility
well, just to clarify, is it 600KB-1MB? (This will be between 4Mbps to 8Mbps [roughly];
ideally, in a perfect world, 1ms of latency between both ends (end to end bandwidth is 100Mbps, no bandwidth utilization), assuming 1500 byte packets, both end points with an RWIN value of 12500 Bytes, you should expect to see 95-98 Mbps (There is overhead to account for).
with 35ms of latency, the RWIN value should be 400KB (Assuming the operating system has RWIN auto-tuning enabled [This is key]. So as you can see, depending on many variables, you can expect different results. If you want a little more interesting read, give this a shot: http://www.experts-exchange.com/Networking/Network_Management/Network_Analysis/A_3094-Understanding-Network-and-Internet-Latency.html

Billy
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 24

Expert Comment

by:rfc1180
Comment Utility
>Its a gigabit circuit that is rate limited at 300mbps
I am assuming that the connection between you and the provider is 1000 Full duplex (Both ends have to match in duplex or you can expect performance issues that are related to duplex mismatches (typically, the connection you be hard-set between you and the data center or set to auto. Ensure that you and the data center are matched on both ends.

Billy
0
 

Author Comment

by:ntmyflt
Comment Utility
Hi Billy,
Yes 1000Mbps FDX.
I'm assuming that if the duplexing was incorrect I would be seeing some errors on the interface (but nothing at all).
0
 
LVL 24

Expert Comment

by:rfc1180
Comment Utility
>I'm assuming that if the duplexing was incorrect I would be seeing some errors on
Not necessarily, if you are hard-set and the ISP is auto, they will negotiate at 1000 Half (and they will see the errors, not you), but I believe you stated that this was a fiber uplink to the data center (using GBIC or SFP at 1GIG, the specs are auto anyway, I do not believe there is an option for manual selection). I am assuming this is a single connection to the ISP and you have no redundancy built into you network? What is interesting is that iperf reveals no issues (60+Mbps), this tells me that end to end, the network is solid and is capable of at least 60Mbps+, so I would focus on the local TCP stack and/or application of the servers you are having issues with. I see this on a daily basis and iperf is the tool we always use to rule on network/application issues.
0
 

Author Comment

by:ntmyflt
Comment Utility
So for example on a Windows 2003 Server how can I go about optimizing the TCP stack?
0
 

Author Comment

by:ntmyflt
Comment Utility
Could there be any issues with the Cisco Catalyst 4560, for example is there any way I can upgrade it in a way to prioritize TCP traffic, or is that a bad idea?
0
 
LVL 24

Expert Comment

by:rfc1180
Comment Utility
>prioritize TCP traffic, or is that a bad idea?
yeah, bad idea. At this point, it is very possibly that at times you might be hitting the rate-limit. The ISP should be able to tell by looking at the QoS policies they have set. If this is the case, you just need to upgrade the burst.

>So for example on a Windows 2003 Server how can I go about optimizing the TCP stack?
http://technet.microsoft.com/en-us/library/bb878127.aspx

http://technet.microsoft.com/en-us/magazine/2007.01.cableguy.aspx

I have also seen auto-tuning cause really bad performance with auto-tuning enabled:
http://www.mydigitallife.info/2007/03/22/disable-tcp-auto-tuning-to-solve-slow-network-cannot-load-web-page-or-download-email-problems-in-vista/

However, auto-tuning is just supposed to work, if you are having issues with the TCP stack, I would open a ticket with MS (I do not pretend to be a SysAdmin, I have executed some teaks, but I leave that up to MS engineers[By the way, Linux works beautifully with Auto-tuning!).

Billy
0
 

Accepted Solution

by:
ntmyflt earned 0 total points
Comment Utility
Hi Billy,
I was able to get the download speeds from my windows 2003 server to increase dramatically after running the following registry changes:
http://www.speedguide.net/read_articles.php?id=2665

Before the tune-up I was getting between 500 - 600K download speeds after having applied these changes on the server (which I'm downloading from) I get a consistent 1.1MB  to 1.2MB download speed.

Thanks again for your help with this.

0
 
LVL 24

Expert Comment

by:rfc1180
Comment Utility
you bet!
0
 
LVL 24

Expert Comment

by:rfc1180
Comment Utility
ntmyflt, any reason you did not award my any points? You did modify the TCP stack which was one of my solutions in comment

ID: 33442565

SO I believe I guided you in some direction towards the OS and TCP Stack which I believe deserves some points.

Please advise

Thanks
0

Featured Post

6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

Join & Write a Comment

Managing 24/7 IT Operations is a hands-on job and indeed a difficult one. Over the years I have found some simple tips and techniques to increase the efficiency of the overall operations. The core concept has always been on continuous improvement; a…
Creating an OSPF network that automatically (dynamically) reroutes network traffic over other connections to prevent network downtime.
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now