Solved

Extremely Slow Network Download

Posted on 2010-08-24
14
855 Views
Last Modified: 2012-05-10
Hi,

I have a networking related issue that I cannot figure out.  We have two Datacenters and two offices all connected via Cisco 2800 Routers / VPN devices.  It is a multipoint VPN to either datacenter, with internet traffic for the offices routed direct out and not through the VPN.  

In our NYC location, users are experiencing issues downloading from certain websites.  Notable ones include download.google.com, or any other CDN type download.  It seems like the connection starts and downloads the first 300 KB, then grinds to a whopping 56 bytes per second.  I spoke to the ISP (Cogent) and they confirmed nothing is wrong with the line (100 MB fiber line).  

Could there be something misconfigured on this end with the routers?  I cannot figure out where to start.

Any help much appreciated.  This is starting to become a real issue for executives downloading PDF's off Financial Websites.
0
Comment
Question by:krutzer
  • 7
  • 6
14 Comments
 
LVL 6

Expert Comment

by:jkratzer
ID: 33516191
Do you have any web content filtering systems in place?

If so you may have certain file types/site types that are set to restrict download speeds.
0
 

Author Comment

by:krutzer
ID: 33516211
Nope.  There are no restrictions on outbound Internet
0
 
LVL 24

Expert Comment

by:rfc1180
ID: 33516675
>In our NYC location, users are experiencing issues downloading from certain websites
any recent changes?
Is this location new? If not, when did it start happening?

>Could there be something misconfigured on this end with the routers?  I cannot figure out where to start.
Sound like a duplex mismatch somewhere or can even be related to packet loss. Packet loss and TCP do not play well with each other.

What you want to look at is there packet loss? Run a 60 second ping test to your destination:

Linux: ping -c 60 google.com

Windows: ping -n 60 google.com

You also want to check the latency end to end; some operating systems are not configure to work over long high latency paths, so you will need to run a traceroute to and from the source and destination (Getting the return path can be a little tricky), one path will not help if the latency is high, if the latency is relatively acceptable (every 125 miles is 1 ms [one-way]) then do not worry about getting a return path for the traceroute. You will also want to test your bandwidth to a few NDT servers that are hosted near your location (Do not use any other speed test sites, they are all shared and unreliable!)

New Note 84

New York:
http://web100.rit.edu:7123/

California:
http://nitro.ucsc.edu/

Denver:
http://www.medctrrockies.org/

Texas:
http://speedtest.tx.charter.com/

Washingtion:
http://ndt.iupui.sea01.measurement-lab.org:7123/

Florida:
http://ndt.server.ufl.edu:7123/

Argonne - IL
http://ndt.anl.gov:7123/


Please post the output after you test

Billy
0
 

Author Comment

by:krutzer
ID: 33517243
Hi Billy,

I can't do a copy and paste of all the data using this console at the moment, but here are the results.

1. download.google.com is the only site to have packet loss of 40% (used mtrace), but this is the same from the sites that have no issue as well, including my house.

2. Using an mtrace to the other sites show latency to be normal and no packet loss, including sites that have issues.  

3. Here are the results from the NY Link you sent me...

TCP/Web100 Network Diagnostic Tool v5.5.4b
click START to begin

** Starting test 1 of 1 **
Connected to: web100.rit.edu  --  Using IPv4 address
Checking for Middleboxes . . . . . . . . . . . . . . . . . .  Done
checking for firewalls . . . . . . . . . . . . . . . . . . .  Done
running 10s outbound test (client-to-server [C2S]) . . . . . 17.0kb/s
running 10s inbound test (server-to-client [S2C]) . . . . . . 88.03kb/s
The slowest link in the end-to-end path is a a 622 Mbps OC-12 subnet
 [S2C]: Packet queuing detected

click START to re-test
0
 
LVL 24

Expert Comment

by:rfc1180
ID: 33517302
yeah, I am not familiar with mtrace; it is probably related to mtrace

>running 10s outbound test (client-to-server [C2S]) . . . . . 17.0kb/s
>running 10s inbound test (server-to-client [S2C]) . . . . . . 88.03kb/s

That is pretty bad for you being on a 100Mbps link.

I would rule out any duplex issues between the host and your edge device connecting to the data center. Those results are bad. I would also rule out any bandwidth saturation issues too. Do you have a network diagram that you can post, what is the make and models of the network devices in your network?
You say you link to the data center is 100Mbps, is this connection rate-limited, or do you have the full 100Mbps bandwidth (CIR)?

Billy
0
 

Author Comment

by:krutzer
ID: 33517862
Hi Billy,

Mtrace is a very useful linux tool for diagnosing network issues.  "mtrace combines the functionality of the 'traceroute' and 'ping' programs in a single network diagnostic tool."
http://www.bitwizard.nl/mtr/

Yeah, the test is very bad.  What's weird is that I do get a full 100 MB/s file transfers over the VPN with the same link.  The issue only seems to occur with routes not using the VPN.  I've attached a network diagram but cut out most of the external IP's and Order - ID's.  All the routers are Cisco 2800 Series, can't remember which ones.  The rest of the Switches are all Cisco Catalyst 3750Gs, aside from the NY one's which are 3550 series.  

In regards to the bandwidth, we are not rate limited.  We also have a full block of IP's we bought from ARIN and are running BGP at the datacenters.

Thanks for the continued help.
network.jpg
0
 
LVL 24

Expert Comment

by:rfc1180
ID: 33522773
>Yeah, the test is very bad.  What's weird is that I do get a full 100 MB/s file transfers over the VPN with the same link.

That is a little interesting.

om-nyc-ms-1 and 2, there are 2 physical connections (.158 and .94); are both links active to the Internet or is the .94 used as a backup link? If so, might be a good idea to failover to the backup link.

What about checking all switchport and layer 3 interfaces upstream for port/interface errors?

Is the New York location new?
Did this start all of a sudden, was anything changed on the network?


Billy
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 

Author Comment

by:krutzer
ID: 33533792
>om-nyc-ms-1 and 2, there are 2 physical connections (.158 and .94); are both links active to the Internet or is the .94 used as a backup link? If so, might be a good idea to failover to the backup link.

There aren't two physical links, the extra links denote the VPN path.  I can't find any port errors or anything.  Nothing was changed on the network.  It started out of nowhere about 1 month ago and has gotten worse.  

It's really odd because it goes at full speed when going to the datacenter.  
0
 
LVL 24

Accepted Solution

by:
rfc1180 earned 500 total points
ID: 33536767
>It's really odd because it goes at full speed when going to the datacenter.  
Yup, you got yourself a fun one!

I still have a concern with:
1. download.google.com is the only site to have packet loss of 40% (used mtrace), but this is the same from the sites that have no issue as well, including my house.

using mtr, pathping, tracepath to detect packet loss is not the best tool to use, let me explain why; Internet routers in the path from point A to point B will/can typically drop ICMP packets due to how routers handle packets based on priority and ICMP is not in that class, and most vendors such as Cisco and Juniper rate-limit ICMP packets by default. You need a tool that can send an ICMP packet end to end without an intermediary router processing the packet (Data Plane vs the control plane; any ICMP traffic going to the router will typically be processed by the CPU [Control Plane], and any ICMP traffic going through a router will be sent through the data plane).

Typically I like using plain old ping (One that supports the flood option)

evilbit@neteng01 ~ $ sudo ping -f downloads.google.com
[sudo] password for evil:
PING www2.l.google.com (74.125.47.103) 56(84) bytes of data.
....^C
--- www2.l.google.com ping statistics ---
794 packets transmitted, 794 received, 0% packet loss, time 10995ms
rtt min/avg/max/mdev = 38.394/46.739/61.121/7.928 ms, pipe 5, ipg/ewma 13.866/52.927 ms

And if ICMP is an issue on your network or something filtering in the path or at the far end:
you will need hping.

hping3 -S downloads.google.com -p 80 -c 100 --fast


At this point, I would really recommend that you SPAN a port on your switches for one of the vlans you are having this issue with. This will ultimately tell you what the issue is or give you and idea where you need to look

Good Luck
Billy
0
 

Author Comment

by:krutzer
ID: 33545314
Thanks again Billy,

What do you mean by "SPAN" a port?

I will continue troubleshooting next week.
0
 
LVL 24

Expert Comment

by:rfc1180
ID: 33545340
0
 

Author Comment

by:krutzer
ID: 33545383
Thanks,

If I can't figure this out with the ISP I'd be willing to contract you for your hourly rate.  Are you available next week or the week after.  I would be able to provide remote access.
0
 
LVL 24

Expert Comment

by:rfc1180
ID: 33545541
Sure, there should be a hire button next to my name; I am not sure how it all works as I have never been hired via EE.

Billy
0
 

Author Comment

by:krutzer
ID: 33854724
We finally got to the bottom of this issue.  It ended up being that Router EIGRP was misconfigured.

Someone took out the line that set it to not "auto-sum" the routes, therein splitting the traffic.
0

Featured Post

PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

Join & Write a Comment

Let’s list some of the technologies that enable smooth teleworking. 
If you're not part of the solution, you're part of the problem.   Tips on how to secure IoT devices, even the dumbest ones, so they can't be used as part of a DDoS botnet.  Use PRTG Network Monitor as one of the building blocks, to detect unusual…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now