Network Bandwidth bogging down with any download/upload (Cisco)

I'm running into some issues where our network bandwidth is being completely consumed when any download or upload is happening on the office network.  I recently upgraded our bandwidth from 20Mbps down/up (I know, still pretty slow in today's world) from 10Mbps down/up and also replaced our Cisco 1761 & Cisco 2851 routers with two (2) Cisco 2911's to increase WAN/LAN bandwidth.

I just started noticing this recently, but if a download or upload is going on, the network peaks to 20Mbps usage (either down or up) and the entire network crawls until the download/upload is finished.  Yesterday, I was downloading a 25MB file at around 2Mbps, and the outbound network literally stopped until my small download completed.

In our ASA 5510, I can watch the bandwidth graph jump from almost no bandwidth usage to max as soon as the download starts and then drops to nothing when it ends.


Any thoughts on this?  The Cisco router configs are identical to what they were in the original 1761 and 2851.
sciggsAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Benjamin Van DitmarsCommented:
can you send me the config of youre 2911. and do a show interfaces and send me the results.
this looks like a small configuration issue.
0
dpcsitCommented:
Look to see if your Cisco is creating bad packets that are flooding your network with trash. I know with our Dell PCs some of them do not handshake well on their own and we have had to fix the Speed and duplex for them to clean up the traffic.

Secondly your connection to our ISP may also be producing bad packets.

Do you do any work inside your network only or is it all cloud based work? If not cloud based are the internal responses ok or are they slowing down also?

Without a download what does speed test tell you your bandwidth is?

My money says you have a handshaking issue between your new Cisco and the ISP or your switch. We also have had handshaking issues with out ISP and our 5505 ASA. Since it's only stable at 100/full the ISP had to reduce his speed from a gig to 100.
0
sciggsAuthor Commented:
I'm building my configurations and will attach those shortly.

dpcsit, that's interesting.  We are all Dell gear other than our Cisco routers and ASA.  We have PowerConnect 2724's that feed our offices and servers.

We do a lot of internal working.  We have around 50 servers for development and 30 users in our office.
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

Benjamin Van DitmarsCommented:
check youre interface settings. you need to set them at 100mbit full duplex
and check if youre shapper. for this reason i asked you youre config and interface statistisc :)
0
sciggsAuthor Commented:
Benjamin,

That may be the disconnect.  On my edge router that connects to the line from our ISP, one interface is set for Full Duplex 100mbps and the other interface that routes the traffic from the ISP to our firewall is set to Full Duplex 1Gbps.

On our other router which handles the internal traffic, 2 interfaces are setup for Full Duplex 1Gbps and the Interface for our VoIP network is Full Duplex 100Mbps.

Should I set all 5 interfaces to Full Duplex 100Mbps?
0
dpcsitCommented:
Well it's our HP switches that do not like some model dells, some I have to force to 100/full or gig/full.
0
giltjrCommented:
If possible I would look at doing packet captures on your new routers and on the ASA.

The speed (100 vs. 1000) will not really matter.  If two devices have different speeds they will not talk at all.  It's the duplex that matters.  If you have a duplex mismatch someplace then on side will think it can send and receive at the same time (full) and the other side will think it can only do one at a time (half).

When this occurs the half duplex side will start seeing collisions and throughput will drastically drop, but link utilization may actually increase.  This is due to all of the re-transmits.
0
Benjamin Van DitmarsCommented:
alot of NTU of youre internet connection wants on connections till 100 mbit a full duplex link.
we sell alot of fibre conenctions. and this is the first thing i ask the company when they have bandwitdh problems. the second problem is normaly the shaper on the outside interface.

when you dont have a shaper. the router is pushing at full speed. youre connection can normaly burst some. but then it want's back to the maximum vlan speed. and this is what the shaper is for.

and the result is exact what the topic starter is saying to have.
0
sciggsAuthor Commented:
Below is the running config and also the interfaces.  I took out any information and replaced first 3 IP octets with 1.1.1.x or 2.2.2.x.


ROUTER1#show running-config
Building configuration...

Current configuration : 3925 bytes
!
!
version 15.4
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
no service dhcp
!
hostname ROUTER1
!
boot-start-marker
boot-end-marker
!
!
logging buffered 51200 warnings
enable secret 5 $1$H9Mq$m7VSvzI0rqVpOIv43YYCg/
!
no aaa new-model
clock timezone EST -4 0
!
!
!
!
!
!
!
!
!
!
no ip domain lookup
ip name-server 208.67.222.222
ip name-server 8.8.8.8
ip cef
no ipv6 cef
multilink bundle-name authenticated
!
!
cts logging verbose
!
!
redundancy
!
!
interface Embedded-Service-Engine0/0
 no ip address
 shutdown
!
interface GigabitEthernet0/0
 description WAN
ip address 1.1.1.166 255.255.255.252
 duplex auto
 speed auto
 no keepalive
!
interface GigabitEthernet0/1
 description IP Block
 ip address 2.2.2.65 255.255.255.192
 duplex auto
 speed auto
 no keepalive
!
interface GigabitEthernet0/2
 no ip address
 shutdown
 duplex auto
 speed auto
!
ip default-gateway 1.1.1.165
ip forward-protocol nd
!
no ip http server
ip http access-class 23
ip http authentication local
no ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
!
ip route 0.0.0.0 0.0.0.0 1.1.1.165
!
!
!
access-list 50 permit 2.2.2.64 0.0.0.63
!
control-plane
!
!
!
line con 0
 login local
line aux 0
line 2
 no activation-character
 no exec
 transport preferred none
 transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
 stopbits 1
line vty 0 4
 access-class 23 in
 privilege level 15
 login local
 transport input telnet ssh
line vty 5 15
 access-class 23 in
privilege level 15
 login local
 transport input ssh
!
scheduler allocate 20000 1000
!
end

ROUTER1#show int
Embedded-Service-Engine0/0 is administratively down, line protocol is down 
  Hardware is Embedded Service Engine, address is 0000.0000.0000 (bia 0000.0000.0000)
  MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/64/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles 
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 input packets with dribble condition detected
     0 packets output, 0 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
	 0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out
GigabitEthernet0/0 is up, line protocol is up 
  Hardware is CN Gigabit Ethernet, address is 188b.9dc2.2620 (bia 188b.9dc2.2620)
  Description: WAN
  Internet address is 1.1.1.166/30
  MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec, 
     reliability 255/255, txload 2/255, rxload 8/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full Duplex, 100Mbps, media type is RJ45
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:16, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 108213
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 3513000 bits/sec, 494 packets/sec
  5 minute output rate 893000 bits/sec, 377 packets/sec
	333882510 packets input, 4021350135 bytes, 108208 no buffer
     Received 39298 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles 
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     283751011 packets output, 915184340 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
GigabitEthernet0/1 is up, line protocol is up 
  Hardware is CN Gigabit Ethernet, address is 188b.9dc2.2621 (bia 188b.9dc2.2621)
  Description: IPBlock
  Internet address is 2.2.2.2.65/26
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full Duplex, 1Gbps, media type is RJ45
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:02:34, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 905000 bits/sec, 377 packets/sec
  5 minute output rate 3511000 bits/sec, 495 packets/sec
     283274492 packets input, 1675614373 bytes, 0 no buffer
     Received 8400 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles 
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     333483661 packets output, 3604797528 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
GigabitEthernet0/2 is administratively down, line protocol is down 
  Hardware is CN Gigabit Ethernet, address is 188b.9dc2.2622 (bia 188b.9dc2.2622)
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Auto Duplex, Auto Speed, media type is RJ45
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles 
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 packets output, 0 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
ROUTER1#

Open in new window

0
sciggsAuthor Commented:
Did the config help any?  Do you need the config on my other router as well?

The interfaces are all set to full duplex, but at 100 and 1000mbps.  I do not believe there is any traffic shaping
0
giltjrCommented:
Did you put in the "scheduler allocate 20000 1000"?

The configuration looks fine.  However, in the configuration the interfaces are configured for auto speed and auto duplex, but you said they are set for 100 and 1000 Mpbs.  Do you really mean that they are getting negotiated to 100 and 1000 and full duplex?  If so, that is fine.

The only thing I might try is hard coding "100 mbps" interface to full duplex and see if anything changes.  Although a bit unusual for a device to negotiate to 100 Mbps and default to half duplex, it is possible.

I would also try setting up Gi0/2 for a mirror/monitor session on Gi0/0 and do a packet capture while you are doing the download and see if anything unusual shows up.
0
sciggsAuthor Commented:
I did not add the scheduler config.  I'm actually not really sure what that does.  Is it necessary?  Could it be causing any issues?

Yes, sorry, the interfaces are configured for auto but are being set to 100 and 1000 mbps.  Why would they negotiate to different values?
0
giltjrCommented:
I don't know if the scheduler could be causing a problem.  The values are a little high in my opinion, but I have not checked to see what the recommendation is for a 2911.

The speed negotiation (and duplex) is based on what the devices can handle that are connected to those ports.  So the port that shows 1000 Mbps is connected to a device that supports 1 Gbps, the one that shows 100 Mbps is connected to a device that supports 100 Mbps.

The possible problem on duplex is when duplex negotiation first came out not all vendors implemented it the same way and so sometimes one device thinks is should be full duplex while another device thinks it should be half.   When you have a duplex mismatch one side sees a lot of collisions, and the longer a connection lasts, the slower the speed gets because of the way collision detection/recovery works.
0
sciggsAuthor Commented:
Ok, the interface that is being negotiated to 100Mbps is the interface that is connected to the 20Mbps fiber converter from our ISP.  I confirmed that it is a 10/100 device.  

I may need to get my old 1761 powered on and check the interfaces on it to confirm how they were configured.  I thought Full Auto, but maybe not.


http://www.transition.com/TransitionNetworks/Products2/Family.aspx?Name=SSRFB10xx-100
0
giltjrCommented:
O.K., let me make sure I understand something.   In your original post you had "Yesterday, I was downloading a 25MB file at around 2Mbps,  ..... "

Did you really mean that you were getting 2 Mbps or did you mean you were getting 20 Mbps?

If you meant you were getting 20 Mbps, then yes all traffic that flows over this link would crawl.
0
sciggsAuthor Commented:
I meant that I was getting 2Mbps down.  I have tested since I started this thread and any download/upload to/from the Internet is crawling the network.  It doesn't matter the size of the file or really the speed of the down/up.  

For instance, I just ran some Java updates and while I was downloading the update I had a couple of employees come by and ask if the Internet was down.  As soon as the download finished, the network resumes as normal.
0
giltjrCommented:
If you have problems and you are only using 10% of your bandwidth I would definitely setup a mirror port on the router and do a packet capture on the Internet side of the router.


I would also, during off hours, do some speed tests from http://www.speedtest.net/.  This will choose a location as close to you has possible and see if you can get close to 20 Mbps.  If you are only getting 2 Mbps and the duplex all match something is wrong.  Maybe your ISP mis-provisioned the link and gave you 2 Mbps.
0
sciggsAuthor Commented:
I have done a SpeedTest and and my down and up are both right below 20Mbps.

My concern is that anytime someone downloads a file here or there, the network crawls.  This was not the case when we had the 1761 in place and our ISP was only supply us half our current connection (10Mbps).
0
sciggsAuthor Commented:
Does this look correct for setting up mirror port?

source port 0/0
destination port 0/2

monitor session 1 source interface GE0/0 0/0
monitor session 2 destination interface GE0/2

show monitor session 1
0
giltjrCommented:
You should not need the "0/0" on the end of the source definition, you just need:

  monitor session 1 source interface GE0/0

What concerns me is that when you are doing a file download you only get 10% of your available speed.  

When you do the file download do you know if you are using http or ftp?  Speed tests basically do a file download using http, so if you are downloading a file using http, I would expect it to get 20 Mbps and I would expect your Internet connection to crawl until the file down load was complete.
0
giltjrCommented:
Oh just remember that when you connect the computer you are doing the capture on to GE0/2 that you can't use that interface for anything but capturing data.  You will not be able to do any "normal" networking functions over that port.
0
sciggsAuthor Commented:
I'm not familiar with setting up port mirroring on router.  I also can't really find much documentation.  Do you have any linkage?


So for more examples.  Over the weekend, I needed to grab a file via from a remote FTP site.  I logged into a server remotely and started the ftp transfer.  As soon as it started, the RDP connection to the server timed out.  I hopped into our ASA and saw that our bandwidth was maxed at 20Mbps.  As soon as the FTP DL completed, it dropped back down to almost nothing (since weekend has very little traffic on our network).  I was then able to log back into the server.

This morning, a user was downloading Visual Studio files from Microsoft, and while it was downloading no one could browse out to the Internet.



I logged into my old router this morning and confirmed the interfaces were setup as full duplex and auto-speed.  But, the old router interfaces were only 10/100 which matches what the ISPs Fiber converter runs at.  Our new routers are 10/100/1000.  Does this have any relevance to you?
0
giltjrCommented:
I would expect a FTP connection to saturate a 20 Mbps link, so that does not surprise me that your link is saturated and a saturated link will have performance issues.

The part the concerns me is earlier you said when the firewall is showing that the link is running at 20 Mbps, your FTP session was only showing 2 Mbps, which is 10% of what is going over the link.  Is that still true?    If it is then you need to figure out what is using the other 18 Mbps.

The the 10/100 and 10/100/1000 is what the interface is capable  What the interface is capable of does not matter as long as the link speed is equal to or less than the max speed of the interface.

The mirror port is easy, it basically what you had:

monitor session 1 source interface Gi0/0 0/0
monitor session 1 destination interface Gi0/0/2
show monitor session 1

Then connect a PC to Gi0/0/2, start a packet capture utility like Wireshark, go to another computer and start the download.  Once the download is finished, stop wireshark, save the file, then do:

no monitor session 1

Now you can look at the Wireshark trace to see what was going on other than the FTP session.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
sciggsAuthor Commented:
Thanks.  That helps a lot.  I'm going to get this setup now so I can try to find the extra traffic.

Yes, when I was pulling the FTP my transfer speed was only just above 2000kbps, but it completely hosed the network.
0
giltjrCommented:
Just to make sure are you 100% positive it 2000kbps = 2 Mbps and not 2000 KBps = 2MBps?

Little b = bits and big B is Bytes.  2 MBps is about 20 Mbps because of the overhead.  It makes way more sense that you are getting 2 MBps and that hoses up your network.

Alot of FTP programs show the speed in Bytes, not bits.
0
sciggsAuthor Commented:
I was using WinSCP.  They notate transfer speeds as KB/s.  Our connection from ISP is 20Mbps which I take is transfer rates of around 2.5MB/s.  I get the point that in EX of the FTP download, I was pulling a close amount to that 2.5MBps max.

I guess there are a few points where I'm confused:

1.  Why was this never noticeable when we only had a 10Mbps line and an 1761 older router?  I know for a fact that it was not an issue then.
2.  Is this standard?  Should one user/server downloading Windows Updates be able crash the entire network until it's finished?
3.  At my home (AT&T Uverse) connection of 18Mbps, I don't experience this issue.  Over the weekend I pulled down a 60GB file over FTP to home network.  There were no issues with other Internet attempts during this time.
       

I feel like I"m missing something....
0
giltjrCommented:
O.K., 2.5 MB/ps is the max a 20Mbps link can do for raw through-put.  So you are maxing out your link.

I would have to do a little more research but  using process switching a 1761 can't even do 1 Mbps, usign CEF/Fast Switching it can get to 8Mpbs.

A 2851using process switching can get ti about 7 Mbps, using CEF/Fast Switching it can get to about 112 Mbps.

The 2911 can only do CEF/Fast Switching and can do up to 180 Mpbs.

So it possible that the other routers just happened to be configured it a way that they could never drive the 10 Mbps to its maximum individually.

http://www.cisco.com/web/partners/downloads/765/tools/quickreference/routerperformance.pdf
0
giltjrCommented:
You may want to look at the old configs, if you still have them and see if they had "ip cef.".
0
sciggsAuthor Commented:
I was looking at the old configs earlier today and noticed the ip cef settings.

Old configs
1761 Router (WAN) - ip cef
2851 Router (LAN) - no ip cef

New configs
2911 (WAN) - ip cef
2911 (LAN) - ip cef
0
giltjrCommented:
O.K, based on the CEF settings, neither of your old routers could drive your 10 Mbps link to 100% by itself.   Now the maximums in the link I sent you are based on "perfect" conditions, so most likely you would not reach those numbers.  Depending on how traffic within your network was configured to reach the routers, there was always some bandwidth left over while doing large downloads.

A single 2911 has the ability to drive your 20 Mbps to 100%.  I would look at doing some type of traffic shaping so that a single stream on either 2911 can't bring your whole Internet connect to a stop.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Network Analysis

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.