Solved

Cisco Total Output Drops but no congestion or other errors? Causing RIP route flapping.

Posted on 2014-01-31
27
11,127 Views
Last Modified: 2014-04-08
Hi,

I have an issue with RIP routes flapping. In turned out the routes expired. I narrowed it down to the connection between the router and the switch.

The router connects with port gi0/0 to the switch on port 1/0/45. The router also connects with port gi0/1 to another switch. The RIP routes advertises to that interface have no problems.

When comparing the the counters with the gi0/0 and gi0/1 and related switch interfaces, the gi0/0 and gi1/0/45 which has the issues shows an increasing number of Total Output drops. Note this is a LAN 1Gbps link.
The other router interface where there is no issue shows no output drops at all.

Both interfaces are operating at 1000mbps, Full-duplex. The cable has been replaced and also I have moved to another switchport - the problem stays the same.
I checked the switch port and router port utilization and there is no congestion. Also the RAM and CPU utilization stays at about 50%.

The stats in the following output do not show collisions or buffer issues as far as I can tell. Is there anything else I can check to determine what the cause could be?

ROUTER:

GigabitEthernet0/0 is up, line protocol is up
  Hardware is MV96340 Ethernet, address is 001d.a188.c3e8 (bia 001d.a188.c3e8)
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 15/255, rxload 25/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is T
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 01:20:36
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 136
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 98869000 bits/sec, 15356 packets/sec
  5 minute output rate 61103000 bits/sec, 12528 packets/sec
     99849600 packets input, 4126409058 bytes, 0 no buffer
     Received 425455 broadcasts, 0 runts, 0 giants, 0 throttles
     86 input errors, 0 CRC, 0 frame, 53 overrun, 33 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     63684999 packets output, 3522032750 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 40 pause output
     0 output buffer failures, 0 output buffers swapped out

sh int gi0/0 stat
GigabitEthernet0/0
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor     593656  102003348     370929  189146886
             Route cache  109154112  992042457   73186952  208451216
                   Total  109747768 1094045805   73557881  397598102

#sh int gi0/0 switching
GigabitEthernet0/0
          Throttle count      19845
                   Drops         RP     754355         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs   57622670      Drops       1128

    Protocol  IP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process  167541736 2392721368  198576585 1057661994
            Cache misses          0          -          -          -
                    Fast 3626440266 3670415559 1743450588  446753404
               Auton/SSE          0          0          0          0

    Protocol  DEC MOP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0       5378     414106
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  ARP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process   23814141 1428858904   17718332 1104313164
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  CDP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process      53999   26351109      54567   23687545
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Other
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process    1682332   98123160     324160   19449600
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

#sh controllers gi0/0
Interface GigabitEthernet0/0
Hardware is MV96340
HWIDB: 44BDB6D8, INSTANCE: 44BDC7A0, FASTSEND: 425BB67C, MCI_INDEX: 0

Aggregate MIB Counters
----------------------
  Rx Good Bytes: 73869674736539        Rx Good Frames: 137091987
  Rx Bad Bytes: 0                      Rx Bad Frames: 0
  Rx Broadcast Frames: 424169          Rx Multicast Frames: 710630
  Tx Good Bytes: 40591651525           Tx Good Frames: 75717119
  Tx Broadcast Frames: 21594           Tx Multicast Frames: 67039
  Rx+Tx Min-64B Frames: 31071624       Rx+Tx 65-127B Frames: 58114245
  Rx+Tx 128-255B Frames: 7345060       Rx+Tx 256-511B Frames: 3300622
  Rx+Tx 512-1023B Frames: 1905727      Rx+Tx 1024-MaxB Frames: 111071828
  Rx Unrecog MAC Ctrl Frames: 0
  Rx Good FC Frames: 0                 Rx Bad FC Frames: 0
  Rx Undersize Frames: 0               Rx Fragment Frames: 0
  Rx Oversize Frames: 0                Rx Jabber Frames: 0
  Rx MAC Errors: 0                     Rx Bad CRCs: 0
  Tx Collisions: 0                     Tx Late Collisions: 0


SWITCH
========
#sh  int gi1/0/45 controller
GigabitEthernet1/0/45 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is f872.ead8.eaad (bia f872.ead8.eaad)
  Description: Link to LOROU000022-Internal Router GE0/0
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 28/255, rxload 15/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:57, output 00:00:00, output hang never
  Last clearing of "show interface" counters 01:38:59
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 67309
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 59458000 bits/sec, 12816 packets/sec
  5 minute output rate 111140000 bits/sec, 17149 packets/sec
     77485319 packets input, 41714170487 bytes, 0 no buffer
     Received 90441 broadcasts (68378 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 68378 multicast, 66 pause input
     0 input packets with dribble condition detected
     139259140 packets output, 139695027159 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

     Transmit GigabitEthernet1/0/45           Receive
   3548876580 Bytes                       1413733114 Bytes
   3512531302 Unicast frames              3967292070 Unicast frames
    406003061 Multicast frames              43777318 Multicast frames
    347377717 Broadcast frames              17619864 Broadcast frames
            0 Too old frames              3504628715 Unicast bytes
            0 Deferred frames             2981901373 Multicast bytes
            0 MTU exceeded frames         1200924241 Broadcast bytes
            0 1 collision frames                   0 Alignment errors
            0 2 collision frames                   0 FCS errors
            0 3 collision frames                   0 Oversize frames
            0 4 collision frames                   0 Undersize frames
            0 5 collision frames                   0 Collision fragments
            0 6 collision frames
            0 7 collision frames           229890069 Minimum size frames
            0 8 collision frames           118470682 65 to 127 byte frames
            0 9 collision frames          2754739620 128 to 255 byte frames
            0 10 collision frames         1748226139 256 to 511 byte frames
            0 11 collision frames          866208463 512 to 1023 byte frames
            0 12 collision frames          847612711 1024 to 1518 byte frames
            0 13 collision frames                  0 Overrun frames
            0 14 collision frames            1083138 Pause frames
            0 15 collision frames
            0 Excessive collisions                 0 Symbol error frames
            0 Late collisions                      0 Invalid frames, too large
            0 VLAN discard frames         1759592082 Valid frames, too large
            0 Excess defer frames                  0 Invalid frames, too small
   1665111570 64 byte frames                       0 Valid frames, too small
    989687737 127 byte frames
   4163293522 255 byte frames                      0 Too old frames
   1295948904 511 byte frames                     80 Valid oversize frames
    944047865 1023 byte frames                     0 System FCS error frames
   2695698037 1518 byte frames                     0 RxPortFifoFull drop frame
   1102059037 Too large frames
            0 Good (1 coll) frames
            0 Good (>1 coll) frames
0
Comment
Question by:salt-eit
  • 11
  • 4
  • 4
  • +3
27 Comments
 
LVL 15

Expert Comment

by:The_Warlock
ID: 39824565
Hi, can you do a:
<router#>sh run int gi0/0

And post the result back here. Thanks.
0
 
LVL 12

Expert Comment

by:Infamus
ID: 39824645
Just making sure, is cef enabled globally?
0
 

Author Comment

by:salt-eit
ID: 39824792
Hi, thanks for response.

CEF is enabled globally with "ip cef"

Interfaces speed was on auto. I set it to 1000 as a test, same issue.

ROUTER:
--------------
interface GigabitEthernet0/0
 no ip address
 ip route-cache flow
 duplex full
 speed 1000
end


SWITCH
------------
interface GigabitEthernet1/0/45
 description Link to ROU000022-Internal Router GE0/0
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 1-989
 switchport mode trunk
 speed 1000
end
0
 
LVL 12

Expert Comment

by:Infamus
ID: 39824890
Is there a reason you have configured gi1/0/45 as trunk?

Can you do "sh int status" and "sh int trunk" on the switch?
0
 
LVL 15

Expert Comment

by:The_Warlock
ID: 39824913
Ok, I didn't think CEF was going to be the issue here. I'm actually leaning towards QOS or flow control. I see you have one set to on and the switch side is off and or unsupported. Also, both sides are set for FIFO(QOS). In addition, it doesn't seem to appear as if your interfaces are congested at all considering the output below you are only using maybe 7-10% utilization.

Router:
reliability 255/255, txload 15/255, rxload 25/255
Switch:
 reliability 255/255, txload 28/255, rxload 15/255


 Lets try something since you are manually setting the intf's with speed and duplex which is only typical for 10/100mb speed interfaces.  Try using the "Negotiation Forced" command instead of manual settings. This is only used on Gig E interfaces and sets them to 1000/FULL while disabling flow control.  Let us know.
0
 
LVL 12

Expert Comment

by:Infamus
ID: 39824920
I like Warlock's idea.  Good point.
0
 
LVL 15

Expert Comment

by:The_Warlock
ID: 39824944
Also, maybe consider increasing your output buffer since its still set to the default "40" value but only after you try the other comment above that I left and clearing your counters. Wait a bit to see the the problem or issue stops.
0
 

Author Comment

by:salt-eit
ID: 39825010
It is a router-on-a-stick configuration therefore the trunk port.

Thee negotiation command does not exist on either the router or switch. Router is 2800 and switch is 3750x. I don't think they support that command.

Also, one side using flow control and the other side not - it is the same thing on the gi0/1 and other switch which has exact same config and RIP etc. but no issues.

This setup has been the same for years and has worked without issue. No config changes were done but the issue just popped up all of a sudden.
I am thinking that the interface could physically be damaged? The router is quite old, but it is a major process to replace the router etc. and major downtime so I want to rule out everything else before I take that route.

Here are the outputs of the other router interface and switch which do work:

ROUTER:
=======

#sh int gi0/1
GigabitEthernet0/1 is up, line protocol is up
  Hardware is MV96340 Ethernet, address is 001d.a188.c3e9 (bia 001d.a188.c3e9)
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 7/255, rxload 1/255
  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is T
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 1d21h
  Input queue: 0/75/525/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 2070000 bits/sec, 526 packets/sec
  5 minute output rate 30131000 bits/sec, 2717 packets/sec
     470194085 packets input, 3338585946 bytes, 0 no buffer
     Received 16224458 broadcasts, 0 runts, 0 giants, 352 throttles
     4181 input errors, 0 CRC, 0 frame, 50 overrun, 4131 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     704036441 packets output, 1845861858 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 5926 pause output
     0 output buffer failures, 0 output buffers swapped out

SWITCH
=======
SWI000047#sh int gi0/45
GigabitEthernet0/45 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 001e.49d3.b3ad (bia 001e.49d3.b3ad)
  Description: Link to LOROU000022-Internal Router GE0/1
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 7/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 1000Mb/s, link type is auto, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:50, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 28369000 bits/sec, 2425 packets/sec
  5 minute output rate 6811000 bits/sec, 1066 packets/sec
     5246088261 packets input, 6954369118816 bytes, 0 no buffer
     Received 14127646 broadcasts (0 multicasts)
     0 runts, 1 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 13043757 multicast, 183865 pause input
     0 input packets with dribble condition detected
     7043159501 packets output, 6272156189969 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
0
 
LVL 57

Expert Comment

by:giltjr
ID: 39825045
How busy is the 3750x that the router G0/0 is connected to?

Per: https://supportforums.cisco.com/thread/2159986

You may want to issue the following commands on the 3750x to see how many asic's you have, what the stats are and what ports they are associated to.

Although the port may not be overloaded, the asic associated with that port could be.

show controllers ethernet-controller port-asic statistics
show platform pm if-numbers
show platform pm platform-block
0
 
LVL 15

Expert Comment

by:The_Warlock
ID: 39825066
I was going to ask the same thing. Has their been a spike in traffic lately (micro-bursts)? And, thanks for clarifying the equipment in question because it makes more sense.
0
 

Author Comment

by:salt-eit
ID: 39825083
Yeah the switch used to be a 2960G but then got replaced with the 3750x about 2 months ago. The other switch where there is no issue is still a 2960G. When I changed switch ports for a test I changed it to port 47 instead of 45, which is on the same ASIC.

Output:

SWI000080#show controllers ethernet-controller port-asic statistics
===========================================================================
 Switch 1, PortASIC 0 Statistics
---------------------------------------------------------------------------

         0 TxBufferFull Drop Count               0 Rx Fcs Error Frames
         0 TxBufferFrameDesc BadCrc16            0 Rx Invalid Oversize Frames
         0 TxBuffer Bandwidth Drop Cou           0 Rx Invalid Too Large Frames
         0 TxQueue Bandwidth Drop Coun           0 Rx Invalid Too Large Frames
         0 TxQueue Missed Drop Statist           0 Rx Invalid Too Small Frames
1261764413 RxBuffer Drop DestIndex Cou           0 Rx Too Old Frames
         0 SneakQueue Drop Count                 0 Tx Too Old Frames
       156 Learning Queue Overflow Fra           0 System Fcs Error Frames
         0 TxBufferFrameDesc BadCrc16            0 Rx NP Packet Count0
         0 RxBuffer Drop DestIndex Cou           0 Rx NP Packet Count1

         0 Sup Queue 0 Drop Frames               0 Sup Queue 8 Drop Frames
         0 Sup Queue 1 Drop Frames               0 Sup Queue 9 Drop Frames
         0 Sup Queue 2 Drop Frames               0 Sup Queue 10 Drop Frames
         0 Sup Queue 3 Drop Frames               0 Sup Queue 11 Drop Frames
         0 Sup Queue 4 Drop Frames               0 Sup Queue 12 Drop Frames
         0 Sup Queue 5 Drop Frames               0 Sup Queue 13 Drop Frames
         0 Sup Queue 6 Drop Frames               0 Sup Queue 14 Drop Frames
         0 Sup Queue 7 Drop Frames               0 Sup Queue 15 Drop Frames
===========================================================================
 Switch 1, PortASIC 1 Statistics
---------------------------------------------------------------------------

         0 TxBufferFull Drop Count               0 Rx Fcs Error Frames
         0 TxBufferFrameDesc BadCrc16            0 Rx Invalid Oversize Frames
         0 TxBuffer Bandwidth Drop Cou           0 Rx Invalid Too Large Frames
         0 TxQueue Bandwidth Drop Coun           0 Rx Invalid Too Large Frames
         0 TxQueue Missed Drop Statist           0 Rx Invalid Too Small Frames
      2312 RxBuffer Drop DestIndex Cou           0 Rx Too Old Frames
         0 SneakQueue Drop Count                 0 Tx Too Old Frames
         0 Learning Queue Overflow Fra           0 System Fcs Error Frames
         0 TxBufferFrameDesc BadCrc16            0 Rx NP Packet Count0
      9749 RxBuffer Drop DestIndex Cou           0 Rx NP Packet Count1

         0 Sup Queue 0 Drop Frames               0 Sup Queue 8 Drop Frames
         0 Sup Queue 1 Drop Frames               0 Sup Queue 9 Drop Frames
         0 Sup Queue 2 Drop Frames               0 Sup Queue 10 Drop Frames
         0 Sup Queue 3 Drop Frames               0 Sup Queue 11 Drop Frames
         0 Sup Queue 4 Drop Frames               0 Sup Queue 12 Drop Frames
         0 Sup Queue 5 Drop Frames               0 Sup Queue 13 Drop Frames
         0 Sup Queue 6 Drop Frames               0 Sup Queue 14 Drop Frames
         0 Sup Queue 7 Drop Frames               0 Sup Queue 15 Drop Frames
===========================================================================
 Switch 1, PortASIC 2 Statistics
---------------------------------------------------------------------------

         0 TxBufferFull Drop Count               0 Rx Fcs Error Frames
         0 TxBufferFrameDesc BadCrc16            0 Rx Invalid Oversize Frames
         0 TxBuffer Bandwidth Drop Cou           0 Rx Invalid Too Large Frames
         0 TxQueue Bandwidth Drop Coun           0 Rx Invalid Too Large Frames
         0 TxQueue Missed Drop Statist           0 Rx Invalid Too Small Frames
    447663 RxBuffer Drop DestIndex Cou           0 Rx Too Old Frames
         0 SneakQueue Drop Count                 0 Tx Too Old Frames
     18463 Learning Queue Overflow Fra           0 System Fcs Error Frames
         0 TxBufferFrameDesc BadCrc16            0 Rx NP Packet Count0
    107080 RxBuffer Drop DestIndex Cou           0 Rx NP Packet Count1

         0 Sup Queue 0 Drop Frames        34166901 Sup Queue 8 Drop Frames
        65 Sup Queue 1 Drop Frames               0 Sup Queue 9 Drop Frames
         0 Sup Queue 2 Drop Frames        19829984 Sup Queue 10 Drop Frames
  47538550 Sup Queue 3 Drop Frames               0 Sup Queue 11 Drop Frames
         0 Sup Queue 4 Drop Frames               0 Sup Queue 12 Drop Frames
         0 Sup Queue 5 Drop Frames               0 Sup Queue 13 Drop Frames
         0 Sup Queue 6 Drop Frames               0 Sup Queue 14 Drop Frames
         0 Sup Queue 7 Drop Frames               0 Sup Queue 15 Drop Frames





SWI000080#show platform pm if-numbers

interface gid  gpn  lpn  port slot unit slun port-type lpn-idb gpn-idb
----------------------------------------------------------------------
Gi1/0/1   1    1    1    1/1  1    1    1    local     Yes     Yes
Gi1/0/2   2    2    2    1/0  1    2    2    local     Yes     Yes
Gi1/0/3   3    3    3    1/3  1    3    3    local     Yes     Yes
Gi1/0/4   4    4    4    1/2  1    4    4    local     Yes     Yes
Gi1/0/5   5    5    5    1/5  1    5    5    local     Yes     Yes
Gi1/0/6   6    6    6    1/4  1    6    6    local     Yes     Yes
Gi1/0/7   7    7    7    1/9  1    7    7    local     Yes     Yes
Gi1/0/8   8    8    8    1/8  1    8    8    local     Yes     Yes
Gi1/0/9   9    9    9    1/11 1    9    9    local     Yes     Yes
Gi1/0/10  10   10   10   1/10 1    10   10   local     Yes     Yes
Gi1/0/11  11   11   11   1/13 1    11   11   local     Yes     Yes
Gi1/0/12  12   12   12   1/12 1    12   12   local     Yes     Yes
Gi1/0/13  13   13   13   1/15 1    13   13   local     Yes     Yes
Gi1/0/14  14   14   14   1/14 1    14   14   local     Yes     Yes
Gi1/0/15  15   15   15   1/17 1    15   15   local     Yes     Yes
Gi1/0/16  16   16   16   1/16 1    16   16   local     Yes     Yes
Gi1/0/17  17   17   17   1/19 1    17   17   local     Yes     Yes
Gi1/0/18  18   18   18   1/18 1    18   18   local     Yes     Yes
Gi1/0/19  19   19   19   1/23 1    19   19   local     Yes     Yes
Gi1/0/20  20   20   20   1/22 1    20   20   local     Yes     Yes
Gi1/0/21  21   21   21   1/25 1    21   21   local     Yes     Yes
Gi1/0/22  22   22   22   1/24 1    22   22   local     Yes     Yes
Gi1/0/23  23   23   23   1/27 1    23   23   local     Yes     Yes
Gi1/0/24  24   24   24   1/26 1    24   24   local     Yes     Yes
Gi1/0/25  25   25   25   2/1  1    25   25   local     Yes     Yes
Gi1/0/26  26   26   26   2/0  1    26   26   local     Yes     Yes
Gi1/0/27  27   27   27   2/3  1    27   27   local     Yes     Yes
Gi1/0/28  28   28   28   2/2  1    28   28   local     Yes     Yes
Gi1/0/29  29   29   29   2/5  1    29   29   local     Yes     Yes
Gi1/0/30  30   30   30   2/4  1    30   30   local     Yes     Yes
Gi1/0/31  31   31   31   2/9  1    31   31   local     Yes     Yes
Gi1/0/32  32   32   32   2/8  1    32   32   local     Yes     Yes
Gi1/0/33  33   33   33   2/11 1    33   33   local     Yes     Yes

interface gid  gpn  lpn  port slot unit slun port-type lpn-idb gpn-idb
----------------------------------------------------------------------
Gi1/0/34  34   34   34   2/10 1    34   34   local     Yes     Yes
Gi1/0/35  35   35   35   2/13 1    35   35   local     Yes     Yes
Gi1/0/36  36   36   36   2/12 1    36   36   local     Yes     Yes
Gi1/0/37  37   37   37   2/15 1    37   37   local     Yes     Yes
Gi1/0/38  38   38   38   2/14 1    38   38   local     Yes     Yes
Gi1/0/39  39   39   39   2/17 1    39   39   local     Yes     Yes
Gi1/0/40  40   40   40   2/16 1    40   40   local     Yes     Yes
Gi1/0/41  41   41   41   2/19 1    41   41   local     Yes     Yes
Gi1/0/42  42   42   42   2/18 1    42   42   local     Yes     Yes
Gi1/0/43  43   43   43   2/23 1    43   43   local     Yes     Yes
Gi1/0/44  44   44   44   2/22 1    44   44   local     Yes     Yes
Gi1/0/45  45   45   45   2/25 1    45   45   local     Yes     Yes
Gi1/0/46  46   46   46   2/24 1    46   46   local     Yes     Yes
Gi1/0/47  47   47   47   2/27 1    47   47   local     Yes     Yes
Gi1/0/48  48   48   48   2/26 1    48   48   local     Yes     Yes
Gi1/1/1   49   49   49   0/6  1    49   49   local     Yes     Yes
Gi1/1/2   50   50   50   0/7  1    50   50   local     Yes     Yes
Gi1/1/3   51   51   51   0/20 1    51   51   local     Yes     Yes
Gi1/1/4   52   52   52   0/21 1    52   52   local     Yes     Yes
Te1/1/1   53   53   53   0/0  1    1    53   local     Yes     Yes
Te1/1/2   54   54   54   0/14 1    2    54   local     Yes     Yes
Po1       512  512  0    2/0  10   1    1    local     No      No
St1       896  896  0    16/0  1    0    0    internal  Yes     Yes




SWI000080#show platform pm platform-block

interface gid gpn lpn asic hw-i flags sp dp bundle vlan mvid mac so_di i_vlan
-----------------------------------------------------------------------------
Gi1/0/1   1   1   1   1    1       U  3  2  no     0    0    0   61441  0
Gi1/0/2   2   2   2   1    0       U  3  2  no     0    0    0   61442  0
Gi1/0/3   3   3   3   1    3       U  2  2  no     0    0    0   61443  0
Gi1/0/4   4   4   4   1    2       D  0  0  no     0    0    0   61444  0
Gi1/0/5   5   5   5   1    5       U  2  2  no     0    0    0   61445  0
Gi1/0/6   6   6   6   1    4       U  3  2  no     0    0    0   61446  0
Gi1/0/7   7   7   7   1    9       U  3  2  no     0    0    0   61447  0
Gi1/0/8   8   8   8   1    8       U  3  2  no     0    0    0   61448  0
Gi1/0/9   9   9   9   1    11      U  3  2  no     0    0    0   61449  0
Gi1/0/10  10  10  10  1    10      D  3  2  no     0    0    0   61450  0
Gi1/0/11  11  11  11  1    13      U  3  2  no     0    0    0   61451  0
Gi1/0/12  12  12  12  1    12      U  2  2  no     0    0    0   61452  0
Gi1/0/13  13  13  13  1    15      U  3  2  no     0    0    0   61453  0
Gi1/0/14  14  14  14  1    14      U  3  2  no     0    0    0   61454  0
Gi1/0/15  15  15  15  1    17      U  3  2  no     0    0    0   61455  0
Gi1/0/16  16  16  16  1    16      U  2  2  no     0    0    0   61456  0
Gi1/0/17  17  17  17  1    19      D  0  0  no     0    0    0   61457  0
Gi1/0/18  18  18  18  1    18      U  3  2  no     0    0    0   61458  0
Gi1/0/19  19  19  19  1    23      D  0  0  no     0    0    0   61459  0
Gi1/0/20  20  20  20  1    22      U  3  2  no     0    0    0   61460  0
Gi1/0/21  21  21  21  1    25      U  2  2  no     0    0    0   61461  0
Gi1/0/22  22  22  22  1    24      U  3  2  no     0    0    0   61462  0
Gi1/0/23  23  519 23  1    27      U  3  2  no     0    0    0   61959  0
Gi1/0/24  24  517 24  1    26      U  3  2  no     0    0    0   61957  0
Gi1/0/25  25  518 25  2    1       U  3  2  no     0    0    0   61958  0
Gi1/0/26  26  516 26  2    0       U  3  2  no     0    0    0   61956  0
Gi1/0/27  27  27  27  2    3       D  3  2  no     0    0    0   61467  0
Gi1/0/28  28  28  28  2    2       U  2  2  no     0    0    0   61468  0
Gi1/0/29  29  29  29  2    5       D  0  0  no     0    0    0   61469  0
Gi1/0/30  30  30  30  2    4       U  2  2  no     0    0    0   61470  0
Gi1/0/31  31  31  31  2    9       U  3  2  no     0    0    0   61471  0
Gi1/0/32  32  32  32  2    8       U  3  2  no     0    0    0   61472  0
Gi1/0/33  33  33  33  2    11      D  0  0  no     0    0    0   61473  0

interface gid gpn lpn asic hw-i flags sp dp bundle vlan mvid mac so_di i_vlan
-----------------------------------------------------------------------------
Gi1/0/34  34  34  34  2    10      U  3  2  no     0    0    0   61474  0
Gi1/0/35  35  35  35  2    13      U  3  2  no     0    0    0   61475  0
Gi1/0/36  36  36  36  2    12      U  3  2  no     0    0    0   61476  0
Gi1/0/37  37  37  37  2    15      U  2  2  no     0    0    0   61477  0
Gi1/0/38  38  38  38  2    14      U  2  2  no     0    0    0   61478  0
Gi1/0/39  39  39  39  2    17      U  3  2  no     0    0    0   61479  0
Gi1/0/40  40  40  40  2    16      U  3  2  no     0    0    0   61480  0
Gi1/0/41  41  41  41  2    19      U  3  2  no     0    0    0   61481  0
Gi1/0/42  42  42  42  2    18      U  3  2  no     0    0    0   61482  0
Gi1/0/43  43  43  43  2    23      U  3  2  no     0    0    0   61483  0
Gi1/0/44  44  44  44  2    22      U  3  2  no     0    0    0   61484  0
Gi1/0/45  45  45  45  2    25      U  3  2  no     0    0    0   61485  0
Gi1/0/46  46  46  46  2    24      U  3  2  no     0    0    0   61486  0
Gi1/0/47  47  47  47  2    27      D  3  2  no     0    0    0   61487  0
Gi1/0/48  48  48  48  2    26      D  3  2  no     0    0    0   61488  0
Gi1/1/1   49  49  49  0    6       U  3  2  no     0    0    0   61489  0
Gi1/1/2   50  50  50  0    7       D  0  0  no     0    0    0   61490  0
Gi1/1/3   51  51  51  0    20      D  0  0  no     0    0    0   61491  0
Gi1/1/4   52  52  52  0    21      D  0  0  no     0    0    0   61492  0
Te1/1/1   53  53  53  0    0       D  0  0  no     0    0    0   61493  0
Te1/1/2   54  54  54  0    14      D  0  0  no     0    0    0   61494  0
Po1       512 512 0   0    0       U  0  0  no     0    0    0   61952  0
0
 

Author Comment

by:salt-eit
ID: 39825128
There might be spikes in latency, but I do not see how that would make RIP routes expire with the long timers etc.

Here is a ping I ran with 5000byte packet size:

This is not a lot of time so not the most accurate test I suppose. There is a 12ms up from the average 3ms but I would not see that as a micro-burst?


Reply from 172.0.0.1: bytes=5000 time=3ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=5ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=2ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=3ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=4ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=4ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=2ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=2ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=2ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=2ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=2ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=3ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=3ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=2ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=3ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=12ms TTL=25
Reply from 172.0.0.1: bytes=5000 time=3ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=3ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=3ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=3ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=3ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=4ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=3ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=3ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=2ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=3ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=3ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=3ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=4ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=9ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=2ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=3ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=2ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=2ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=2ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=2ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=5ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=3ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=3ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=2ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=3ms TTL=255
Reply from 172.0.0.1: bytes=5000 time=2ms TTL=255
0
 

Author Comment

by:salt-eit
ID: 39825225
Perhaps this is evidence of microbursts? Look at the max latency.

This is from the router to the switch it is connected to:

Type escape sequence to abort.
Sending 2000, 1000-byte ICMP Echos to 10.0.4.48, timeout is 2 seconds:
.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 99 percent (1999/2000), round-trip min/avg/max = 1/3/44 ms
ROU000022#ping 10.0.4.48 rep 2000 size 5000

Type escape sequence to abort.
Sending 2000, 5000-byte ICMP Echos to 10.0.4.48, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (2000/2000), round-trip min/avg/max = 1/9/188 ms
ROU000022#
0
Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

 
LVL 61

Expert Comment

by:gheist
ID: 39826303
packet drops are to make higher level like tcp to slow down.
you can disable flow control on particular port and see if it gets better (or apply smartport templatr "switch")
0
 

Author Comment

by:salt-eit
ID: 39826715
I did "no mls qos" on the switch to disabled QoS. Since then the switch has had no output drops at all. I suppose this is due to buffers etc.

The router still has many  output drops. The RIP does not seem to have flapped again but I still need to monitor more to confirm.

Will let you know.
0
 
LVL 45

Expert Comment

by:Craig Beck
ID: 39826719
The switch is sending more traffic than the router can handle.  The_Warlock pointed this out...

Router:
reliability 255/255, txload 15/255, rxload 25/255
Switch:
 reliability 255/255, txload 28/255, rxload 15/255

You can see there that the switch is sending more than the router is receiving. so it's probably just that the hardware buffer is being flooded, hence the overruns on the router's interface.
0
 

Author Comment

by:salt-eit
ID: 39827387
Thanks. But if that is the case, is it not that the switch should show output drops and the router show input drops? Is it not that the router is sending more than the switch can handle?
0
 
LVL 45

Expert Comment

by:Craig Beck
ID: 39827534
I'm sure the other experts will correct me if I'm wrong, but...

No, the switch can't detect that the packet wasn't accepted at the router - the router has to tell the switch to resend.  The switch didn't fail to send the packet, so there's no error from the switch-side, but the router failed to receive it because of a full hardware buffer.
0
 
LVL 45

Expert Comment

by:Craig Beck
ID: 39827535
I suppose what you have to consider is that a 3750 is a far superior switch to a 2960.  You're not seeing this with the 2960, and I would think performance is a likely reason for this.
0
 
LVL 57

Assisted Solution

by:giltjr
giltjr earned 167 total points
ID: 39827734
Actually I think that neither the router NOR the switch will detect a drop and request a retransmit.  This is not part of L2 or L3 processing.

This is part of TCP and the two hosts involved in the TCP connection should detect that it received "packet sequence #4" but never got "packet sequence #3" and request a retransmit.

For UDP it is up to the application to detect that something was missed if you need "reliable UDP" communication.  Which if you need reliable UDP you should be using TCP anyway.  But that is another issue.

I believe the drops from the router to the switch indicates that the switch can't accept inbound traffic and has sent a XOFF to the router.  The router output queue got full, so it drop packets.  Once the switch can accept traffic it sends XON and more traffic flows in.

How busy are the other ports on the 3750 that share the ASIC with port gi1/0/45?

Although I'm fairly sure that craigbeck is correct that the 3750 is superior to 2960G, did you have the same number of devices connected to the 2960G as you do the 3750?  

In other words is the 3750 busier than the 2960 was?
1
 

Author Comment

by:salt-eit
ID: 39827839
Thanks for all the responses.

Giltjr explanation makes the most sense to me, but the following output should then show RxPortFifoFull as having increased right?

The ports connected to the 3750x is exactly the same as was connected to the previous 2960, so no increased load in that sense.


SWI000080#sh controllers | beg GigabitEthernet1/0/45
     Transmit GigabitEthernet1/0/45           Receive
   2703631028 Bytes                       4075869157 Bytes
    638700633 Unicast frames               506572274 Unicast frames
     13998329 Multicast frames               1278100 Multicast frames
      8500210 Broadcast frames                447704 Broadcast frames
            0 Too old frames              2541559695 Unicast bytes
            0 Deferred frames              598848988 Multicast bytes
            0 MTU exceeded frames           30425872 Broadcast bytes
            0 1 collision frames                   0 Alignment errors
            0 2 collision frames                   0 FCS errors
            0 3 collision frames                   0 Oversize frames
            0 4 collision frames                   0 Undersize frames
            0 5 collision frames                   0 Collision fragments
            0 6 collision frames
            0 7 collision frames            71107660 Minimum size frames
            0 8 collision frames           187966291 65 to 127 byte frames
            0 9 collision frames            52613241 128 to 255 byte frames
            0 10 collision frames           19910476 256 to 511 byte frames
            0 11 collision frames           12341813 512 to 1023 byte frames
            0 12 collision frames           66054345 1024 to 1518 byte frames
            0 13 collision frames                  0 Overrun frames
            0 14 collision frames               4201 Pause frames
            0 15 collision frames
            0 Excessive collisions                 0 Symbol error frames
            0 Late collisions                      0 Invalid frames, too large
            0 VLAN discard frames           98308457 Valid frames, too large
            0 Excess defer frames                  0 Invalid frames, too small
     23489237 64 byte frames                       0 Valid frames, too small
    249122103 127 byte frames
    126602851 255 byte frames                      0 Too old frames
     20520378 511 byte frames                      4 Valid oversize frames
     13516709 1023 byte frames                     0 System FCS error frames
    118206822 1518 byte frames                     0 RxPortFifoFull drop frame
    109741072 Too large frames
            0 Good (1 coll) frames
            0 Good (>1 coll) frames
0
 

Author Comment

by:salt-eit
ID: 39827853
To answer the other question, the other ports aren't really too busy. Ports 25 to 48 are on the same ASIC as I understand.

GigabitEthernet1/0/1 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/2 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/3 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/4 is down, line protocol is down (notconnect)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/5 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/6 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/7 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/8 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/9 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/10 is down, line protocol is down (notconnect)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/11 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/12 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/13 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/14 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/15 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/16 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/17 is down, line protocol is down (notconnect)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/18 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/19 is down, line protocol is down (notconnect)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/20 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/21 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/22 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/23 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/24 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/25 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/26 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/27 is down, line protocol is down (notconnect)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/28 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/29 is down, line protocol is down (notconnect)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/30 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/31 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/32 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/33 is down, line protocol is down (notconnect)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/34 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/35 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/36 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/37 is up, line protocol is up (connected)
     reliability 255/255, txload 14/255, rxload 5/255
GigabitEthernet1/0/38 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/39 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/40 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/41 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/42 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/43 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/44 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/45 is up, line protocol is up (connected)
     reliability 255/255, txload 2/255, rxload 2/255
GigabitEthernet1/0/46 is up, line protocol is up (connected)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/47 is down, line protocol is down (notconnect)
     reliability 255/255, txload 1/255, rxload 1/255
GigabitEthernet1/0/48 is down, line protocol is down (err-disabled)
0
 
LVL 12

Assisted Solution

by:Infamus
Infamus earned 167 total points
ID: 39828217
Just a basic recommendation and not sure if you already try this.

Have you tried swap the router ports between 2960 and 3750 and see if the issue still occurs on the same switch?
0
 
LVL 45

Assisted Solution

by:Craig Beck
Craig Beck earned 166 total points
ID: 39828519
Of course giltjr you are correct - I think I need more coffee :-)

I still think it's just an issue with the amount of traffic the router can handle from the 3750 though.  It is probably very bursty as The_Warlock was implying.
0
 
LVL 57

Expert Comment

by:giltjr
ID: 39828643
craigbeck, we all need more coffee sometimes.

salt-eit, Are any of the other ports Gi1/0/25 - 48 on the 3750 connected to a switch or router?  If so, are they seeing drops.

I'm wondering if there is a hardware issue with that ASIC.  If not, then I agree with The_Warlock too, bursty traffic.
0
 

Accepted Solution

by:
salt-eit earned 0 total points
ID: 39829042
At this point I can confirm that turning off QoS on switch stopped the RIP route flapping. Also, the switch has had no output drops whatsoever. Just the router still has output and input drops.

I have another Internet router connected to the same ASIC. There are no drops on that Interface as per below.

Could we by this safely assume that the switch ASIC is not too busy by the following?
1. None of the switch ports has had input queue drops.
2. Another router connected to the same ASIC does not experience output drops

Two things I will still attempt, but can only do much later as the environment is live:
1. Change the router to connect to another switch and see if problem persists. If it does, switch is ruled out.
2. Disconnect the LAN that connects to the router port and see if problem persists. If not, it is most likely bursty traffic from this LAN which causes the issue (because the other router port which is fine does not have this LAN connected to it).

OUTPUT OF OTHER ROUTER ON SAME ASIC:
=====================================

ROU000023#sh int gi 0/0
GigabitEthernet0/0 is up, line protocol is up
  Hardware is MV96340 Ethernet, address is 001d.a188.c428 (bia 001d.a188.c428)
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 3/255, rxload 3/255
  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is T
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/36/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 12006000 bits/sec, 2824 packets/sec
  5 minute output rate 14054000 bits/sec, 2901 packets/sec
     728125731 packets input, 193300290 bytes, 0 no buffer
     Received 887841 broadcasts, 0 runts, 0 giants, 2 throttles
     1916 input errors, 0 CRC, 0 frame, 0 overrun, 1916 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     726155067 packets output, 2154954223 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 3060 pause output
     0 output buffer failures, 0 output buffers swapped out
0
 

Author Closing Comment

by:salt-eit
ID: 39985495
The initial issue was RIP routes flapping and I found by myself that disabling QoS on switch solved issue. Why QoS did this is still a question but I think all other parties gave valuable info to assist and will assist greatly other people.
0

Featured Post

Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Suggested Solutions

If your business is like most, chances are you still need to maintain a fax infrastructure for your staff. It’s hard to believe that a communication technology that was thriving in the mid-80s could still be an essential part of your team’s modern I…
Data center, now-a-days, is referred as the home of all the advanced technologies. In-fact, most of the businesses are now establishing their entire organizational structure around the IT capabilities.
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now