Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1497
  • Last Modified:

# of output errors exceeds total # of packets

I'm having an issue with dropped packets.  While troubleshooting, I noticed something odd...If I run a "sh int ___" command once, I see 682 or some number of dropped packets.  But sometimes, it returns a number much higher than the total number of packets that have gone through interface at all (31672318 drops, but only 6494 total packets out....)!!   Am I missing something?

Have a case open with TAC.....made no sense to them either.  The odd thing is the CNA shows 0 drops at all....this is happening on multiple ports, and across two totally different switch stacks.
MDC-3750#sh int g2/0/13
GigabitEthernet2/0/13 is up, line protocol is up (connected) 
  Hardware is Gigabit Ethernet, address is 001e.790e.b48d (bia 001e.790e.b48d)
  Description: oracle4 -- nic B
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is on, output flow-control is unsupported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:10, output hang never
  Last clearing of "show interface" counters 00:43:22
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 682
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 1000 bits/sec, 1 packets/sec
  5 minute output rate 1000 bits/sec, 1 packets/sec
     6146 packets input, 4094108 bytes, 0 no buffer
     Received 0 broadcasts (0 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     6494 packets output, 871514 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
MDC-3750#sh int g2/0/13
GigabitEthernet2/0/13 is up, line protocol is up (connected) 
  Hardware is Gigabit Ethernet, address is 001e.790e.b48d (bia 001e.790e.b48d)
  Description: oracle4 -- nic B
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is on, output flow-control is unsupported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:13, output hang never
  Last clearing of "show interface" counters 00:43:25
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 31672318
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 1000 bits/sec, 1 packets/sec
  5 minute output rate 1000 bits/sec, 1 packets/sec
     6146 packets input, 4094108 bytes, 0 no buffer
     Received 0 broadcasts (0 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     6494 packets output, 871514 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
MDC-3750#

Open in new window

0
j4llen
Asked:
j4llen
  • 3
  • 2
1 Solution
 
Don JohnstonInstructorCommented:
>Have a case open with TAC.....made no sense to them either.

Now that's... scary. :-o

The number of drops is not dependent on the number of packets coming in or going out. A "drop" simply means that was no room in the queue for the packet. If the queue for an interface is full, the next packet in gets dropped. This is fairly common when you have traffic entering a 1gig interface but destined to exit a 100m interface. There's going to be a backup and the queues will fill up.

Even if the inbound and outbound interfaces are both 1gig this can happen. If there are multiple ports receiving traffic that is destined for a single outgoing port.

http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a0080094791.shtml


0
 
j4llenAuthor Commented:
Even if the inbound and outbound interfaces are both 1gig this can happen. If there are multiple ports receiving traffic that is destined for a single outgoing port.

I kind of reached that conclusion (that it was normal....congestion....)  We're seeing it on several ports, but they were all either connected to either an oracle server's iSCSI MPIO interface, or an iSCSI disk array.

My guess on the oracle-bound interfaces was that either the NICs didn't have large enough cache/buffer (the oracle error that prompted all of this searching on my part recommended this as a solution), or something to do with the MPIO.

For the iSCSI arrays....exactly as you said, multiple gigE connected servers trying to talk to an iSCSI array via a single gigE interface.

You've answered how the total drops and total # of packets can be different -- but why does the same command run back-to-back vary so wildly in results???  And an interface that was just cleared 30 mins ago I can't imagine would have already accumulated 32 million discarded packets?
0
 
j4llenAuthor Commented:
Just to clarify -- take a close look at the output I posted -- I actually ran a "show interface g2/0/13" twice, back-to-back, a second apart.  Everything was about the same except the "Total output drops."  Happens on all of the affected switch ports, and on 2 different stacks.

First time--Total output drops: 682
2nd time--Total output drops: 31672318
0
 
Don JohnstonInstructorCommented:
No doubt about it. That's a bunch of drops in a short period of time with not a large amount of traffic. What's connected to g2/0/13? I would be looking at what's coming in that interface and where it's going.

Then again, it could be buggy code. Have you run the IOS version through the feature navigator to see if there are any known bugs associated with it?

Output interpreter had this to say:

Interface GigabitEthernet2/0/13 (up/up (connected))
  INFO: The last input for this interface is 'never' and the 'packets input' counter
  is greater than 0. The counters contradict each other. This is because the last
  input counter only gets timestamped when the CPU has to process an input packet.
  This does not get updated when network traffic is simply forwarded in hardware,
  without the CPU touching it. Most likely this output is from a switch interface
  or this interface supports some form of switching, such as fast, autonomous,
  silicon, netflow, etc.

Which is another way of saying that it's normal (for a switch). It's curious that it didn't jump on the number of drops.

0
 
j4llenAuthor Commented:
It's a bug!  Not on my version 12.2(40)SE, but on 12.2(44)SE -- TAC suspects it goes back to my version.  They told me to run a "sh platform port-asics port drops" saying it would show the correct number of drops, but unfortunately it lists the drops by asic, so I can't confirm whether it is correct or not either.  But I'm more or less satisfied at this point that its not a configuration issue on my end.

CSCso81660 [QDDTS] [CCO] [CDETS Bug Display] Externally found minor (Sev4) bug: Verified (V) 3750 stack shows incorrect values for output drops

Symptom:
3750 stack shows incorrect values for output drops on show interfaces

Conditions:
This is seen on 3750 G stacks running 12.2(44)SE or 12.2(44)SE1.

Workaround:
None.  Fixed in 12.2(46)SE

Further Problem Description:
The 'show platform port-asic stats drops' shows the correct values. The issue is causing interoperablity issue with SNMP.
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 3
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now