Solved

# of output errors exceeds total # of packets

Posted on 2008-10-21
5
1,465 Views
Last Modified: 2012-05-05
I'm having an issue with dropped packets.  While troubleshooting, I noticed something odd...If I run a "sh int ___" command once, I see 682 or some number of dropped packets.  But sometimes, it returns a number much higher than the total number of packets that have gone through interface at all (31672318 drops, but only 6494 total packets out....)!!   Am I missing something?

Have a case open with TAC.....made no sense to them either.  The odd thing is the CNA shows 0 drops at all....this is happening on multiple ports, and across two totally different switch stacks.
MDC-3750#sh int g2/0/13

GigabitEthernet2/0/13 is up, line protocol is up (connected) 

  Hardware is Gigabit Ethernet, address is 001e.790e.b48d (bia 001e.790e.b48d)

  Description: oracle4 -- nic B

  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, 

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX

  input flow-control is on, output flow-control is unsupported 

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input never, output 00:00:10, output hang never

  Last clearing of "show interface" counters 00:43:22

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 682

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 1000 bits/sec, 1 packets/sec

  5 minute output rate 1000 bits/sec, 1 packets/sec

     6146 packets input, 4094108 bytes, 0 no buffer

     Received 0 broadcasts (0 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 0 multicast, 0 pause input

     0 input packets with dribble condition detected

     6494 packets output, 871514 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out

MDC-3750#sh int g2/0/13

GigabitEthernet2/0/13 is up, line protocol is up (connected) 

  Hardware is Gigabit Ethernet, address is 001e.790e.b48d (bia 001e.790e.b48d)

  Description: oracle4 -- nic B

  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, 

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX

  input flow-control is on, output flow-control is unsupported 

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input never, output 00:00:13, output hang never

  Last clearing of "show interface" counters 00:43:25

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 31672318

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 1000 bits/sec, 1 packets/sec

  5 minute output rate 1000 bits/sec, 1 packets/sec

     6146 packets input, 4094108 bytes, 0 no buffer

     Received 0 broadcasts (0 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 0 multicast, 0 pause input

     0 input packets with dribble condition detected

     6494 packets output, 871514 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out

MDC-3750#

Open in new window

0
Comment
Question by:j4llen
  • 3
  • 2
5 Comments
 
LVL 50

Expert Comment

by:Don Johnston
ID: 22772779
>Have a case open with TAC.....made no sense to them either.

Now that's... scary. :-o

The number of drops is not dependent on the number of packets coming in or going out. A "drop" simply means that was no room in the queue for the packet. If the queue for an interface is full, the next packet in gets dropped. This is fairly common when you have traffic entering a 1gig interface but destined to exit a 100m interface. There's going to be a backup and the queues will fill up.

Even if the inbound and outbound interfaces are both 1gig this can happen. If there are multiple ports receiving traffic that is destined for a single outgoing port.

http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a0080094791.shtml


0
 

Author Comment

by:j4llen
ID: 22772822
Even if the inbound and outbound interfaces are both 1gig this can happen. If there are multiple ports receiving traffic that is destined for a single outgoing port.

I kind of reached that conclusion (that it was normal....congestion....)  We're seeing it on several ports, but they were all either connected to either an oracle server's iSCSI MPIO interface, or an iSCSI disk array.

My guess on the oracle-bound interfaces was that either the NICs didn't have large enough cache/buffer (the oracle error that prompted all of this searching on my part recommended this as a solution), or something to do with the MPIO.

For the iSCSI arrays....exactly as you said, multiple gigE connected servers trying to talk to an iSCSI array via a single gigE interface.

You've answered how the total drops and total # of packets can be different -- but why does the same command run back-to-back vary so wildly in results???  And an interface that was just cleared 30 mins ago I can't imagine would have already accumulated 32 million discarded packets?
0
 

Author Comment

by:j4llen
ID: 22772878
Just to clarify -- take a close look at the output I posted -- I actually ran a "show interface g2/0/13" twice, back-to-back, a second apart.  Everything was about the same except the "Total output drops."  Happens on all of the affected switch ports, and on 2 different stacks.

First time--Total output drops: 682
2nd time--Total output drops: 31672318
0
 
LVL 50

Accepted Solution

by:
Don Johnston earned 500 total points
ID: 22772889
No doubt about it. That's a bunch of drops in a short period of time with not a large amount of traffic. What's connected to g2/0/13? I would be looking at what's coming in that interface and where it's going.

Then again, it could be buggy code. Have you run the IOS version through the feature navigator to see if there are any known bugs associated with it?

Output interpreter had this to say:

Interface GigabitEthernet2/0/13 (up/up (connected))
  INFO: The last input for this interface is 'never' and the 'packets input' counter
  is greater than 0. The counters contradict each other. This is because the last
  input counter only gets timestamped when the CPU has to process an input packet.
  This does not get updated when network traffic is simply forwarded in hardware,
  without the CPU touching it. Most likely this output is from a switch interface
  or this interface supports some form of switching, such as fast, autonomous,
  silicon, netflow, etc.

Which is another way of saying that it's normal (for a switch). It's curious that it didn't jump on the number of drops.

0
 

Author Closing Comment

by:j4llen
ID: 31509020
It's a bug!  Not on my version 12.2(40)SE, but on 12.2(44)SE -- TAC suspects it goes back to my version.  They told me to run a "sh platform port-asics port drops" saying it would show the correct number of drops, but unfortunately it lists the drops by asic, so I can't confirm whether it is correct or not either.  But I'm more or less satisfied at this point that its not a configuration issue on my end.

CSCso81660 [QDDTS] [CCO] [CDETS Bug Display] Externally found minor (Sev4) bug: Verified (V) 3750 stack shows incorrect values for output drops

Symptom:
3750 stack shows incorrect values for output drops on show interfaces

Conditions:
This is seen on 3750 G stacks running 12.2(44)SE or 12.2(44)SE1.

Workaround:
None.  Fixed in 12.2(46)SE

Further Problem Description:
The 'show platform port-asic stats drops' shows the correct values. The issue is causing interoperablity issue with SNMP.
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

If you’re like me and you like peace and quiet, saving money, and pretty lights, then this article is for you. For financial reasons, I buy all the Cisco equipment for my home lab second-hand. The first thing to wear out is usually one of the coo…
This tutorial will go through the steps required to write a script that will back up the configuration settings of a HP-ProCurve switch. You will need to get the following things to follow this tutorial: Telnet Scripting Tool e.g. TST10.exe …
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…
This video demonstrates how to create an example email signature rule for a department in a company using CodeTwo Exchange Rules. The signature will be inserted beneath users' latest emails in conversations and will be displayed in users' Sent Items…

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

23 Experts available now in Live!

Get 1:1 Help Now