Solved

Cisco:  output buffer failures

Posted on 2006-11-21
13
4,419 Views
Last Modified: 2008-11-27
Hi gurus

I have something odd in my Cisco switch

(Cisco Internetwork Operating System Software
IOS (tm) C3500XL Software (C3500XL-C3H2S-M), Version 12.0(5)WC13, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2005 by cisco Systems, Inc.
Compiled Tue 20-Sep-05 10:05 by antonino
Image text-base: 0x00003000, data-base: 0x00351FFC

ROM: Bootstrap program is C3500XL boot loader

Uutiset uptime is 4 weeks, 5 days, 9 hours, 37 minutes
System returned to ROM by power-on
System restarted at 02:05:55 UTC Thu Oct 19 2006
System image file is "flash:c3500xl-c3h2s-mz.120-5.WC13.bin"
)


I have "output buffer failures" in several interfaces. Here is one:


FastEthernet0/45 is up, line protocol is up
  Hardware is Fast Ethernet, address is 0001.426f.562d (bia 0001.426f.562d)
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, Auto Speed (100), 100BaseTX/FX
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:00, output hang never
  Last clearing of "show interface" counters 19:22:09
  Queueing strategy: fifo
  Output queue 0/40, 0 drops; input queue 0/75, 0 drops
  5 minute input rate 1000 bits/sec, 0 packets/sec
  5 minute output rate 4000 bits/sec, 6 packets/sec
     40331 packets input, 6677452 bytes
     Received 309 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast
     0 input packets with dribble condition detected
     256860 packets output, 47928385 bytes, 286 underruns                                        | 286 underruns !!!!!
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     286 output buffer failures, 0 output buffers swapped out                                        | 286 output buffer failures !!!!              




Now I really would like to get hints that what causing these errors ? What I should check next ?

Thanks for any help.

-Jussi
0
Comment
Question by:salmjuh
  • 5
  • 5
13 Comments
 
LVL 6

Expert Comment

by:nexissteve
ID: 17985936
Hi,

Can you also so a sh run on that interface and post back

Cheers

S
0
 

Author Comment

by:salmjuh
ID: 17985978
Here you are:

SW_2#sh run interface  fast 0/43
Building configuration...

Current configuration:
!
interface FastEthernet0/43
 duplex full
 speed 100
 spanning-tree portfast
end

SW_2#


-Jussi
0
 
LVL 6

Expert Comment

by:nexissteve
ID: 17986003
That is interface 0/43 - what about 0/45?

cheers

S
0
 

Author Comment

by:salmjuh
ID: 17986013
Sorry, stupid me ....

Well, here you are:

SW_2#sh run int fast 0/45
Building configuration...

Current configuration:
!
interface FastEthernet0/45
 duplex full
 spanning-tree portfast
end



-jussi
0
 
LVL 6

Expert Comment

by:nexissteve
ID: 17986030
Theres your problem I think.

You have stipulated duplex full with no speed statement.

Make sure your speed and duplex is set both ends, clear the couters on the interface and see how you go.

cheers

S
0
Zoho SalesIQ

Hassle-free live chat software re-imagined for business growth. 2 users, always free.

 

Author Comment

by:salmjuh
ID: 17986044
No I dount think so. Here is interface counters for 0/43 :


SW_2#sh run int fast 0/43
Building configuration...

Current configuration:
!
interface FastEthernet0/43
 duplex full
 speed 100
 spanning-tree portfast
end

SW_2#sh interfaces fastEthernet 0/43
FastEthernet0/43 is up, line protocol is up
  Hardware is Fast Ethernet, address is 0001.426f.562b (bia 0001.426f.562b)
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 100Mb/s, 100BaseTX/FX
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:00, output hang never
  Last clearing of "show interface" counters 21:10:47
  Queueing strategy: fifo
  Output queue 0/40, 0 drops; input queue 0/75, 0 drops
  5 minute input rate 1000 bits/sec, 1 packets/sec
  5 minute output rate 3000 bits/sec, 4 packets/sec
     5987569 packets input, 365058537 bytes
     Received 5185 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 1885 multicast
     0 input packets with dribble condition detected
     3534702 packets output, 275157695 bytes, 65 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     65 output buffer failures, 0 output buffers swapped out
SW_2#


So as you can see, there is also problems with packets and the interface is static speed state.


-jussi
0
 
LVL 6

Expert Comment

by:nexissteve
ID: 17986052
Hmmm but they are all output buffer hits...... this would imply the switch is hitting its limit on the uplink >

What is the switch uplinked to? And what speed backbone are you running?

Cheers

S
0
 

Author Comment

by:salmjuh
ID: 17986110
Hi S


The topology is like this


SW_1_Catalyst_C3560 ----- (Interface Gi 0/2) SW_2_Catalyst_3548 (Interface Gi 0/1) -------BackBone_Catalyst_4006_SUPIV



I checked bouth Gi interfaces on SW_2_Catalyst_3548 and those are 1000 Gb nad not even near maximum thru put. Peak is something like 100 Mbps.


Any more ideas ?

-Jussi
0
 
LVL 6

Expert Comment

by:nexissteve
ID: 17986147
Yep - memory utilisation on the switch "possibly" sh mem free

To better understand buffer hits and failures check these out.

http://www.cisco.com/warp/public/650/41.html

http://www.cisco.com/warp/public/63/buffertuning.html

And do a sh buff on your switch - should point you in the right direction.

Cheers

S
0
 

Author Comment

by:salmjuh
ID: 17986244
Hi S


I have cheked buffer, and I think its normal :

SW_2#sh buf
Buffer elements:
     478 in free list (500 max allowed)
     139011526 hits, 0 misses, 0 created

Public buffer pools:
Small buffers, 104 bytes (total 61, permanent 25):
     54 in free list (20 min, 60 max allowed)
     1346116109 hits, 32 misses, 60 trims, 96 created
     0 failures (0 no memory)
Middle buffers, 600 bytes (total 43, permanent 15):
     20 in free list (10 min, 30 max allowed)
     268031 hits, 915 misses, 2795 trims, 2823 created
     0 failures (0 no memory)
Big buffers, 1524 bytes (total 11, permanent 5):
     10 in free list (5 min, 10 max allowed)
     2159103 hits, 1561 misses, 4677 trims, 4683 created
     0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 2, permanent 0):
     2 in free list (0 min, 10 max allowed)
     3585936 hits, 2 misses, 2 trims, 4 created
     0 failures (0 no memory)
Large buffers, 5024 bytes (total 0, permanent 0):
     0 in free list (0 min, 5 max allowed)
     0 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
Huge buffers, 18024 bytes (total 0, permanent 0):
     0 in free list (0 min, 2 max allowed)
     5 hits, 2 misses, 4 trims, 4 created
     0 failures (0 no memory)

Interface buffer pools:
CPU10 buffers, 64 bytes (total 32, permanent 32):
     16 in free list (0 min, 32 max allowed)
     16 hits, 0 misses
CPU11 buffers, 64 bytes (total 32, permanent 32):
     16 in free list (0 min, 32 max allowed)
     706560 hits, 0 misses
CPU13 buffers, 1516 bytes (total 8, permanent 8):
     4 in free list (0 min, 8 max allowed)
     4 hits, 0 misses
CPU12 buffers, 1524 bytes (total 190, permanent 190):
     93 in free list (0 min, 190 max allowed)
     9413691 hits, 51 misses
CPU0 buffers, 1524 bytes (total 32, permanent 32):
     16 in free list (0 min, 32 max allowed)
     16 hits, 0 misses
CPU1 buffers, 1524 bytes (total 32, permanent 32):
     16 in free list (0 min, 32 max allowed)
     310607 hits, 0 misses
CPU2 buffers, 1524 bytes (total 4, permanent 4):
     2 in free list (0 min, 4 max allowed)
     190736000 hits, 0 misses
CPU3 buffers, 1524 bytes (total 4, permanent 4):
     2 in free list (0 min, 4 max allowed)
     65429513 hits, 0 misses
CPU4 buffers, 1524 bytes (total 4, permanent 4):
     2 in free list (0 min, 4 max allowed)
     65429548 hits, 0 misses
CPU5 buffers, 1524 bytes (total 4, permanent 4):
     2 in free list (0 min, 4 max allowed)
     190740748 hits, 0 misses
CPU6 buffers, 1524 bytes (total 4, permanent 4):
     2 in free list (0 min, 4 max allowed)
     190740747 hits, 0 misses
CPU7 buffers, 1524 bytes (total 4, permanent 4):
     2 in free list (0 min, 4 max allowed)
     190740746 hits, 0 misses
CPU8 buffers, 1524 bytes (total 4, permanent 4):
     2 in free list (0 min, 4 max allowed)
     190740745 hits, 0 misses
CPU9 buffers, 1524 bytes (total 4, permanent 4):
     2 in free list (0 min, 4 max allowed)
     190740745 hits, 0 misses


SW_2#





Memory looks good ?

SW_2#sh mem
               Head   Total(b)    Used(b)    Free(b)  Lowest(b) Largest(b)
Processor    4D7080   11702144    2804868    8897276    8775300    8815912
      I/O  70000000    1048576     641500     407076     294812     347384

------------------ CUT -------------------------------------------------


I'm running out of ideas ?


-jussi
0
 
LVL 5

Accepted Solution

by:
WGhen earned 125 total points
ID: 17986555
Hi,
I ran this through Cisco's "Output Interpretter" for you.




Interface FastEthernet0/45 (up/up)
  WARNING: Interface FastEthernet0/45 is up/up but keepalives are not set (set
  to 0 secs) so the interface could be spoofing. Spoofing means the interface appears
  up to higher layers so that packets can be routed to it, such as a DDR interface.
  TRY THIS: Set the keepalive timer to an integer value using the 'keepalive' interface
  configuration command. The default is 10 secs.
 
  INFO: The CRC erros found on the interface are less than 0.0001% of the total
  input packets and can be ignored.
 
  INFO: There have been 286 'output buffer failures' reported.
  If outgoing interface buffers are not available, an output buffer failure is reported.
  If an interface buffer is available but the Transmit Queue Limit is reached,
  the packet is dropped. However, if 'transmit-buffers backing-store' is enabled,
  the packet is placed in a System Buffer (which has to be obtained from an appropriate
  Free-List), and enqueued in the Output Hold queue for future transmission at
  the Process level, and an Output Buffer Swap is reported.
 
  WARNING: 286 underruns have been reported, which amounts to 0.11122% of the
  total input traffic. This is because, the far-end transmitter runs faster than
  the receiver of the near-end router can handle.
  TRY THIS: This problem can occur because the router is not powerful enough, and/or
  the interface is running at a slower speed. Analyze traffic patterns to determine
  the source of large amount of traffic received by the interface. However, this
  may not be possible because, these counters could have been incremented at some
  point in the past. Consider pasting the 'show buffer' command output into Output
  Interpreter to see if the buffers can be tuned.
  REFERENCE: For more information, see Performance Tuning Basics
 
  INFO: The last input for this interface is 'never' and the 'packets input' counter
  is greater than 0. The counters contradict each other. This is because the last
  input counter only gets timestamped when the CPU has to process an input packet.
  This does not get updated when network traffic is simply forwarded in hardware,
  without the CPU touching it. Most likely this output is from a switch interface
  or this interface supports some form of switching, such as fast, autonomous,
  silicon, netflow, etc.
0

Featured Post

Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

Join & Write a Comment

What’s a web proxy server? A proxy server is a server that goes between clients and web servers, used in corporate to enforce corporate browsing policy and ensure security. Proxy servers are commonly used in three modes. A)    Forward proxy …
When it comes to security, there are always trade-offs between security and convenience/ease of administration. This article examines some of the main pros and cons of using key authentication vs password authentication for hosting an SFTP server.
Viewers will learn how to connect to a wireless network using the network security key. They will also learn how to access the IP address and DNS server for connections that must be done manually. After setting up a router, find the network security…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now