Solved

Cisco 6509 err-disabled

Posted on 2012-12-26
8
1,699 Views
Last Modified: 2013-01-03
we have recently had issues with our 6509 were about 12 ports in our blade went in to err-disabled mode. for some reason the syslog didn't give me much info. I have to give an explaination on how this happened. I figured it has to deal with excessive collisions on the back plane, that why all the ports went in to err-disabled mode at the same time? but i have no proof of this. I brought the ports down and back up and it brought them out of err-disabled mode. but im hessitent to plug any servers in to those ports. the chaise is around 8 years so they are defiantly dated.. any theories and thoughts are always welcome
0
Comment
Question by:mattlast
  • 2
  • 2
  • 2
  • +2
8 Comments
 
LVL 25

Expert Comment

by:Ken Boone
ID: 38721867
Well don't know why, but if you plug something into those ports and those ports go err-disabled, it will show a log message as to why the port went err-disabled.  So unless you plug something in to those ports and watch the logs you probably won't know why they went err-disabled.
0
 
LVL 10

Expert Comment

by:akhalighi
ID: 38721970
do you have port security enabled ? this usually happens if port security is configured and triggered . any recent acclivity that could change MAC addresses ? ( e.g. NIC card change on ESX servers , etc..)
0
 
LVL 10

Expert Comment

by:akhalighi
ID: 38721981
What do you get if you tye this command for one of those interfaces ?

show port-security interface fa 0/1

do you see any violation counters ?
0
Space-Age Communications Transitions to DevOps

ViaSat, a global provider of satellite and wireless communications, securely connects businesses, governments, and organizations to the Internet. Learn how ViaSat’s Network Solutions Engineer, drove the transition from a traditional network support to a DevOps-centric model.

 
LVL 50

Expert Comment

by:Don Johnston
ID: 38721983
What type of line card? There are some line cards where there is one ASIC which services 12 ports. Although I've never heard of an internal (ASIC or backplane) errdisabling ports.

What were the ports in question connected to? Are the ports part of an etherchannel group? Are they connected to one device? Is port security enabled? There are a number of things that can cause a port to go error disable. What is your current logging level?

Could you post the config (at the very least for the ports in question).
0
 
LVL 20

Expert Comment

by:rauenpc
ID: 38722046
show int status err-disable can show more details on why it went to an err-disabled state.

Setting errdisable recovery can at least give the ports a chance to automatically attempt to go back to a normal status. The interval shown is the default, and that can change either direction. I wouldn't go less than a minute because, depending on the error, this could cause more problems than solve.

errdisable recovery cause udld
errdisable recovery cause bpduguard
errdisable recovery cause security-violation
errdisable recovery cause channel-misconfig
errdisable recovery cause pagp-flap
errdisable recovery cause dtp-flap
errdisable recovery cause link-flap
errdisable recovery cause psecure-violation
errdisable recovery cause gbic-invalid
errdisable recovery cause dhcp-rate-limit
errdisable recovery cause vmps
errdisable recovery cause loopback
errdisable recovery interval 300
0
 

Author Comment

by:mattlast
ID: 38722095
These are server ports in a secured room so there was no port security set at the time. don't ask why........

I was just given a secondary syslog list what reported the following

2012-12-24 21:29:51 local7.warning X.X.X.X 1229277: Dec 24 22:29:51: %PM-SP-STDBY-4-ERR_DISABLE: packet-buffer error detected on Fa8/25, putting Fa8/25 in err-disable state

this happen to 11 ports

there was a second error
2012-12-24 21:29:51 local7.critical X.X.X.X  1229293: Dec 24 22:29:51: %PM_SCP-SP-2-LCP_FW_ERR_INFORM: Module 8 is experiencing the following error: LTL Parity error detected on Coil #2.


the full log look like this
2012-12-24 21:29:51 local7.warning x.x.x.x 1229277: Dec 24 22:29:51: %PM-SP-STDBY-4-ERR_DISABLE: packet-buffer error detected on Fa8/25, putting Fa8/25 in err-disable state
2012-12-24 21:29:51 local7.warning x.x.x.x 1229278: Dec 24 22:29:51: %PM-SP-STDBY-4-ERR_DISABLE: packet-buffer error detected on Fa8/26, putting Fa8/26 in err-disable state
2012-12-24 21:29:51 local7.warning x.x.x.x 1229279: Dec 24 22:29:51: %PM-SP-STDBY-4-ERR_DISABLE: packet-buffer error detected on Fa8/27, putting Fa8/27 in err-disable state
2012-12-24 21:29:51 local7.warning x.x.x.x 1229280: Dec 24 22:29:51: %PM-SP-STDBY-4-ERR_DISABLE: packet-buffer error detected on Fa8/28, putting Fa8/28 in err-disable state
2012-12-24 21:29:51 local7.warning x.x.x.x 1229281: Dec 24 22:29:51: %PM-SP-STDBY-4-ERR_DISABLE: packet-buffer error detected on Fa8/29, putting Fa8/29 in err-disable state
2012-12-24 21:29:51 local7.warning x.x.x.x 1229282: Dec 24 22:29:51: %PM-SP-STDBY-4-ERR_DISABLE: packet-buffer error detected on
2012-12-24 21:29:51 local7.warning x.x.x.x 1229283: Dec 24 22:29:51: %PM-SP-STDBY-4-ERR_DISABLE: packet-buffer error detected on Fa8/31, putting Fa8/31 in err-disable state
2012-12-24 21:29:51 local7.warning x.x.x.x 1229284: Dec 24 22:29:51: %PM-SP-STDBY-4-ERR_DISABLE: packet-buffer error detected on Fa8/32, putting Fa8/32 in err-disable state
2012-12-24 21:29:51 local7.warning x.x.x.x 1229285: Dec 24 22:29:51: %PM-SP-STDBY-4-ERR_DISABLE: packet-buffer error detected on Fa8/33, putting Fa8/33 in err-disable state
2012-12-24 21:29:51 local7.warning x.x.x.x 1229286: Dec 24 22:29:51: %PM-SP-STDBY-4-ERR_DISABLE: packet-buffer error detected on Fa8/34, putting Fa8/34 in err-disable state
2012-12-24 21:29:51 local7.warning x.x.x.x 1229287: Dec 24 22:29:51: %PM-SP-STDBY-4-ERR_DISABLE: packet-buffer error detected on Fa8/35, putting Fa8/35 in err-disable state
2012-12-24 21:29:51 local7.warning x.x.x.x 1229288: Dec 24 22:29:51: %PM-SP-STDBY-4-ERR_DISABLE: packet-buffer error detected on Fa8/36, putting Fa8/36 in err-disable state
2012-12-24 21:29:51 local7.warning x.x.x.x 1229289: Dec 24 22:29:51: %PM-SP-STDBY-4-ERR_DISABLE: packet-buffer error detected on Fa8/27, putting Fa8/27 in err-disable state
2012-12-24 21:29:51 local7.warning x.x.x.x 1229290: Dec 24 22:29:51: %PM-SP-STDBY-4-ERR_DISABLE: packet-buffer error detected on Fa8/28, putting Fa8/28 in err-disable state
2012-12-24 21:29:51 local7.warning x.x.x.x 1229291: Dec 24 22:29:51: %PM-SP-STDBY-4-ERR_DISABLE: packet-buffer error detected on Fa8/29, putting Fa8/29 in err-disable state
2012-12-24 21:29:51 local7.warning x.x.x.x 1229292: Dec 24 22:29:51: %PM-SP-STDBY-4-ERR_DISABLE: packet-buffer error detected on
2012-12-24 21:29:51 local7.critical x.x.x.x 1229293: Dec 24 22:29:51: %PM_SCP-SP-2-LCP_FW_ERR_INFORM: Module 8 is experiencing the following error: LTL Parity error detected on Coil #2.
2012-12-24 21:29:51 local7.warning x.x.x.x 1229294: Dec 24 22:29:51: %PM-SP-4-ERR_DISABLE: packet-buffer error detected on Fa8/25, putting Fa8/25 in err-disable state
2012-12-24 21:29:51 local7.warning x.x.x.x 1229295: Dec 24 22:29:51: %PM-SP-4-ERR_DISABLE: packet-buffer error detected on Fa8/26, putting Fa8/26 in err-disable state
2012-12-24 21:29:51 local7.warning x.x.x.x 1229296: Dec 24 22:29:51: %PM-SP-4-ERR_DISABLE: packet-buffer error detected on Fa8/27, putting Fa8/27 in err-disable state
2012-12-24 21:29:51 local7.warning x.x.x.x 1229297: Dec 24 22:29:51: %PM-SP-4-ERR_DISABLE: packet-buffer error detected on Fa8/28, putting Fa8/28 in err-disable state
2012-12-24 21:29:51 local7.warning x.x.x.x 1229298: Dec 24 22:29:51: %PM-SP-4-ERR_DISABLE: packet-buffer error detected on Fa8/29, putting Fa8/29 in err-disable state
2012-12-24 21:29:51 local7.warning x.x.x.x 1229299: Dec 24 22:29:51: %PM-SP-4-ERR_DISABLE: packet-buffer error detected on Fa8/30, putting Fa8/30 in err-disabl
2012-12-24 21:29:52 local7.warning x.x.x.x 1229300: Dec 24 22:29:51: %PM-SP-4-ERR_DISABLE: packet-buffer error detected on Fa8/31, putting Fa8/31 in err-disable state
2012-12-24 21:29:52 local7.warning x.x.x.x 1229301: Dec 24 22:29:51: %PM-SP-4-ERR_DISABLE: packet-buffer error detected on Fa8/32, putting Fa8/32 in err-disable state
2012-12-24 21:29:52 local7.warning x.x.x.x 1229302: Dec 24 22:29:51: %PM-SP-4-ERR_DISABLE: packet-buffer error detected on Fa8/33, putting Fa8/33 in err-disable state
2012-12-24 21:29:52 local7.warning x.x.x.x 1229303: Dec 24 22:29:51: %PM-SP-4-ERR_DISABLE: packet-buffer error detected on Fa8/34, putting Fa8/34 in err-disable state
2012-12-24 21:29:52 local7.warning x.x.x.x  1229304: Dec 24 22:29:52: %PM-SP-4-ERR_DISABLE: packet-buffer error detected on Fa8/35, putting Fa8/35 in err-disable state
2012-12-24 21:29:52 local7.warning x.x.x.x  1229305: Dec 24 22:29:52: %PM-SP-4-ERR_DISABLE: packet-buffer error detected on Fa8/36, putting Fa8/36 in err-disable state



As you can see based on the timestamp didnt make for a fun christmas ahah....
0
 
LVL 20

Accepted Solution

by:
rauenpc earned 500 total points
ID: 38722117
Bummer on the timestamps. I've seen these before, and we ended up getting the module replaced via TAC RMA. We had dozens of the same line cards in our data center that ended up getting replaced for this type of error. At the time we were told it was a hardware issue, but later I heard rumor that it turned out to be a software issue.

If you have smartnet, I would definitely start a TAC case.
0
 

Author Comment

by:mattlast
ID: 38722129
thanks i will bring it up in my next meeting any other ideas or similar exp. out there...
0

Featured Post

How our DevOps Teams Maximize Uptime

Our Dev teams are like yours. They’re continually cranking out code for new features/bugs fixes, testing, deploying, responding to production monitoring events and more. It’s complex. So, we thought you’d like to see what’s working for us. Read the use case whitepaper.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When posting a question about a Cisco ASA, Cisco Router or Cisco Switch, it can aid diagnosis if a suitably sanitised copy of the config is provided. It is much better to leave as much of the configuration as original as possible, as it could be tha…
I eventually solved a perplexing problem setting up telnet for a new switch.  I installed a new Cisco WS-03560X-24P switch connected to an existing Cisco 4506 running a WS-X4013-10GE Sup II-Plus. After configuring vlans and trunking,  I could no…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…

809 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question