Solved

cisco 7206 flapping

Posted on 2011-09-14
10
646 Views
Last Modified: 2012-05-12
I have a 7206 with a fast ethernet connection to an isp that flaps on a random basis. Sometimes it comes back up without intervention and sometimes I have to go in and do a clear interface. A test that was done on my end was to swap that interface with another interface on the 7206 being used for another circuit. So we moved ISP a from fa4/0 to fa5/0 and isp b from fa5/0 to fa4/0 for instance. The problem seemed to follow isp a also. They are saying nothing is wrong on their end. Not to sure where to go from here, any suggestions?
0
Comment
Question by:chipsch
  • 7
  • 3
10 Comments
 

Author Comment

by:chipsch
ID: 36536214
interface Loopback0
 description xxxx B-Peer for Inbound Routes
 ip address 38.103.65.59 255.255.255.255

interface FastEthernet5/0
 description xxxx
 ip address 38.104.96.30 255.255.255.252
 duplex full
 no cdp enable
!
interface FastEthernet6/0
 description xxxx
 ip address 64.132.85.134 255.255.255.252
 duplex full
 no cdp enable
!
router eigrp 6349
 redistribute connected
 redistribute static
 network 216.114.2.0
 no default-information in
 no default-information out
 distribute-list 99 out
 distribute-list 99 in
 no auto-summary
!
router bgp 6349
 no synchronization
 bgp router-id 38.104.96.30
 bgp log-neighbor-changes
 network 38.103.65.59 mask 255.255.255.255
 network 206.126.224.0
 network 206.126.229.0
 network 206.126.230.0
 network 206.126.231.0
 network 206.126.234.0
 network 206.126.241.0
 network 206.126.245.0
 network 206.126.250.0
 network 206.126.253.0
 network 206.126.254.0
 network 216.114.0.0 mask 255.255.224.0
 neighbor 38.103.65.58 remote-as 198
 neighbor 38.103.65.58 description xxxxx
 neighbor 38.103.65.58 ebgp-multihop 5
 neighbor 38.103.65.58 soft-reconfiguration inbound
 neighbor 38.103.65.58 prefix-list xxxx out
 neighbor 38.103.65.58 filter-list 10 out
 neighbor 38.104.96.29 remote-as 198
 neighbor 38.104.96.29 description xxxxx
 neighbor 38.104.96.29 ebgp-multihop 5
 neighbor 38.104.96.29 soft-reconfiguration inbound
 neighbor 38.104.96.29 prefix-list xxxx
 neighbor 38.104.96.29 filter-list 10 out
 neighbor 64.132.85.133 remote-as 2121
 neighbor 64.132.85.133 description xxxx
 neighbor 64.132.85.133 soft-reconfiguration inbound
 neighbor 64.132.85.133 prefix-list AS6349-EXPORT out
 neighbor 64.132.85.133 filter-list 10 out
 maximum-paths 2
 no auto-summary
!
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 38.104.96.29 100
ip route 0.0.0.0 0.0.0.0 64.132.85.133 200
ip route 206.126.245.0 255.255.255.0 38.104.96.29
ip route 206.126.245.16 255.255.255.240 38.104.96.29
ip route 206.126.245.32 255.255.255.240 38.104.96.29
ip route 206.126.245.40 255.255.255.248 38.104.96.29
ip route 206.126.245.48 255.255.255.248 38.104.96.29
ip route 206.126.245.56 255.255.255.248 38.104.96.29
ip route 206.126.245.72 255.255.255.248 38.104.96.29
ip route 206.126.250.0 255.255.255.0 38.104.96.29
ip route 206.126.253.0 255.255.255.0 38.104.96.29
ip route 206.126.254.0 255.255.255.0 38.104.96.29
ip route 216.114.0.0 255.255.224.0 Null0
!
no ip http server
no ip http secure-server
!
ip as-path access-list 10 permit ^$
!
!
ip prefix-list 75 seq 5 permit 216.114.0.0/19
!
ip prefix-list AS6349-EXPORT seq 5 permit 216.114.0.0/19
ip prefix-list AS6349-EXPORT seq 15 permit 206.126.224.0/24
ip prefix-list AS6349-EXPORT seq 20 permit 206.126.229.0/24
ip prefix-list AS6349-EXPORT seq 25 permit 206.126.230.0/24
ip prefix-list AS6349-EXPORT seq 30 permit 206.126.231.0/24
ip prefix-list AS6349-EXPORT seq 35 permit 206.126.234.0/24
ip prefix-list AS6349-EXPORT seq 40 permit 206.126.241.0/24
ip prefix-list AS6349-EXPORT seq 45 permit 206.126.245.0/24
ip prefix-list AS6349-EXPORT seq 50 permit 206.126.250.0/24
ip prefix-list AS6349-EXPORT seq 55 permit 206.126.253.0/24
ip prefix-list AS6349-EXPORT seq 60 permit 206.126.254.0/24
ip prefix-list AS6349-EXPORT seq 65 permit 206.126.224.0/19
ip prefix-list AS6349-EXPORT seq 70 deny 0.0.0.0/0 le 32
!
ip prefix-list xxxx seq 5 permit 216.114.0.0/19
ip prefix-list xxxx seq 10 permit 38.103.65.59/32
ip prefix-list xxxx seq 20 permit 206.126.224.0/24
ip prefix-list xxxx seq 25 permit 206.126.229.0/24
ip prefix-list xxxx seq 30 permit 206.126.230.0/24
ip prefix-list xxxx seq 35 permit 206.126.231.0/24
ip prefix-list xxxx seq 40 permit 206.126.234.0/24
ip prefix-list xxxx seq 45 permit 206.126.241.0/24
ip prefix-list xxxx seq 50 permit 206.126.245.0/24
ip prefix-list xxxx seq 55 permit 206.126.250.0/24
ip prefix-list xxxx seq 60 permit 206.126.253.0/24
ip prefix-list xxxx seq 65 permit 206.126.254.0/24
ip prefix-list xxxx seq 70 permit 206.126.224.0/19
ip prefix-list xxxx seq 75 deny 0.0.0.0/0 le 32

relevant configuration is here also.



0
 
LVL 1

Expert Comment

by:bitsmakebytzesrun
ID: 36538128
sounds like a layer 2 problem, so try hardcoding the interface to 100/full if they will do the same on their end.  also try auto/auto (no duplex full) and see their equipment likes this better.  swap out the cable also. this will cause brief outages, so keep that in mind when doing the work.  also, keep in mind that if bgp notices the bounces it may enforce route damping (your end and/or theirs) which could last upwards of 15 minutes.
0
 

Author Comment

by:chipsch
ID: 36540000
Layer 1 and 2 do stay up whenever this happens. Probably should have included that earlier. Whenever it flaps I am seeing BGP down notification sent and the ISP is seeing BGP down notification received. I am wondering since we are receiving full updates if something is happening on their end and then flooding our end with updates and causing the keepalives to be stored in queue and eventually dropped. Duplex is full and speed is 100 also so that is not an issue either. Didn't think about route dampening though. Normally when my eem applet I have configured as a temporary fix kicks off to do a clear interface it comes right back up. Added a line for an e-mail to be sent with the $_cli_result of show ip bgp neighbors and show ip bgp summary. Hoping that could give some insight. The odd thing is since we are receiving full updates we have to peer with a directly connected and their core which is from a loopback address we advertise in BGP. They stated this had something to do with not wanting to cause excessive cpu processes on our end. Never heard of anything like that before.
0
 
LVL 1

Expert Comment

by:bitsmakebytzesrun
ID: 36544229
I would recommend do snmp polls of the interface, CPU, and memory for the box to make sure it's not a resource issue.  That being said, also confirm you have no layer 2 issues, specifically checking the Input queue for drops and for input errors/CRC/framing, etc.  These should all be clean, but you might need to clear counters and wait for it to happen again if you've got old stats.  If stats are clean, I would try setting interface to auto/auto to see if their are negotiation problems not showing up in interface stats.

We use loopbacks for advertisement, but it's not a CPU issue it has more to do with not wanting neighbor relationships to bounce if an interface is bouncing/having problems (ie loopbacks never go down/bounce,etc).  Basically keeps routing more stable by not tying it to interface behavior.

sho int f0/5
..
  Input queue: 1/75/0/0 (size/max/drops/flushes); Total output drops: 0
..
    3348744943 packets input, 3916042109 bytes, 0 no buffer
     Received 37719941 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 37763734 multicast, 0 pause input
     0 input packets with dribble condition detected
     3358351898 packets output, 3580700362 bytes, 0 underruns
     4 output errors, 0 collisions, 3 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     4 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

0
 

Author Comment

by:chipsch
ID: 36548796
We have Orion polling the device and I have never noticed a resource issue actually. There are input errors and they increment sporadically. Interface stats are below:


 Last clearing of "show interface" counters 1d23h
  Input queue: 1/75/154/3 (size/max/drops/flushes); Total output drops: 9118
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 493000 bits/sec, 291 packets/sec
  5 minute output rate 571000 bits/sec, 249 packets/sec
     72828155 packets input, 46899869699 bytes
     Received 0 broadcasts, 0 runts, 0 giants, 45 throttles
     1213 input errors, 0 CRC, 0 frame, 5 overrun, 1208 ignored
     0 watchdog
     0 input packets with dribble condition detected
     47962936 packets output, 20087105486 bytes, 1 underruns
     1 output errors, 1 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out

Odd thing is that the other circuit handles most of the traffic and never sees the output drops and doesn't increment input errors at that rate. What is odd is that normally when I see a number of input errors there are some CRC errors to go along with it. None in this case.
0
Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

 

Author Comment

by:chipsch
ID: 36583639
Another thing that I have noticed is that the input errors and ignored increase drastically whenever this happens. Could it be a buffer issue where a large number of packets are filling the buffers and causing failures or even a buffer leak holding the buffer slots?
0
 

Author Comment

by:chipsch
ID: 36709240
Any suggestions from anyone? I am hoping next time this happens and my eem script kicks off that the clear interface doesn't bring it right back up so that I can maybe hop in and get some debug output to post.
0
 
LVL 1

Accepted Solution

by:
bitsmakebytzesrun earned 500 total points
ID: 36711199
could be buffers, if so you could adjust these (carefully).  It would be interesting to have a sniffer on the net to see what is causing these traffic spikes, and if it is out of the norm then stop the traffic source (vs adjust buffer to accommodate abnormal traffic).  also, there are hardware buffer for the physical interfaces that cannot be adjusted and you will have to solve what is causing the traffic problem/ie no buffer solution (ie ignored errors):

buffer adjustment:
http://www.cisco.com/en/US/products/hw/modules/ps2643/products_tech_note09186a0080093fc5.shtml

ignored:
Shows the number of received packets ignored by the interface because the interface hardware ran low on internal buffers. These buffers are different from the system buffers mentioned previously in the buffer description. Broadcast storms and bursts of noise can cause the ignored count to be increased.

no buffers:
Gives the number of received packets discarded because there was no buffer space in the main system. Compare this with the ignored count. Broadcast storms on Ethernet networks and bursts of noise on serial lines are often responsible for no input buffer events.

input error:
Includes runts, giants, no buffer, CRC, frame, overrun, and ignored counts. Other input-related errors can also cause the input error count to be increased, and some datagrams may have more than one error; therefore, this sum may not balance with the sum of enumerated input error counts.
0
 

Author Comment

by:chipsch
ID: 36711605
I have been thinking more and more about doing a sniffer also. I had a cheat sheet somewhere showing how to do a capture with the 7206 if running the 12.4 train but I can't seem to find it. Thanks for all of the help also bitsmakebytzesrun, which by the way is an awesome name.
0
 

Author Closing Comment

by:chipsch
ID: 36711609
Thanks again for the input.
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Short answer to this question: there is no effective WiFi manager in iOS devices as seen in Windows WiFi or Macbook OSx WiFi management, but this article will try and provide some amicable solutions to better suite your needs.
Moving your enterprise fax infrastructure from in-house fax machines and servers to the cloud makes sense — from both an efficiency and productivity standpoint. But does migrating to a cloud fax solution mean you will no longer be able to send or re…
Viewers will learn how to connect to a wireless network using the network security key. They will also learn how to access the IP address and DNS server for connections that must be done manually. After setting up a router, find the network security…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now