Solved

Ospf caused outage?

Posted on 2010-11-21
11
728 Views
Last Modified: 2012-05-10
We have a core switch, mpls ce router ( we are the customer) and a firewall that faces the internet.

All 3 devices run ospf. And all 3 devices are in the same subnet.  Tonight we rebooted the core switch. When the switch came back up, i noticed the router and firewall had no connectivity to the switch, but had connectivity to outside networks.  The switch seemed to be able to ping all local hosts, but not the firewall or router. The mpls routers' logs showed several messages stating " dead timer expired"

Even if ospf adjacency was lost, these devices should have still been able to speak since they are in the same subnet

So what happened"
0
Comment
Question by:orus
  • 4
  • 4
  • 2
  • +1
11 Comments
 
LVL 17

Expert Comment

by:rochey2009
Comment Utility
hi,

please post

show ip int brief

show ip ospf int brief

from the switch
0
 
LVL 29

Accepted Solution

by:
Alan Huseyin Kayahan earned 333 total points
Comment Utility
Assuming the router and firewall are directly connected to the switch, this appears to be a Spanning tree or duplex mismatch issue. According to best practises make sure the speed and duplex parameters are manually hardcoded at both switch and non-host devices. Dont know your STP topology, but the ports connected to router and firewall should be configured with one of portfast,uplinkfast or backbonefast.
0
 

Author Comment

by:orus
Comment Utility
sh commands attached

Also, could this be STP? The connection from the switch to the router, already had portfast enabled. Also, during the downtime, the switchports to the router and firewall were "up up", not "err disabled"

thanks
c4507R_Core#sh ip ospf int brief

Interface    PID   Area            IP Address/Mask    Cost  State Nbrs F/C

Gi3/22       1     0               172.17.1.1/24      1     P2MP  3/3

Vl16         1     1               172.16.16.1/24     1     DR    0/0

Vl15         1     1               172.16.15.1/24     1     DR    0/0

Vl14         1     1               172.16.14.1/24     1     DR    0/0

Vl13         1     1               172.16.13.1/24     1     DR    0/0

Vl12         1     1               172.16.12.1/24     1     DR    0/0

Vl11         1     1               172.16.11.1/24     1     DR    0/0

Vl10         1     1               172.16.10.1/24     1     DR    0/0

Vl3          1     1               172.16.3.1/24      1     DR    0/0

Vl2          1     1               172.16.2.1/24      1     DR    0/0

Vl1          1     1               172.16.1.1/24      1     DR    3/3

c4507R_Core#

Open in new window

sh ip int brief

c4507R_Core#show ip int brief

Interface              IP-Address      OK? Method Status                Protocol

Vlan1                  172.16.1.1      YES NVRAM  up                    up

Vlan2                  172.16.2.1      YES NVRAM  up                    up

Vlan3                  172.16.3.1      YES NVRAM  up                    up

Vlan10                 172.16.10.1     YES NVRAM  up                    up

Vlan11                 172.16.11.1     YES NVRAM  up                    up

Vlan12                 172.16.12.1     YES NVRAM  up                    up

Vlan13                 172.16.13.1     YES NVRAM  up                    up

Vlan14                 172.16.14.1     YES NVRAM  up                    up

Vlan15                 172.16.15.1     YES NVRAM  up                    up

Vlan16                 172.16.16.1     YES NVRAM  up                    up

Vlan100                172.16.100.1    YES NVRAM  administratively down down

Vlan101                172.16.101.1    YES NVRAM  administratively down down

Vlan102                172.16.102.1    YES NVRAM  administratively down down

Vlan103                172.16.103.1    YES NVRAM  administratively down down

Vlan1010               unassigned      YES NVRAM  administratively down down

TenGigabitEthernet1/1  unassigned      YES unset  up                    up

TenGigabitEthernet1/2  unassigned      YES unset  up                    up

GigabitEthernet1/3     unassigned      YES unset  down                  down

GigabitEthernet1/4     unassigned      YES unset  down                  down

GigabitEthernet1/5     unassigned      YES unset  down                  down

GigabitEthernet1/6     unassigned      YES unset  down                  down

GigabitEthernet3/1     unassigned      YES unset  up                    up

GigabitEthernet3/2     unassigned      YES unset  up                    up

GigabitEthernet3/3     unassigned      YES unset  up                    up

GigabitEthernet3/4     unassigned      YES unset  up                    up

GigabitEthernet3/5     unassigned      YES unset  up                    up

GigabitEthernet3/6     unassigned      YES unset  up                    up

GigabitEthernet3/7     unassigned      YES unset  up                    up

GigabitEthernet3/8     unassigned      YES unset  up                    up

GigabitEthernet3/9     unassigned      YES unset  up                    up

GigabitEthernet3/10    unassigned      YES unset  up                    up

GigabitEthernet3/11    unassigned      YES unset  up                    up

GigabitEthernet3/12    unassigned      YES unset  up                    up

GigabitEthernet3/13    unassigned      YES unset  up                    up

GigabitEthernet3/14    unassigned      YES unset  up                    up

GigabitEthernet3/15    unassigned      YES unset  up                    up

GigabitEthernet3/16    unassigned      YES unset  up                    up

GigabitEthernet3/17    unassigned      YES unset  up                    up

GigabitEthernet3/18    unassigned      YES unset  down                  down

GigabitEthernet3/19    unassigned      YES unset  up                    up

GigabitEthernet3/20    unassigned      YES unset  up                    up

GigabitEthernet3/21    unassigned      YES unset  up                    up

GigabitEthernet3/22    172.17.1.1      YES NVRAM  up                    up

GigabitEthernet3/23    unassigned      YES unset  up                    up

GigabitEthernet3/24    unassigned      YES unset  up                    up

GigabitEthernet4/1     unassigned      YES unset  up                    up

GigabitEthernet4/2     unassigned      YES unset  up                    up

GigabitEthernet4/3     unassigned      YES unset  up                    up

GigabitEthernet4/4     unassigned      YES unset  up                    up

GigabitEthernet4/5     unassigned      YES unset  up                    up

GigabitEthernet4/6     unassigned      YES unset  up                    up

GigabitEthernet4/7     unassigned      YES unset  up                    up

GigabitEthernet4/8     unassigned      YES unset  up                    up

GigabitEthernet4/9     unassigned      YES unset  up                    up

GigabitEthernet4/10    unassigned      YES unset  up                    up

GigabitEthernet4/11    unassigned      YES unset  up                    up

GigabitEthernet4/12    unassigned      YES unset  up                    up

GigabitEthernet4/13    unassigned      YES unset  up                    up

GigabitEthernet4/14    unassigned      YES unset  up                    up

GigabitEthernet4/15    unassigned      YES unset  up                    up

GigabitEthernet4/16    unassigned      YES unset  up                    up

GigabitEthernet4/17    unassigned      YES unset  up                    up

GigabitEthernet4/18    unassigned      YES unset  up                    up

GigabitEthernet4/19    unassigned      YES unset  up                    up

GigabitEthernet4/20    unassigned      YES unset  up                    up

GigabitEthernet4/21    unassigned      YES unset  up                    up

GigabitEthernet4/22    unassigned      YES unset  up                    up

GigabitEthernet4/23    unassigned      YES unset  up                    up

GigabitEthernet4/24    unassigned      YES unset  up                    up

GigabitEthernet5/1     unassigned      YES unset  down                  down

GigabitEthernet5/2     unassigned      YES unset  up                    up

GigabitEthernet5/3     unassigned      YES unset  up                    up

GigabitEthernet5/4     unassigned      YES unset  up                    up

GigabitEthernet5/5     unassigned      YES unset  up                    up

GigabitEthernet5/6     unassigned      YES unset  up                    up

GigabitEthernet5/7     unassigned      YES unset  up                    up

GigabitEthernet5/8     unassigned      YES unset  up                    up

GigabitEthernet5/9     unassigned      YES unset  up                    up

GigabitEthernet5/10    unassigned      YES unset  up                    up

GigabitEthernet5/11    unassigned      YES unset  up                    up

GigabitEthernet5/12    unassigned      YES unset  up                    up

GigabitEthernet5/13    unassigned      YES unset  down                  down

GigabitEthernet5/14    unassigned      YES unset  up                    up

GigabitEthernet5/15    unassigned      YES unset  administratively down down

GigabitEthernet5/16    unassigned      YES unset  up                    up

GigabitEthernet5/17    unassigned      YES unset  up                    up

GigabitEthernet5/18    unassigned      YES unset  down                  down

GigabitEthernet5/19    unassigned      YES unset  down                  down

GigabitEthernet5/20    unassigned      YES unset  up                    up

GigabitEthernet5/21    unassigned      YES unset  up                    up

GigabitEthernet5/22    unassigned      YES unset  down                  down

GigabitEthernet5/23    unassigned      YES unset  up                    up

GigabitEthernet5/24    unassigned      YES unset  up                    up

GigabitEthernet5/25    unassigned      YES unset  up                    up

GigabitEthernet5/26    unassigned      YES unset  up                    up

GigabitEthernet5/27    unassigned      YES unset  up                    up

GigabitEthernet5/28    unassigned      YES unset  up                    up

GigabitEthernet5/29    unassigned      YES unset  up                    up

GigabitEthernet5/30    unassigned      YES unset  up                    up

GigabitEthernet5/31    unassigned      YES unset  up                    up

GigabitEthernet5/32    unassigned      YES unset  up                    up

GigabitEthernet5/33    unassigned      YES unset  up                    up

GigabitEthernet5/34    unassigned      YES unset  up                    up

GigabitEthernet5/35    unassigned      YES unset  up                    up

GigabitEthernet5/36    unassigned      YES unset  up                    up

GigabitEthernet5/37    unassigned      YES unset  up                    up

GigabitEthernet5/38    unassigned      YES unset  up                    up

GigabitEthernet5/39    unassigned      YES unset  up                    up

GigabitEthernet5/40    unassigned      YES unset  down                  down

GigabitEthernet5/41    unassigned      YES unset  up                    up

GigabitEthernet5/42    unassigned      YES unset  up                    up

GigabitEthernet5/43    unassigned      YES unset  up                    up

GigabitEthernet5/44    unassigned      YES unset  up                    up

GigabitEthernet5/45    unassigned      YES unset  up                    up

GigabitEthernet5/46    unassigned      YES unset  up                    up

GigabitEthernet5/47    unassigned      YES unset  up                    up

GigabitEthernet5/48    unassigned      YES unset  up                    up

GigabitEthernet6/1     unassigned      YES unset  up                    up

GigabitEthernet6/2     unassigned      YES unset  up                    up

GigabitEthernet6/3     unassigned      YES unset  up                    up

GigabitEthernet6/4     unassigned      YES unset  up                    up

GigabitEthernet6/5     unassigned      YES unset  up                    up

GigabitEthernet6/6     unassigned      YES unset  up                    up

GigabitEthernet6/7     unassigned      YES unset  up                    up

GigabitEthernet6/8     unassigned      YES unset  up                    up

GigabitEthernet6/9     unassigned      YES unset  up                    up

GigabitEthernet6/10    unassigned      YES unset  up                    up

GigabitEthernet6/11    unassigned      YES unset  up                    up

GigabitEthernet6/12    unassigned      YES unset  up                    up

GigabitEthernet6/13    unassigned      YES unset  up                    up

GigabitEthernet6/14    unassigned      YES unset  down                  down

GigabitEthernet6/15    unassigned      YES unset  up                    up

GigabitEthernet6/16    unassigned      YES unset  up                    up

GigabitEthernet6/17    unassigned      YES unset  up                    up

GigabitEthernet6/18    unassigned      YES unset  up                    up

GigabitEthernet6/19    unassigned      YES unset  up                    up

GigabitEthernet6/20    unassigned      YES unset  up                    up

GigabitEthernet6/21    unassigned      YES unset  up                    up

GigabitEthernet6/22    unassigned      YES unset  up                    up

GigabitEthernet6/23    unassigned      YES unset  up                    up

GigabitEthernet6/24    unassigned      YES unset  down                  down

GigabitEthernet6/25    unassigned      YES unset  up                    up

GigabitEthernet6/26    unassigned      YES unset  up                    up

GigabitEthernet6/27    unassigned      YES unset  up                    up

GigabitEthernet6/28    unassigned      YES unset  down                  down

GigabitEthernet6/29    unassigned      YES unset  up                    up

GigabitEthernet6/30    unassigned      YES unset  up                    up

GigabitEthernet6/31    unassigned      YES unset  up                    up

GigabitEthernet6/32    unassigned      YES unset  down                  down

GigabitEthernet6/33    unassigned      YES unset  up                    up

GigabitEthernet6/34    unassigned      YES unset  up                    up

GigabitEthernet6/35    unassigned      YES unset  up                    up

GigabitEthernet6/36    unassigned      YES unset  up                    up

GigabitEthernet6/37    unassigned      YES unset  up                    up

GigabitEthernet6/38    unassigned      YES unset  down                  down

GigabitEthernet6/39    unassigned      YES unset  down                  down

GigabitEthernet6/40    unassigned      YES unset  down                  down

GigabitEthernet6/41    unassigned      YES unset  up                    up

GigabitEthernet6/42    unassigned      YES unset  down                  down

GigabitEthernet6/43    unassigned      YES unset  up                    up

GigabitEthernet6/44    unassigned      YES unset  down                  down

GigabitEthernet6/45    unassigned      YES unset  up                    up

GigabitEthernet6/46    unassigned      YES unset  down                  down

GigabitEthernet6/47    unassigned      YES unset  down                  down

GigabitEthernet6/48    unassigned      YES unset  up                    up

GigabitEthernet7/1     unassigned      YES unset  down                  down

GigabitEthernet7/2     unassigned      YES unset  up                    up

GigabitEthernet7/3     unassigned      YES unset  down                  down

GigabitEthernet7/4     unassigned      YES unset  down                  down

GigabitEthernet7/5     unassigned      YES unset  administratively down down

GigabitEthernet7/6     unassigned      YES unset  down                  down

GigabitEthernet7/7     unassigned      YES unset  down                  down

GigabitEthernet7/8     unassigned      YES unset  administratively down down

GigabitEthernet7/9     unassigned      YES unset  down                  down

GigabitEthernet7/10    unassigned      YES unset  up                    up

GigabitEthernet7/11    unassigned      YES unset  up                    up

GigabitEthernet7/12    unassigned      YES unset  down                  down

GigabitEthernet7/13    unassigned      YES unset  down                  down

GigabitEthernet7/14    unassigned      YES unset  down                  down

GigabitEthernet7/15    unassigned      YES unset  up                    up

GigabitEthernet7/16    unassigned      YES unset  up                    up

GigabitEthernet7/17    unassigned      YES unset  up                    up

GigabitEthernet7/18    unassigned      YES unset  down                  down

GigabitEthernet7/19    unassigned      YES unset  down                  down

GigabitEthernet7/20    unassigned      YES unset  down                  down

GigabitEthernet7/21    unassigned      YES unset  down                  down

GigabitEthernet7/22    unassigned      YES unset  down                  down

GigabitEthernet7/23    unassigned      YES unset  administratively down down

GigabitEthernet7/24    unassigned      YES unset  down                  down

GigabitEthernet7/25    unassigned      YES unset  down                  down

GigabitEthernet7/26    unassigned      YES unset  down                  down

GigabitEthernet7/27    unassigned      YES unset  up                    up

GigabitEthernet7/28    unassigned      YES unset  down                  down

GigabitEthernet7/29    unassigned      YES unset  up                    up

GigabitEthernet7/30    unassigned      YES unset  up                    up

GigabitEthernet7/31    unassigned      YES unset  down                  down

GigabitEthernet7/32    unassigned      YES unset  down                  down

GigabitEthernet7/33    unassigned      YES unset  down                  down

GigabitEthernet7/34    unassigned      YES unset  down                  down

GigabitEthernet7/35    unassigned      YES unset  down                  down

GigabitEthernet7/36    unassigned      YES unset  up                    up

GigabitEthernet7/37    unassigned      YES unset  up                    up

GigabitEthernet7/38    unassigned      YES unset  up                    up

GigabitEthernet7/39    unassigned      YES unset  up                    up

GigabitEthernet7/40    unassigned      YES unset  down                  down

GigabitEthernet7/41    unassigned      YES unset  down                  down

GigabitEthernet7/42    unassigned      YES unset  down                  down

GigabitEthernet7/43    unassigned      YES unset  down                  down

GigabitEthernet7/44    unassigned      YES unset  down                  down

GigabitEthernet7/45    unassigned      YES unset  down                  down

GigabitEthernet7/46    unassigned      YES unset  down                  down

GigabitEthernet7/47    unassigned      YES unset  up                    up

GigabitEthernet7/48    unassigned      YES unset  up                    up

Port-channel1          unassigned      YES NVRAM  down                  down

Port-channel3          unassigned      YES unset  up                    up

Port-channel4          unassigned      YES unset  up                    up

Port-channel5          unassigned      YES unset  up                    up

Port-channel16         unassigned      YES NVRAM  down                  down

Port-channel17         unassigned      YES unset  up                    up

Port-channel18         unassigned      YES NVRAM  down                  down

c4507R_Core#

Open in new window

0
 
LVL 29

Expert Comment

by:Alan Huseyin Kayahan
Comment Utility
You better focus on "show spanning-tree" outputs rather than OSPF or interface outputs. Also if possible, set port mirroring for the ports that you had problems (switch-firewall and switch-router) and listen the conversation. That would of course be usefull if you can regenerate the problem
0
 

Author Comment

by:orus
Comment Utility
There is nothing indicating an stp issue, looping or flapping
What exactly should i be looking at?  Any assistance appreciated
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 29

Expert Comment

by:Alan Huseyin Kayahan
Comment Utility
A question,
   Does the problem still persist? Because if not, if it is resolved and everything is working, there is no command that can determine the cause.
Btw is this a Gbic of ethernet port?
0
 

Author Comment

by:orus
Comment Utility
well I looked at the logs when the issue was going on. I saw nothing showing STP or looping etc. It happened last night. I want to make sure it is fully operational and doesnt happen again.

We use tengig fiber modules to connect to our access switches.  We switched from copper to tengig fiber last night. I verified the copper ports are in a blocking state on the access switches
0
 
LVL 17

Expert Comment

by:rochey2009
Comment Utility
what is the state of the interfaces going to the firewall and the router?

show interface x
0
 
LVL 29

Assisted Solution

by:Alan Huseyin Kayahan
Alan Huseyin Kayahan earned 333 total points
Comment Utility
For an effective solution, you should regenerate the problem in a controlled and scheduled environment. If you plan to do so, set the clocks of testing host, syslog server and log generating devices correctly. Also mirror the port and listen the conversation from/to specific port with wireshark, again synchronized clock.
So once you launch ping we can see the responses from devices at that time and arrive at a conclusion.

Are there any HSRP or equivalant load balancing configs in place?

Taking the situation into consideration, which is "temporary disconnectivity or (late connectivity) between directly connected nodes, right after switch restart", the answer is usually

Load balancing protocol negotiations>STP issues> Port security issues>Duplex and speed negotiations (epecially between OEM non-standard or faulty gbics/ports>Cabling

In some cases, leaving the speed at auto-sense in such important ports may create similar outcomes.

If i were you and did not have the chance to regenerate the issue, I would
1)Manually enter the speed and duplex of the ports
2)Statically enter the MAC-IP-PORT binding into switch
0
 

Author Comment

by:orus
Comment Utility
No load balancing. I can try and recreate in our lab.  

In the router: I don't see anything indicating a duplex mismatch in any of the buffers either, no input errors or CRCs on the interface.  Everything looks clean. The only errors I saw were "Ospf neighbor down: dead timer expired"

If I go into the switch, and look at the switchport where the router plugs in, I see 858 runts, 1925 input errors and 1067 CRCs. But the counters have never been cleared on this interface. Do they clear during a reboot?

All of this resolved once I did a clean reload of the router and the issue never came back. Could this be due to OSPF or do you still think it is possibly a duplex or STP issue? I figured I would see those logs indicating that in the buffer though...
0
 
LVL 9

Assisted Solution

by:DanJ
DanJ earned 167 total points
Comment Utility
"The switch seemed to be able to ping all local hosts, but not the firewall or router."
You lost layer 3 connectivity between these devices. OSPF runs on top of IP. No IP reachability means OSPF adjacencies are lost.

0

Featured Post

Zoho SalesIQ

Hassle-free live chat software re-imagined for business growth. 2 users, always free.

Join & Write a Comment

There are two basic ways to configure a static route for Cisco IOS devices. I've written this article to highlight a case study comparing the configuration of a static route using the next-hop IP and the configuration of a static route using an outg…
Hello to you all, I hear of many people congratulate AWS (Amazon Web Services) on how easy it is to spin up and create new EC2 (Elastic Compute Cloud) instances, but then fail and struggle to connect to them using simple tools such as SSH (Secure…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

728 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now