Solved

Ospf caused outage?

Posted on 2010-11-21
11
759 Views
Last Modified: 2012-05-10
We have a core switch, mpls ce router ( we are the customer) and a firewall that faces the internet.

All 3 devices run ospf. And all 3 devices are in the same subnet.  Tonight we rebooted the core switch. When the switch came back up, i noticed the router and firewall had no connectivity to the switch, but had connectivity to outside networks.  The switch seemed to be able to ping all local hosts, but not the firewall or router. The mpls routers' logs showed several messages stating " dead timer expired"

Even if ospf adjacency was lost, these devices should have still been able to speak since they are in the same subnet

So what happened"
0
Comment
Question by:orus
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 4
  • 2
  • +1
11 Comments
 
LVL 17

Expert Comment

by:rochey2009
ID: 34186936
hi,

please post

show ip int brief

show ip ospf int brief

from the switch
0
 
LVL 29

Accepted Solution

by:
Alan Huseyin Kayahan earned 333 total points
ID: 34188295
Assuming the router and firewall are directly connected to the switch, this appears to be a Spanning tree or duplex mismatch issue. According to best practises make sure the speed and duplex parameters are manually hardcoded at both switch and non-host devices. Dont know your STP topology, but the ports connected to router and firewall should be configured with one of portfast,uplinkfast or backbonefast.
0
 

Author Comment

by:orus
ID: 34188388
sh commands attached

Also, could this be STP? The connection from the switch to the router, already had portfast enabled. Also, during the downtime, the switchports to the router and firewall were "up up", not "err disabled"

thanks
c4507R_Core#sh ip ospf int brief
Interface    PID   Area            IP Address/Mask    Cost  State Nbrs F/C
Gi3/22       1     0               172.17.1.1/24      1     P2MP  3/3
Vl16         1     1               172.16.16.1/24     1     DR    0/0
Vl15         1     1               172.16.15.1/24     1     DR    0/0
Vl14         1     1               172.16.14.1/24     1     DR    0/0
Vl13         1     1               172.16.13.1/24     1     DR    0/0
Vl12         1     1               172.16.12.1/24     1     DR    0/0
Vl11         1     1               172.16.11.1/24     1     DR    0/0
Vl10         1     1               172.16.10.1/24     1     DR    0/0
Vl3          1     1               172.16.3.1/24      1     DR    0/0
Vl2          1     1               172.16.2.1/24      1     DR    0/0
Vl1          1     1               172.16.1.1/24      1     DR    3/3
c4507R_Core#

Open in new window

sh ip int brief
c4507R_Core#show ip int brief
Interface              IP-Address      OK? Method Status                Protocol
Vlan1                  172.16.1.1      YES NVRAM  up                    up
Vlan2                  172.16.2.1      YES NVRAM  up                    up
Vlan3                  172.16.3.1      YES NVRAM  up                    up
Vlan10                 172.16.10.1     YES NVRAM  up                    up
Vlan11                 172.16.11.1     YES NVRAM  up                    up
Vlan12                 172.16.12.1     YES NVRAM  up                    up
Vlan13                 172.16.13.1     YES NVRAM  up                    up
Vlan14                 172.16.14.1     YES NVRAM  up                    up
Vlan15                 172.16.15.1     YES NVRAM  up                    up
Vlan16                 172.16.16.1     YES NVRAM  up                    up
Vlan100                172.16.100.1    YES NVRAM  administratively down down
Vlan101                172.16.101.1    YES NVRAM  administratively down down
Vlan102                172.16.102.1    YES NVRAM  administratively down down
Vlan103                172.16.103.1    YES NVRAM  administratively down down
Vlan1010               unassigned      YES NVRAM  administratively down down
TenGigabitEthernet1/1  unassigned      YES unset  up                    up
TenGigabitEthernet1/2  unassigned      YES unset  up                    up
GigabitEthernet1/3     unassigned      YES unset  down                  down
GigabitEthernet1/4     unassigned      YES unset  down                  down
GigabitEthernet1/5     unassigned      YES unset  down                  down
GigabitEthernet1/6     unassigned      YES unset  down                  down
GigabitEthernet3/1     unassigned      YES unset  up                    up
GigabitEthernet3/2     unassigned      YES unset  up                    up
GigabitEthernet3/3     unassigned      YES unset  up                    up
GigabitEthernet3/4     unassigned      YES unset  up                    up
GigabitEthernet3/5     unassigned      YES unset  up                    up
GigabitEthernet3/6     unassigned      YES unset  up                    up
GigabitEthernet3/7     unassigned      YES unset  up                    up
GigabitEthernet3/8     unassigned      YES unset  up                    up
GigabitEthernet3/9     unassigned      YES unset  up                    up
GigabitEthernet3/10    unassigned      YES unset  up                    up
GigabitEthernet3/11    unassigned      YES unset  up                    up
GigabitEthernet3/12    unassigned      YES unset  up                    up
GigabitEthernet3/13    unassigned      YES unset  up                    up
GigabitEthernet3/14    unassigned      YES unset  up                    up
GigabitEthernet3/15    unassigned      YES unset  up                    up
GigabitEthernet3/16    unassigned      YES unset  up                    up
GigabitEthernet3/17    unassigned      YES unset  up                    up
GigabitEthernet3/18    unassigned      YES unset  down                  down
GigabitEthernet3/19    unassigned      YES unset  up                    up
GigabitEthernet3/20    unassigned      YES unset  up                    up
GigabitEthernet3/21    unassigned      YES unset  up                    up
GigabitEthernet3/22    172.17.1.1      YES NVRAM  up                    up
GigabitEthernet3/23    unassigned      YES unset  up                    up
GigabitEthernet3/24    unassigned      YES unset  up                    up
GigabitEthernet4/1     unassigned      YES unset  up                    up
GigabitEthernet4/2     unassigned      YES unset  up                    up
GigabitEthernet4/3     unassigned      YES unset  up                    up
GigabitEthernet4/4     unassigned      YES unset  up                    up
GigabitEthernet4/5     unassigned      YES unset  up                    up
GigabitEthernet4/6     unassigned      YES unset  up                    up
GigabitEthernet4/7     unassigned      YES unset  up                    up
GigabitEthernet4/8     unassigned      YES unset  up                    up
GigabitEthernet4/9     unassigned      YES unset  up                    up
GigabitEthernet4/10    unassigned      YES unset  up                    up
GigabitEthernet4/11    unassigned      YES unset  up                    up
GigabitEthernet4/12    unassigned      YES unset  up                    up
GigabitEthernet4/13    unassigned      YES unset  up                    up
GigabitEthernet4/14    unassigned      YES unset  up                    up
GigabitEthernet4/15    unassigned      YES unset  up                    up
GigabitEthernet4/16    unassigned      YES unset  up                    up
GigabitEthernet4/17    unassigned      YES unset  up                    up
GigabitEthernet4/18    unassigned      YES unset  up                    up
GigabitEthernet4/19    unassigned      YES unset  up                    up
GigabitEthernet4/20    unassigned      YES unset  up                    up
GigabitEthernet4/21    unassigned      YES unset  up                    up
GigabitEthernet4/22    unassigned      YES unset  up                    up
GigabitEthernet4/23    unassigned      YES unset  up                    up
GigabitEthernet4/24    unassigned      YES unset  up                    up
GigabitEthernet5/1     unassigned      YES unset  down                  down
GigabitEthernet5/2     unassigned      YES unset  up                    up
GigabitEthernet5/3     unassigned      YES unset  up                    up
GigabitEthernet5/4     unassigned      YES unset  up                    up
GigabitEthernet5/5     unassigned      YES unset  up                    up
GigabitEthernet5/6     unassigned      YES unset  up                    up
GigabitEthernet5/7     unassigned      YES unset  up                    up
GigabitEthernet5/8     unassigned      YES unset  up                    up
GigabitEthernet5/9     unassigned      YES unset  up                    up
GigabitEthernet5/10    unassigned      YES unset  up                    up
GigabitEthernet5/11    unassigned      YES unset  up                    up
GigabitEthernet5/12    unassigned      YES unset  up                    up
GigabitEthernet5/13    unassigned      YES unset  down                  down
GigabitEthernet5/14    unassigned      YES unset  up                    up
GigabitEthernet5/15    unassigned      YES unset  administratively down down
GigabitEthernet5/16    unassigned      YES unset  up                    up
GigabitEthernet5/17    unassigned      YES unset  up                    up
GigabitEthernet5/18    unassigned      YES unset  down                  down
GigabitEthernet5/19    unassigned      YES unset  down                  down
GigabitEthernet5/20    unassigned      YES unset  up                    up
GigabitEthernet5/21    unassigned      YES unset  up                    up
GigabitEthernet5/22    unassigned      YES unset  down                  down
GigabitEthernet5/23    unassigned      YES unset  up                    up
GigabitEthernet5/24    unassigned      YES unset  up                    up
GigabitEthernet5/25    unassigned      YES unset  up                    up
GigabitEthernet5/26    unassigned      YES unset  up                    up
GigabitEthernet5/27    unassigned      YES unset  up                    up
GigabitEthernet5/28    unassigned      YES unset  up                    up
GigabitEthernet5/29    unassigned      YES unset  up                    up
GigabitEthernet5/30    unassigned      YES unset  up                    up
GigabitEthernet5/31    unassigned      YES unset  up                    up
GigabitEthernet5/32    unassigned      YES unset  up                    up
GigabitEthernet5/33    unassigned      YES unset  up                    up
GigabitEthernet5/34    unassigned      YES unset  up                    up
GigabitEthernet5/35    unassigned      YES unset  up                    up
GigabitEthernet5/36    unassigned      YES unset  up                    up
GigabitEthernet5/37    unassigned      YES unset  up                    up
GigabitEthernet5/38    unassigned      YES unset  up                    up
GigabitEthernet5/39    unassigned      YES unset  up                    up
GigabitEthernet5/40    unassigned      YES unset  down                  down
GigabitEthernet5/41    unassigned      YES unset  up                    up
GigabitEthernet5/42    unassigned      YES unset  up                    up
GigabitEthernet5/43    unassigned      YES unset  up                    up
GigabitEthernet5/44    unassigned      YES unset  up                    up
GigabitEthernet5/45    unassigned      YES unset  up                    up
GigabitEthernet5/46    unassigned      YES unset  up                    up
GigabitEthernet5/47    unassigned      YES unset  up                    up
GigabitEthernet5/48    unassigned      YES unset  up                    up
GigabitEthernet6/1     unassigned      YES unset  up                    up
GigabitEthernet6/2     unassigned      YES unset  up                    up
GigabitEthernet6/3     unassigned      YES unset  up                    up
GigabitEthernet6/4     unassigned      YES unset  up                    up
GigabitEthernet6/5     unassigned      YES unset  up                    up
GigabitEthernet6/6     unassigned      YES unset  up                    up
GigabitEthernet6/7     unassigned      YES unset  up                    up
GigabitEthernet6/8     unassigned      YES unset  up                    up
GigabitEthernet6/9     unassigned      YES unset  up                    up
GigabitEthernet6/10    unassigned      YES unset  up                    up
GigabitEthernet6/11    unassigned      YES unset  up                    up
GigabitEthernet6/12    unassigned      YES unset  up                    up
GigabitEthernet6/13    unassigned      YES unset  up                    up
GigabitEthernet6/14    unassigned      YES unset  down                  down
GigabitEthernet6/15    unassigned      YES unset  up                    up
GigabitEthernet6/16    unassigned      YES unset  up                    up
GigabitEthernet6/17    unassigned      YES unset  up                    up
GigabitEthernet6/18    unassigned      YES unset  up                    up
GigabitEthernet6/19    unassigned      YES unset  up                    up
GigabitEthernet6/20    unassigned      YES unset  up                    up
GigabitEthernet6/21    unassigned      YES unset  up                    up
GigabitEthernet6/22    unassigned      YES unset  up                    up
GigabitEthernet6/23    unassigned      YES unset  up                    up
GigabitEthernet6/24    unassigned      YES unset  down                  down
GigabitEthernet6/25    unassigned      YES unset  up                    up
GigabitEthernet6/26    unassigned      YES unset  up                    up
GigabitEthernet6/27    unassigned      YES unset  up                    up
GigabitEthernet6/28    unassigned      YES unset  down                  down
GigabitEthernet6/29    unassigned      YES unset  up                    up
GigabitEthernet6/30    unassigned      YES unset  up                    up
GigabitEthernet6/31    unassigned      YES unset  up                    up
GigabitEthernet6/32    unassigned      YES unset  down                  down
GigabitEthernet6/33    unassigned      YES unset  up                    up
GigabitEthernet6/34    unassigned      YES unset  up                    up
GigabitEthernet6/35    unassigned      YES unset  up                    up
GigabitEthernet6/36    unassigned      YES unset  up                    up
GigabitEthernet6/37    unassigned      YES unset  up                    up
GigabitEthernet6/38    unassigned      YES unset  down                  down
GigabitEthernet6/39    unassigned      YES unset  down                  down
GigabitEthernet6/40    unassigned      YES unset  down                  down
GigabitEthernet6/41    unassigned      YES unset  up                    up
GigabitEthernet6/42    unassigned      YES unset  down                  down
GigabitEthernet6/43    unassigned      YES unset  up                    up
GigabitEthernet6/44    unassigned      YES unset  down                  down
GigabitEthernet6/45    unassigned      YES unset  up                    up
GigabitEthernet6/46    unassigned      YES unset  down                  down
GigabitEthernet6/47    unassigned      YES unset  down                  down
GigabitEthernet6/48    unassigned      YES unset  up                    up
GigabitEthernet7/1     unassigned      YES unset  down                  down
GigabitEthernet7/2     unassigned      YES unset  up                    up
GigabitEthernet7/3     unassigned      YES unset  down                  down
GigabitEthernet7/4     unassigned      YES unset  down                  down
GigabitEthernet7/5     unassigned      YES unset  administratively down down
GigabitEthernet7/6     unassigned      YES unset  down                  down
GigabitEthernet7/7     unassigned      YES unset  down                  down
GigabitEthernet7/8     unassigned      YES unset  administratively down down
GigabitEthernet7/9     unassigned      YES unset  down                  down
GigabitEthernet7/10    unassigned      YES unset  up                    up
GigabitEthernet7/11    unassigned      YES unset  up                    up
GigabitEthernet7/12    unassigned      YES unset  down                  down
GigabitEthernet7/13    unassigned      YES unset  down                  down
GigabitEthernet7/14    unassigned      YES unset  down                  down
GigabitEthernet7/15    unassigned      YES unset  up                    up
GigabitEthernet7/16    unassigned      YES unset  up                    up
GigabitEthernet7/17    unassigned      YES unset  up                    up
GigabitEthernet7/18    unassigned      YES unset  down                  down
GigabitEthernet7/19    unassigned      YES unset  down                  down
GigabitEthernet7/20    unassigned      YES unset  down                  down
GigabitEthernet7/21    unassigned      YES unset  down                  down
GigabitEthernet7/22    unassigned      YES unset  down                  down
GigabitEthernet7/23    unassigned      YES unset  administratively down down
GigabitEthernet7/24    unassigned      YES unset  down                  down
GigabitEthernet7/25    unassigned      YES unset  down                  down
GigabitEthernet7/26    unassigned      YES unset  down                  down
GigabitEthernet7/27    unassigned      YES unset  up                    up
GigabitEthernet7/28    unassigned      YES unset  down                  down
GigabitEthernet7/29    unassigned      YES unset  up                    up
GigabitEthernet7/30    unassigned      YES unset  up                    up
GigabitEthernet7/31    unassigned      YES unset  down                  down
GigabitEthernet7/32    unassigned      YES unset  down                  down
GigabitEthernet7/33    unassigned      YES unset  down                  down
GigabitEthernet7/34    unassigned      YES unset  down                  down
GigabitEthernet7/35    unassigned      YES unset  down                  down
GigabitEthernet7/36    unassigned      YES unset  up                    up
GigabitEthernet7/37    unassigned      YES unset  up                    up
GigabitEthernet7/38    unassigned      YES unset  up                    up
GigabitEthernet7/39    unassigned      YES unset  up                    up
GigabitEthernet7/40    unassigned      YES unset  down                  down
GigabitEthernet7/41    unassigned      YES unset  down                  down
GigabitEthernet7/42    unassigned      YES unset  down                  down
GigabitEthernet7/43    unassigned      YES unset  down                  down
GigabitEthernet7/44    unassigned      YES unset  down                  down
GigabitEthernet7/45    unassigned      YES unset  down                  down
GigabitEthernet7/46    unassigned      YES unset  down                  down
GigabitEthernet7/47    unassigned      YES unset  up                    up
GigabitEthernet7/48    unassigned      YES unset  up                    up
Port-channel1          unassigned      YES NVRAM  down                  down
Port-channel3          unassigned      YES unset  up                    up
Port-channel4          unassigned      YES unset  up                    up
Port-channel5          unassigned      YES unset  up                    up
Port-channel16         unassigned      YES NVRAM  down                  down
Port-channel17         unassigned      YES unset  up                    up
Port-channel18         unassigned      YES NVRAM  down                  down
c4507R_Core#

Open in new window

0
NEW Veeam Agent for Microsoft Windows

Backup and recover physical and cloud-based servers and workstations, as well as endpoint devices that belong to remote users. Avoid downtime and data loss quickly and easily for Windows-based physical or public cloud-based workloads!

 
LVL 29

Expert Comment

by:Alan Huseyin Kayahan
ID: 34188639
You better focus on "show spanning-tree" outputs rather than OSPF or interface outputs. Also if possible, set port mirroring for the ports that you had problems (switch-firewall and switch-router) and listen the conversation. That would of course be usefull if you can regenerate the problem
0
 

Author Comment

by:orus
ID: 34188810
There is nothing indicating an stp issue, looping or flapping
What exactly should i be looking at?  Any assistance appreciated
0
 
LVL 29

Expert Comment

by:Alan Huseyin Kayahan
ID: 34189074
A question,
   Does the problem still persist? Because if not, if it is resolved and everything is working, there is no command that can determine the cause.
Btw is this a Gbic of ethernet port?
0
 

Author Comment

by:orus
ID: 34189135
well I looked at the logs when the issue was going on. I saw nothing showing STP or looping etc. It happened last night. I want to make sure it is fully operational and doesnt happen again.

We use tengig fiber modules to connect to our access switches.  We switched from copper to tengig fiber last night. I verified the copper ports are in a blocking state on the access switches
0
 
LVL 17

Expert Comment

by:rochey2009
ID: 34189438
what is the state of the interfaces going to the firewall and the router?

show interface x
0
 
LVL 29

Assisted Solution

by:Alan Huseyin Kayahan
Alan Huseyin Kayahan earned 333 total points
ID: 34189661
For an effective solution, you should regenerate the problem in a controlled and scheduled environment. If you plan to do so, set the clocks of testing host, syslog server and log generating devices correctly. Also mirror the port and listen the conversation from/to specific port with wireshark, again synchronized clock.
So once you launch ping we can see the responses from devices at that time and arrive at a conclusion.

Are there any HSRP or equivalant load balancing configs in place?

Taking the situation into consideration, which is "temporary disconnectivity or (late connectivity) between directly connected nodes, right after switch restart", the answer is usually

Load balancing protocol negotiations>STP issues> Port security issues>Duplex and speed negotiations (epecially between OEM non-standard or faulty gbics/ports>Cabling

In some cases, leaving the speed at auto-sense in such important ports may create similar outcomes.

If i were you and did not have the chance to regenerate the issue, I would
1)Manually enter the speed and duplex of the ports
2)Statically enter the MAC-IP-PORT binding into switch
0
 

Author Comment

by:orus
ID: 34191007
No load balancing. I can try and recreate in our lab.  

In the router: I don't see anything indicating a duplex mismatch in any of the buffers either, no input errors or CRCs on the interface.  Everything looks clean. The only errors I saw were "Ospf neighbor down: dead timer expired"

If I go into the switch, and look at the switchport where the router plugs in, I see 858 runts, 1925 input errors and 1067 CRCs. But the counters have never been cleared on this interface. Do they clear during a reboot?

All of this resolved once I did a clean reload of the router and the issue never came back. Could this be due to OSPF or do you still think it is possibly a duplex or STP issue? I figured I would see those logs indicating that in the buffer though...
0
 
LVL 9

Assisted Solution

by:DanJ
DanJ earned 167 total points
ID: 34199440
"The switch seemed to be able to ping all local hosts, but not the firewall or router."
You lost layer 3 connectivity between these devices. OSPF runs on top of IP. No IP reachability means OSPF adjacencies are lost.

0

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Data center, now-a-days, is referred as the home of all the advanced technologies. In-fact, most of the businesses are now establishing their entire organizational structure around the IT capabilities.
For months I had no idea how to 'discover' the IP address of the other end of a link (without asking someone who knows), and it drove me batty. Think about it. You can't use Cisco Discovery Protocol (CDP) because it's not implemented on the ASAs.…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
Both in life and business – not all partnerships are created equal. As the demand for cloud services increases, so do the number of self-proclaimed cloud partners. Asking the right questions up front in the partnership, will enable both parties …

738 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question