• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 801
  • Last Modified:

Ospf caused outage?

We have a core switch, mpls ce router ( we are the customer) and a firewall that faces the internet.

All 3 devices run ospf. And all 3 devices are in the same subnet.  Tonight we rebooted the core switch. When the switch came back up, i noticed the router and firewall had no connectivity to the switch, but had connectivity to outside networks.  The switch seemed to be able to ping all local hosts, but not the firewall or router. The mpls routers' logs showed several messages stating " dead timer expired"

Even if ospf adjacency was lost, these devices should have still been able to speak since they are in the same subnet

So what happened"
0
orus
Asked:
orus
  • 4
  • 4
  • 2
  • +1
3 Solutions
 
rochey2009Commented:
hi,

please post

show ip int brief

show ip ospf int brief

from the switch
0
 
Alan Huseyin KayahanCommented:
Assuming the router and firewall are directly connected to the switch, this appears to be a Spanning tree or duplex mismatch issue. According to best practises make sure the speed and duplex parameters are manually hardcoded at both switch and non-host devices. Dont know your STP topology, but the ports connected to router and firewall should be configured with one of portfast,uplinkfast or backbonefast.
0
 
orusAuthor Commented:
sh commands attached

Also, could this be STP? The connection from the switch to the router, already had portfast enabled. Also, during the downtime, the switchports to the router and firewall were "up up", not "err disabled"

thanks
c4507R_Core#sh ip ospf int brief
Interface    PID   Area            IP Address/Mask    Cost  State Nbrs F/C
Gi3/22       1     0               172.17.1.1/24      1     P2MP  3/3
Vl16         1     1               172.16.16.1/24     1     DR    0/0
Vl15         1     1               172.16.15.1/24     1     DR    0/0
Vl14         1     1               172.16.14.1/24     1     DR    0/0
Vl13         1     1               172.16.13.1/24     1     DR    0/0
Vl12         1     1               172.16.12.1/24     1     DR    0/0
Vl11         1     1               172.16.11.1/24     1     DR    0/0
Vl10         1     1               172.16.10.1/24     1     DR    0/0
Vl3          1     1               172.16.3.1/24      1     DR    0/0
Vl2          1     1               172.16.2.1/24      1     DR    0/0
Vl1          1     1               172.16.1.1/24      1     DR    3/3
c4507R_Core#

Open in new window

sh ip int brief
c4507R_Core#show ip int brief
Interface              IP-Address      OK? Method Status                Protocol
Vlan1                  172.16.1.1      YES NVRAM  up                    up
Vlan2                  172.16.2.1      YES NVRAM  up                    up
Vlan3                  172.16.3.1      YES NVRAM  up                    up
Vlan10                 172.16.10.1     YES NVRAM  up                    up
Vlan11                 172.16.11.1     YES NVRAM  up                    up
Vlan12                 172.16.12.1     YES NVRAM  up                    up
Vlan13                 172.16.13.1     YES NVRAM  up                    up
Vlan14                 172.16.14.1     YES NVRAM  up                    up
Vlan15                 172.16.15.1     YES NVRAM  up                    up
Vlan16                 172.16.16.1     YES NVRAM  up                    up
Vlan100                172.16.100.1    YES NVRAM  administratively down down
Vlan101                172.16.101.1    YES NVRAM  administratively down down
Vlan102                172.16.102.1    YES NVRAM  administratively down down
Vlan103                172.16.103.1    YES NVRAM  administratively down down
Vlan1010               unassigned      YES NVRAM  administratively down down
TenGigabitEthernet1/1  unassigned      YES unset  up                    up
TenGigabitEthernet1/2  unassigned      YES unset  up                    up
GigabitEthernet1/3     unassigned      YES unset  down                  down
GigabitEthernet1/4     unassigned      YES unset  down                  down
GigabitEthernet1/5     unassigned      YES unset  down                  down
GigabitEthernet1/6     unassigned      YES unset  down                  down
GigabitEthernet3/1     unassigned      YES unset  up                    up
GigabitEthernet3/2     unassigned      YES unset  up                    up
GigabitEthernet3/3     unassigned      YES unset  up                    up
GigabitEthernet3/4     unassigned      YES unset  up                    up
GigabitEthernet3/5     unassigned      YES unset  up                    up
GigabitEthernet3/6     unassigned      YES unset  up                    up
GigabitEthernet3/7     unassigned      YES unset  up                    up
GigabitEthernet3/8     unassigned      YES unset  up                    up
GigabitEthernet3/9     unassigned      YES unset  up                    up
GigabitEthernet3/10    unassigned      YES unset  up                    up
GigabitEthernet3/11    unassigned      YES unset  up                    up
GigabitEthernet3/12    unassigned      YES unset  up                    up
GigabitEthernet3/13    unassigned      YES unset  up                    up
GigabitEthernet3/14    unassigned      YES unset  up                    up
GigabitEthernet3/15    unassigned      YES unset  up                    up
GigabitEthernet3/16    unassigned      YES unset  up                    up
GigabitEthernet3/17    unassigned      YES unset  up                    up
GigabitEthernet3/18    unassigned      YES unset  down                  down
GigabitEthernet3/19    unassigned      YES unset  up                    up
GigabitEthernet3/20    unassigned      YES unset  up                    up
GigabitEthernet3/21    unassigned      YES unset  up                    up
GigabitEthernet3/22    172.17.1.1      YES NVRAM  up                    up
GigabitEthernet3/23    unassigned      YES unset  up                    up
GigabitEthernet3/24    unassigned      YES unset  up                    up
GigabitEthernet4/1     unassigned      YES unset  up                    up
GigabitEthernet4/2     unassigned      YES unset  up                    up
GigabitEthernet4/3     unassigned      YES unset  up                    up
GigabitEthernet4/4     unassigned      YES unset  up                    up
GigabitEthernet4/5     unassigned      YES unset  up                    up
GigabitEthernet4/6     unassigned      YES unset  up                    up
GigabitEthernet4/7     unassigned      YES unset  up                    up
GigabitEthernet4/8     unassigned      YES unset  up                    up
GigabitEthernet4/9     unassigned      YES unset  up                    up
GigabitEthernet4/10    unassigned      YES unset  up                    up
GigabitEthernet4/11    unassigned      YES unset  up                    up
GigabitEthernet4/12    unassigned      YES unset  up                    up
GigabitEthernet4/13    unassigned      YES unset  up                    up
GigabitEthernet4/14    unassigned      YES unset  up                    up
GigabitEthernet4/15    unassigned      YES unset  up                    up
GigabitEthernet4/16    unassigned      YES unset  up                    up
GigabitEthernet4/17    unassigned      YES unset  up                    up
GigabitEthernet4/18    unassigned      YES unset  up                    up
GigabitEthernet4/19    unassigned      YES unset  up                    up
GigabitEthernet4/20    unassigned      YES unset  up                    up
GigabitEthernet4/21    unassigned      YES unset  up                    up
GigabitEthernet4/22    unassigned      YES unset  up                    up
GigabitEthernet4/23    unassigned      YES unset  up                    up
GigabitEthernet4/24    unassigned      YES unset  up                    up
GigabitEthernet5/1     unassigned      YES unset  down                  down
GigabitEthernet5/2     unassigned      YES unset  up                    up
GigabitEthernet5/3     unassigned      YES unset  up                    up
GigabitEthernet5/4     unassigned      YES unset  up                    up
GigabitEthernet5/5     unassigned      YES unset  up                    up
GigabitEthernet5/6     unassigned      YES unset  up                    up
GigabitEthernet5/7     unassigned      YES unset  up                    up
GigabitEthernet5/8     unassigned      YES unset  up                    up
GigabitEthernet5/9     unassigned      YES unset  up                    up
GigabitEthernet5/10    unassigned      YES unset  up                    up
GigabitEthernet5/11    unassigned      YES unset  up                    up
GigabitEthernet5/12    unassigned      YES unset  up                    up
GigabitEthernet5/13    unassigned      YES unset  down                  down
GigabitEthernet5/14    unassigned      YES unset  up                    up
GigabitEthernet5/15    unassigned      YES unset  administratively down down
GigabitEthernet5/16    unassigned      YES unset  up                    up
GigabitEthernet5/17    unassigned      YES unset  up                    up
GigabitEthernet5/18    unassigned      YES unset  down                  down
GigabitEthernet5/19    unassigned      YES unset  down                  down
GigabitEthernet5/20    unassigned      YES unset  up                    up
GigabitEthernet5/21    unassigned      YES unset  up                    up
GigabitEthernet5/22    unassigned      YES unset  down                  down
GigabitEthernet5/23    unassigned      YES unset  up                    up
GigabitEthernet5/24    unassigned      YES unset  up                    up
GigabitEthernet5/25    unassigned      YES unset  up                    up
GigabitEthernet5/26    unassigned      YES unset  up                    up
GigabitEthernet5/27    unassigned      YES unset  up                    up
GigabitEthernet5/28    unassigned      YES unset  up                    up
GigabitEthernet5/29    unassigned      YES unset  up                    up
GigabitEthernet5/30    unassigned      YES unset  up                    up
GigabitEthernet5/31    unassigned      YES unset  up                    up
GigabitEthernet5/32    unassigned      YES unset  up                    up
GigabitEthernet5/33    unassigned      YES unset  up                    up
GigabitEthernet5/34    unassigned      YES unset  up                    up
GigabitEthernet5/35    unassigned      YES unset  up                    up
GigabitEthernet5/36    unassigned      YES unset  up                    up
GigabitEthernet5/37    unassigned      YES unset  up                    up
GigabitEthernet5/38    unassigned      YES unset  up                    up
GigabitEthernet5/39    unassigned      YES unset  up                    up
GigabitEthernet5/40    unassigned      YES unset  down                  down
GigabitEthernet5/41    unassigned      YES unset  up                    up
GigabitEthernet5/42    unassigned      YES unset  up                    up
GigabitEthernet5/43    unassigned      YES unset  up                    up
GigabitEthernet5/44    unassigned      YES unset  up                    up
GigabitEthernet5/45    unassigned      YES unset  up                    up
GigabitEthernet5/46    unassigned      YES unset  up                    up
GigabitEthernet5/47    unassigned      YES unset  up                    up
GigabitEthernet5/48    unassigned      YES unset  up                    up
GigabitEthernet6/1     unassigned      YES unset  up                    up
GigabitEthernet6/2     unassigned      YES unset  up                    up
GigabitEthernet6/3     unassigned      YES unset  up                    up
GigabitEthernet6/4     unassigned      YES unset  up                    up
GigabitEthernet6/5     unassigned      YES unset  up                    up
GigabitEthernet6/6     unassigned      YES unset  up                    up
GigabitEthernet6/7     unassigned      YES unset  up                    up
GigabitEthernet6/8     unassigned      YES unset  up                    up
GigabitEthernet6/9     unassigned      YES unset  up                    up
GigabitEthernet6/10    unassigned      YES unset  up                    up
GigabitEthernet6/11    unassigned      YES unset  up                    up
GigabitEthernet6/12    unassigned      YES unset  up                    up
GigabitEthernet6/13    unassigned      YES unset  up                    up
GigabitEthernet6/14    unassigned      YES unset  down                  down
GigabitEthernet6/15    unassigned      YES unset  up                    up
GigabitEthernet6/16    unassigned      YES unset  up                    up
GigabitEthernet6/17    unassigned      YES unset  up                    up
GigabitEthernet6/18    unassigned      YES unset  up                    up
GigabitEthernet6/19    unassigned      YES unset  up                    up
GigabitEthernet6/20    unassigned      YES unset  up                    up
GigabitEthernet6/21    unassigned      YES unset  up                    up
GigabitEthernet6/22    unassigned      YES unset  up                    up
GigabitEthernet6/23    unassigned      YES unset  up                    up
GigabitEthernet6/24    unassigned      YES unset  down                  down
GigabitEthernet6/25    unassigned      YES unset  up                    up
GigabitEthernet6/26    unassigned      YES unset  up                    up
GigabitEthernet6/27    unassigned      YES unset  up                    up
GigabitEthernet6/28    unassigned      YES unset  down                  down
GigabitEthernet6/29    unassigned      YES unset  up                    up
GigabitEthernet6/30    unassigned      YES unset  up                    up
GigabitEthernet6/31    unassigned      YES unset  up                    up
GigabitEthernet6/32    unassigned      YES unset  down                  down
GigabitEthernet6/33    unassigned      YES unset  up                    up
GigabitEthernet6/34    unassigned      YES unset  up                    up
GigabitEthernet6/35    unassigned      YES unset  up                    up
GigabitEthernet6/36    unassigned      YES unset  up                    up
GigabitEthernet6/37    unassigned      YES unset  up                    up
GigabitEthernet6/38    unassigned      YES unset  down                  down
GigabitEthernet6/39    unassigned      YES unset  down                  down
GigabitEthernet6/40    unassigned      YES unset  down                  down
GigabitEthernet6/41    unassigned      YES unset  up                    up
GigabitEthernet6/42    unassigned      YES unset  down                  down
GigabitEthernet6/43    unassigned      YES unset  up                    up
GigabitEthernet6/44    unassigned      YES unset  down                  down
GigabitEthernet6/45    unassigned      YES unset  up                    up
GigabitEthernet6/46    unassigned      YES unset  down                  down
GigabitEthernet6/47    unassigned      YES unset  down                  down
GigabitEthernet6/48    unassigned      YES unset  up                    up
GigabitEthernet7/1     unassigned      YES unset  down                  down
GigabitEthernet7/2     unassigned      YES unset  up                    up
GigabitEthernet7/3     unassigned      YES unset  down                  down
GigabitEthernet7/4     unassigned      YES unset  down                  down
GigabitEthernet7/5     unassigned      YES unset  administratively down down
GigabitEthernet7/6     unassigned      YES unset  down                  down
GigabitEthernet7/7     unassigned      YES unset  down                  down
GigabitEthernet7/8     unassigned      YES unset  administratively down down
GigabitEthernet7/9     unassigned      YES unset  down                  down
GigabitEthernet7/10    unassigned      YES unset  up                    up
GigabitEthernet7/11    unassigned      YES unset  up                    up
GigabitEthernet7/12    unassigned      YES unset  down                  down
GigabitEthernet7/13    unassigned      YES unset  down                  down
GigabitEthernet7/14    unassigned      YES unset  down                  down
GigabitEthernet7/15    unassigned      YES unset  up                    up
GigabitEthernet7/16    unassigned      YES unset  up                    up
GigabitEthernet7/17    unassigned      YES unset  up                    up
GigabitEthernet7/18    unassigned      YES unset  down                  down
GigabitEthernet7/19    unassigned      YES unset  down                  down
GigabitEthernet7/20    unassigned      YES unset  down                  down
GigabitEthernet7/21    unassigned      YES unset  down                  down
GigabitEthernet7/22    unassigned      YES unset  down                  down
GigabitEthernet7/23    unassigned      YES unset  administratively down down
GigabitEthernet7/24    unassigned      YES unset  down                  down
GigabitEthernet7/25    unassigned      YES unset  down                  down
GigabitEthernet7/26    unassigned      YES unset  down                  down
GigabitEthernet7/27    unassigned      YES unset  up                    up
GigabitEthernet7/28    unassigned      YES unset  down                  down
GigabitEthernet7/29    unassigned      YES unset  up                    up
GigabitEthernet7/30    unassigned      YES unset  up                    up
GigabitEthernet7/31    unassigned      YES unset  down                  down
GigabitEthernet7/32    unassigned      YES unset  down                  down
GigabitEthernet7/33    unassigned      YES unset  down                  down
GigabitEthernet7/34    unassigned      YES unset  down                  down
GigabitEthernet7/35    unassigned      YES unset  down                  down
GigabitEthernet7/36    unassigned      YES unset  up                    up
GigabitEthernet7/37    unassigned      YES unset  up                    up
GigabitEthernet7/38    unassigned      YES unset  up                    up
GigabitEthernet7/39    unassigned      YES unset  up                    up
GigabitEthernet7/40    unassigned      YES unset  down                  down
GigabitEthernet7/41    unassigned      YES unset  down                  down
GigabitEthernet7/42    unassigned      YES unset  down                  down
GigabitEthernet7/43    unassigned      YES unset  down                  down
GigabitEthernet7/44    unassigned      YES unset  down                  down
GigabitEthernet7/45    unassigned      YES unset  down                  down
GigabitEthernet7/46    unassigned      YES unset  down                  down
GigabitEthernet7/47    unassigned      YES unset  up                    up
GigabitEthernet7/48    unassigned      YES unset  up                    up
Port-channel1          unassigned      YES NVRAM  down                  down
Port-channel3          unassigned      YES unset  up                    up
Port-channel4          unassigned      YES unset  up                    up
Port-channel5          unassigned      YES unset  up                    up
Port-channel16         unassigned      YES NVRAM  down                  down
Port-channel17         unassigned      YES unset  up                    up
Port-channel18         unassigned      YES NVRAM  down                  down
c4507R_Core#

Open in new window

0
Become a Leader in Data Analytics

Gain the power to turn raw data into better business decisions and outcomes in your industry. Transform your career future by earning your MS in Data Analytics. WGU’s MSDA program curriculum features IT certifications from Oracle and SAS.  

 
Alan Huseyin KayahanCommented:
You better focus on "show spanning-tree" outputs rather than OSPF or interface outputs. Also if possible, set port mirroring for the ports that you had problems (switch-firewall and switch-router) and listen the conversation. That would of course be usefull if you can regenerate the problem
0
 
orusAuthor Commented:
There is nothing indicating an stp issue, looping or flapping
What exactly should i be looking at?  Any assistance appreciated
0
 
Alan Huseyin KayahanCommented:
A question,
   Does the problem still persist? Because if not, if it is resolved and everything is working, there is no command that can determine the cause.
Btw is this a Gbic of ethernet port?
0
 
orusAuthor Commented:
well I looked at the logs when the issue was going on. I saw nothing showing STP or looping etc. It happened last night. I want to make sure it is fully operational and doesnt happen again.

We use tengig fiber modules to connect to our access switches.  We switched from copper to tengig fiber last night. I verified the copper ports are in a blocking state on the access switches
0
 
rochey2009Commented:
what is the state of the interfaces going to the firewall and the router?

show interface x
0
 
Alan Huseyin KayahanCommented:
For an effective solution, you should regenerate the problem in a controlled and scheduled environment. If you plan to do so, set the clocks of testing host, syslog server and log generating devices correctly. Also mirror the port and listen the conversation from/to specific port with wireshark, again synchronized clock.
So once you launch ping we can see the responses from devices at that time and arrive at a conclusion.

Are there any HSRP or equivalant load balancing configs in place?

Taking the situation into consideration, which is "temporary disconnectivity or (late connectivity) between directly connected nodes, right after switch restart", the answer is usually

Load balancing protocol negotiations>STP issues> Port security issues>Duplex and speed negotiations (epecially between OEM non-standard or faulty gbics/ports>Cabling

In some cases, leaving the speed at auto-sense in such important ports may create similar outcomes.

If i were you and did not have the chance to regenerate the issue, I would
1)Manually enter the speed and duplex of the ports
2)Statically enter the MAC-IP-PORT binding into switch
0
 
orusAuthor Commented:
No load balancing. I can try and recreate in our lab.  

In the router: I don't see anything indicating a duplex mismatch in any of the buffers either, no input errors or CRCs on the interface.  Everything looks clean. The only errors I saw were "Ospf neighbor down: dead timer expired"

If I go into the switch, and look at the switchport where the router plugs in, I see 858 runts, 1925 input errors and 1067 CRCs. But the counters have never been cleared on this interface. Do they clear during a reboot?

All of this resolved once I did a clean reload of the router and the issue never came back. Could this be due to OSPF or do you still think it is possibly a duplex or STP issue? I figured I would see those logs indicating that in the buffer though...
0
 
DanJCommented:
"The switch seemed to be able to ping all local hosts, but not the firewall or router."
You lost layer 3 connectivity between these devices. OSPF runs on top of IP. No IP reachability means OSPF adjacencies are lost.

0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.

  • 4
  • 4
  • 2
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now