orus
asked on
Ospf caused outage?
We have a core switch, mpls ce router ( we are the customer) and a firewall that faces the internet.
All 3 devices run ospf. And all 3 devices are in the same subnet. Tonight we rebooted the core switch. When the switch came back up, i noticed the router and firewall had no connectivity to the switch, but had connectivity to outside networks. The switch seemed to be able to ping all local hosts, but not the firewall or router. The mpls routers' logs showed several messages stating " dead timer expired"
Even if ospf adjacency was lost, these devices should have still been able to speak since they are in the same subnet
So what happened"
All 3 devices run ospf. And all 3 devices are in the same subnet. Tonight we rebooted the core switch. When the switch came back up, i noticed the router and firewall had no connectivity to the switch, but had connectivity to outside networks. The switch seemed to be able to ping all local hosts, but not the firewall or router. The mpls routers' logs showed several messages stating " dead timer expired"
Even if ospf adjacency was lost, these devices should have still been able to speak since they are in the same subnet
So what happened"
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
sh commands attached
Also, could this be STP? The connection from the switch to the router, already had portfast enabled. Also, during the downtime, the switchports to the router and firewall were "up up", not "err disabled"
thanks
Also, could this be STP? The connection from the switch to the router, already had portfast enabled. Also, during the downtime, the switchports to the router and firewall were "up up", not "err disabled"
thanks
c4507R_Core#sh ip ospf int brief
Interface PID Area IP Address/Mask Cost State Nbrs F/C
Gi3/22 1 0 172.17.1.1/24 1 P2MP 3/3
Vl16 1 1 172.16.16.1/24 1 DR 0/0
Vl15 1 1 172.16.15.1/24 1 DR 0/0
Vl14 1 1 172.16.14.1/24 1 DR 0/0
Vl13 1 1 172.16.13.1/24 1 DR 0/0
Vl12 1 1 172.16.12.1/24 1 DR 0/0
Vl11 1 1 172.16.11.1/24 1 DR 0/0
Vl10 1 1 172.16.10.1/24 1 DR 0/0
Vl3 1 1 172.16.3.1/24 1 DR 0/0
Vl2 1 1 172.16.2.1/24 1 DR 0/0
Vl1 1 1 172.16.1.1/24 1 DR 3/3
c4507R_Core#
sh ip int brief
c4507R_Core#show ip int brief
Interface IP-Address OK? Method Status Protocol
Vlan1 172.16.1.1 YES NVRAM up up
Vlan2 172.16.2.1 YES NVRAM up up
Vlan3 172.16.3.1 YES NVRAM up up
Vlan10 172.16.10.1 YES NVRAM up up
Vlan11 172.16.11.1 YES NVRAM up up
Vlan12 172.16.12.1 YES NVRAM up up
Vlan13 172.16.13.1 YES NVRAM up up
Vlan14 172.16.14.1 YES NVRAM up up
Vlan15 172.16.15.1 YES NVRAM up up
Vlan16 172.16.16.1 YES NVRAM up up
Vlan100 172.16.100.1 YES NVRAM administratively down down
Vlan101 172.16.101.1 YES NVRAM administratively down down
Vlan102 172.16.102.1 YES NVRAM administratively down down
Vlan103 172.16.103.1 YES NVRAM administratively down down
Vlan1010 unassigned YES NVRAM administratively down down
TenGigabitEthernet1/1 unassigned YES unset up up
TenGigabitEthernet1/2 unassigned YES unset up up
GigabitEthernet1/3 unassigned YES unset down down
GigabitEthernet1/4 unassigned YES unset down down
GigabitEthernet1/5 unassigned YES unset down down
GigabitEthernet1/6 unassigned YES unset down down
GigabitEthernet3/1 unassigned YES unset up up
GigabitEthernet3/2 unassigned YES unset up up
GigabitEthernet3/3 unassigned YES unset up up
GigabitEthernet3/4 unassigned YES unset up up
GigabitEthernet3/5 unassigned YES unset up up
GigabitEthernet3/6 unassigned YES unset up up
GigabitEthernet3/7 unassigned YES unset up up
GigabitEthernet3/8 unassigned YES unset up up
GigabitEthernet3/9 unassigned YES unset up up
GigabitEthernet3/10 unassigned YES unset up up
GigabitEthernet3/11 unassigned YES unset up up
GigabitEthernet3/12 unassigned YES unset up up
GigabitEthernet3/13 unassigned YES unset up up
GigabitEthernet3/14 unassigned YES unset up up
GigabitEthernet3/15 unassigned YES unset up up
GigabitEthernet3/16 unassigned YES unset up up
GigabitEthernet3/17 unassigned YES unset up up
GigabitEthernet3/18 unassigned YES unset down down
GigabitEthernet3/19 unassigned YES unset up up
GigabitEthernet3/20 unassigned YES unset up up
GigabitEthernet3/21 unassigned YES unset up up
GigabitEthernet3/22 172.17.1.1 YES NVRAM up up
GigabitEthernet3/23 unassigned YES unset up up
GigabitEthernet3/24 unassigned YES unset up up
GigabitEthernet4/1 unassigned YES unset up up
GigabitEthernet4/2 unassigned YES unset up up
GigabitEthernet4/3 unassigned YES unset up up
GigabitEthernet4/4 unassigned YES unset up up
GigabitEthernet4/5 unassigned YES unset up up
GigabitEthernet4/6 unassigned YES unset up up
GigabitEthernet4/7 unassigned YES unset up up
GigabitEthernet4/8 unassigned YES unset up up
GigabitEthernet4/9 unassigned YES unset up up
GigabitEthernet4/10 unassigned YES unset up up
GigabitEthernet4/11 unassigned YES unset up up
GigabitEthernet4/12 unassigned YES unset up up
GigabitEthernet4/13 unassigned YES unset up up
GigabitEthernet4/14 unassigned YES unset up up
GigabitEthernet4/15 unassigned YES unset up up
GigabitEthernet4/16 unassigned YES unset up up
GigabitEthernet4/17 unassigned YES unset up up
GigabitEthernet4/18 unassigned YES unset up up
GigabitEthernet4/19 unassigned YES unset up up
GigabitEthernet4/20 unassigned YES unset up up
GigabitEthernet4/21 unassigned YES unset up up
GigabitEthernet4/22 unassigned YES unset up up
GigabitEthernet4/23 unassigned YES unset up up
GigabitEthernet4/24 unassigned YES unset up up
GigabitEthernet5/1 unassigned YES unset down down
GigabitEthernet5/2 unassigned YES unset up up
GigabitEthernet5/3 unassigned YES unset up up
GigabitEthernet5/4 unassigned YES unset up up
GigabitEthernet5/5 unassigned YES unset up up
GigabitEthernet5/6 unassigned YES unset up up
GigabitEthernet5/7 unassigned YES unset up up
GigabitEthernet5/8 unassigned YES unset up up
GigabitEthernet5/9 unassigned YES unset up up
GigabitEthernet5/10 unassigned YES unset up up
GigabitEthernet5/11 unassigned YES unset up up
GigabitEthernet5/12 unassigned YES unset up up
GigabitEthernet5/13 unassigned YES unset down down
GigabitEthernet5/14 unassigned YES unset up up
GigabitEthernet5/15 unassigned YES unset administratively down down
GigabitEthernet5/16 unassigned YES unset up up
GigabitEthernet5/17 unassigned YES unset up up
GigabitEthernet5/18 unassigned YES unset down down
GigabitEthernet5/19 unassigned YES unset down down
GigabitEthernet5/20 unassigned YES unset up up
GigabitEthernet5/21 unassigned YES unset up up
GigabitEthernet5/22 unassigned YES unset down down
GigabitEthernet5/23 unassigned YES unset up up
GigabitEthernet5/24 unassigned YES unset up up
GigabitEthernet5/25 unassigned YES unset up up
GigabitEthernet5/26 unassigned YES unset up up
GigabitEthernet5/27 unassigned YES unset up up
GigabitEthernet5/28 unassigned YES unset up up
GigabitEthernet5/29 unassigned YES unset up up
GigabitEthernet5/30 unassigned YES unset up up
GigabitEthernet5/31 unassigned YES unset up up
GigabitEthernet5/32 unassigned YES unset up up
GigabitEthernet5/33 unassigned YES unset up up
GigabitEthernet5/34 unassigned YES unset up up
GigabitEthernet5/35 unassigned YES unset up up
GigabitEthernet5/36 unassigned YES unset up up
GigabitEthernet5/37 unassigned YES unset up up
GigabitEthernet5/38 unassigned YES unset up up
GigabitEthernet5/39 unassigned YES unset up up
GigabitEthernet5/40 unassigned YES unset down down
GigabitEthernet5/41 unassigned YES unset up up
GigabitEthernet5/42 unassigned YES unset up up
GigabitEthernet5/43 unassigned YES unset up up
GigabitEthernet5/44 unassigned YES unset up up
GigabitEthernet5/45 unassigned YES unset up up
GigabitEthernet5/46 unassigned YES unset up up
GigabitEthernet5/47 unassigned YES unset up up
GigabitEthernet5/48 unassigned YES unset up up
GigabitEthernet6/1 unassigned YES unset up up
GigabitEthernet6/2 unassigned YES unset up up
GigabitEthernet6/3 unassigned YES unset up up
GigabitEthernet6/4 unassigned YES unset up up
GigabitEthernet6/5 unassigned YES unset up up
GigabitEthernet6/6 unassigned YES unset up up
GigabitEthernet6/7 unassigned YES unset up up
GigabitEthernet6/8 unassigned YES unset up up
GigabitEthernet6/9 unassigned YES unset up up
GigabitEthernet6/10 unassigned YES unset up up
GigabitEthernet6/11 unassigned YES unset up up
GigabitEthernet6/12 unassigned YES unset up up
GigabitEthernet6/13 unassigned YES unset up up
GigabitEthernet6/14 unassigned YES unset down down
GigabitEthernet6/15 unassigned YES unset up up
GigabitEthernet6/16 unassigned YES unset up up
GigabitEthernet6/17 unassigned YES unset up up
GigabitEthernet6/18 unassigned YES unset up up
GigabitEthernet6/19 unassigned YES unset up up
GigabitEthernet6/20 unassigned YES unset up up
GigabitEthernet6/21 unassigned YES unset up up
GigabitEthernet6/22 unassigned YES unset up up
GigabitEthernet6/23 unassigned YES unset up up
GigabitEthernet6/24 unassigned YES unset down down
GigabitEthernet6/25 unassigned YES unset up up
GigabitEthernet6/26 unassigned YES unset up up
GigabitEthernet6/27 unassigned YES unset up up
GigabitEthernet6/28 unassigned YES unset down down
GigabitEthernet6/29 unassigned YES unset up up
GigabitEthernet6/30 unassigned YES unset up up
GigabitEthernet6/31 unassigned YES unset up up
GigabitEthernet6/32 unassigned YES unset down down
GigabitEthernet6/33 unassigned YES unset up up
GigabitEthernet6/34 unassigned YES unset up up
GigabitEthernet6/35 unassigned YES unset up up
GigabitEthernet6/36 unassigned YES unset up up
GigabitEthernet6/37 unassigned YES unset up up
GigabitEthernet6/38 unassigned YES unset down down
GigabitEthernet6/39 unassigned YES unset down down
GigabitEthernet6/40 unassigned YES unset down down
GigabitEthernet6/41 unassigned YES unset up up
GigabitEthernet6/42 unassigned YES unset down down
GigabitEthernet6/43 unassigned YES unset up up
GigabitEthernet6/44 unassigned YES unset down down
GigabitEthernet6/45 unassigned YES unset up up
GigabitEthernet6/46 unassigned YES unset down down
GigabitEthernet6/47 unassigned YES unset down down
GigabitEthernet6/48 unassigned YES unset up up
GigabitEthernet7/1 unassigned YES unset down down
GigabitEthernet7/2 unassigned YES unset up up
GigabitEthernet7/3 unassigned YES unset down down
GigabitEthernet7/4 unassigned YES unset down down
GigabitEthernet7/5 unassigned YES unset administratively down down
GigabitEthernet7/6 unassigned YES unset down down
GigabitEthernet7/7 unassigned YES unset down down
GigabitEthernet7/8 unassigned YES unset administratively down down
GigabitEthernet7/9 unassigned YES unset down down
GigabitEthernet7/10 unassigned YES unset up up
GigabitEthernet7/11 unassigned YES unset up up
GigabitEthernet7/12 unassigned YES unset down down
GigabitEthernet7/13 unassigned YES unset down down
GigabitEthernet7/14 unassigned YES unset down down
GigabitEthernet7/15 unassigned YES unset up up
GigabitEthernet7/16 unassigned YES unset up up
GigabitEthernet7/17 unassigned YES unset up up
GigabitEthernet7/18 unassigned YES unset down down
GigabitEthernet7/19 unassigned YES unset down down
GigabitEthernet7/20 unassigned YES unset down down
GigabitEthernet7/21 unassigned YES unset down down
GigabitEthernet7/22 unassigned YES unset down down
GigabitEthernet7/23 unassigned YES unset administratively down down
GigabitEthernet7/24 unassigned YES unset down down
GigabitEthernet7/25 unassigned YES unset down down
GigabitEthernet7/26 unassigned YES unset down down
GigabitEthernet7/27 unassigned YES unset up up
GigabitEthernet7/28 unassigned YES unset down down
GigabitEthernet7/29 unassigned YES unset up up
GigabitEthernet7/30 unassigned YES unset up up
GigabitEthernet7/31 unassigned YES unset down down
GigabitEthernet7/32 unassigned YES unset down down
GigabitEthernet7/33 unassigned YES unset down down
GigabitEthernet7/34 unassigned YES unset down down
GigabitEthernet7/35 unassigned YES unset down down
GigabitEthernet7/36 unassigned YES unset up up
GigabitEthernet7/37 unassigned YES unset up up
GigabitEthernet7/38 unassigned YES unset up up
GigabitEthernet7/39 unassigned YES unset up up
GigabitEthernet7/40 unassigned YES unset down down
GigabitEthernet7/41 unassigned YES unset down down
GigabitEthernet7/42 unassigned YES unset down down
GigabitEthernet7/43 unassigned YES unset down down
GigabitEthernet7/44 unassigned YES unset down down
GigabitEthernet7/45 unassigned YES unset down down
GigabitEthernet7/46 unassigned YES unset down down
GigabitEthernet7/47 unassigned YES unset up up
GigabitEthernet7/48 unassigned YES unset up up
Port-channel1 unassigned YES NVRAM down down
Port-channel3 unassigned YES unset up up
Port-channel4 unassigned YES unset up up
Port-channel5 unassigned YES unset up up
Port-channel16 unassigned YES NVRAM down down
Port-channel17 unassigned YES unset up up
Port-channel18 unassigned YES NVRAM down down
c4507R_Core#
You better focus on "show spanning-tree" outputs rather than OSPF or interface outputs. Also if possible, set port mirroring for the ports that you had problems (switch-firewall and switch-router) and listen the conversation. That would of course be usefull if you can regenerate the problem
ASKER
There is nothing indicating an stp issue, looping or flapping
What exactly should i be looking at? Any assistance appreciated
What exactly should i be looking at? Any assistance appreciated
A question,
Does the problem still persist? Because if not, if it is resolved and everything is working, there is no command that can determine the cause.
Btw is this a Gbic of ethernet port?
Does the problem still persist? Because if not, if it is resolved and everything is working, there is no command that can determine the cause.
Btw is this a Gbic of ethernet port?
ASKER
well I looked at the logs when the issue was going on. I saw nothing showing STP or looping etc. It happened last night. I want to make sure it is fully operational and doesnt happen again.
We use tengig fiber modules to connect to our access switches. We switched from copper to tengig fiber last night. I verified the copper ports are in a blocking state on the access switches
We use tengig fiber modules to connect to our access switches. We switched from copper to tengig fiber last night. I verified the copper ports are in a blocking state on the access switches
what is the state of the interfaces going to the firewall and the router?
show interface x
show interface x
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
No load balancing. I can try and recreate in our lab.
In the router: I don't see anything indicating a duplex mismatch in any of the buffers either, no input errors or CRCs on the interface. Everything looks clean. The only errors I saw were "Ospf neighbor down: dead timer expired"
If I go into the switch, and look at the switchport where the router plugs in, I see 858 runts, 1925 input errors and 1067 CRCs. But the counters have never been cleared on this interface. Do they clear during a reboot?
All of this resolved once I did a clean reload of the router and the issue never came back. Could this be due to OSPF or do you still think it is possibly a duplex or STP issue? I figured I would see those logs indicating that in the buffer though...
In the router: I don't see anything indicating a duplex mismatch in any of the buffers either, no input errors or CRCs on the interface. Everything looks clean. The only errors I saw were "Ospf neighbor down: dead timer expired"
If I go into the switch, and look at the switchport where the router plugs in, I see 858 runts, 1925 input errors and 1067 CRCs. But the counters have never been cleared on this interface. Do they clear during a reboot?
All of this resolved once I did a clean reload of the router and the issue never came back. Could this be due to OSPF or do you still think it is possibly a duplex or STP issue? I figured I would see those logs indicating that in the buffer though...
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
please post
show ip int brief
show ip ospf int brief
from the switch