dicecomputers
asked on
Unexplainable network issue
Let me give you a full rundown of what has happened, because this is a nightmare that doesn’t seem to end.
Friday night, we took 1 of our networks (10 OM1P’s, shown in diagram) and changed the dashboard to allow flashing to NG firmware. Shortly after, the 192.168.0.x network started shutting down, no ability to connect to the outside world for intermittent periods of time. When I ran wireshark, I found several broadcasts from protocol “Batman” (the protocol associated with NG firmware). We unplugged the backbone to this network and our 192.168.0.x network started working again. We took all 10 points and manually flashed them to NG, and then put them back. Within a few seconds of bringing that network back online, the 192.168.0.x network started dropping again. We then took all 10 OM1P’s and manually flashed them back to r2690 and put them back in place. Within a few seconds, the 192.168.0.x network started dropping again. We then removed all of the OM1P’s but left the network (2, 900MHz backbone panels, and 2 switches) up and everything was fine. We took a working OM1P from another network (r2693) and plugged it in by itself on the network in question and the network started dropping again. We also tried changing the switch with the same results. We also tried removing the points from the network in the dashboard and creating a new network with the same results.
I am out of ideas. Any help from you would be greatly appreciated.
Thanks,
Rick
EEDiag.pdf
Friday night, we took 1 of our networks (10 OM1P’s, shown in diagram) and changed the dashboard to allow flashing to NG firmware. Shortly after, the 192.168.0.x network started shutting down, no ability to connect to the outside world for intermittent periods of time. When I ran wireshark, I found several broadcasts from protocol “Batman” (the protocol associated with NG firmware). We unplugged the backbone to this network and our 192.168.0.x network started working again. We took all 10 points and manually flashed them to NG, and then put them back. Within a few seconds of bringing that network back online, the 192.168.0.x network started dropping again. We then took all 10 OM1P’s and manually flashed them back to r2690 and put them back in place. Within a few seconds, the 192.168.0.x network started dropping again. We then removed all of the OM1P’s but left the network (2, 900MHz backbone panels, and 2 switches) up and everything was fine. We took a working OM1P from another network (r2693) and plugged it in by itself on the network in question and the network started dropping again. We also tried changing the switch with the same results. We also tried removing the points from the network in the dashboard and creating a new network with the same results.
I am out of ideas. Any help from you would be greatly appreciated.
Thanks,
Rick
EEDiag.pdf
Interesting indeed.
Is there any IPv6 turned on within the new OM1P network?
How are these networks connected to the firewall? Individual NICs, via switch?
Is there any IPv6 turned on within the new OM1P network?
How are these networks connected to the firewall? Individual NICs, via switch?
Just for clarity, are these all /24 subnets?
And since it appears to be where the subnets all converge, what is the firewall appliance?
And since it appears to be where the subnets all converge, what is the firewall appliance?
ASKER
i haven't turned on any ipv6 but when i ran a capture, i did see some ipv6 addresses. yes, they are all on /24 right now.
ASKER
sorry, i forgot, everything connect to our firewall through 1 port right now. 1 inside and 1 outside interface right now
Disable all IPv6 bindings to remove that as a possible concern.
If it continues, run a few tracert sets from the OM1P side to the 192.168.0.x and vise versa - post them if you can.
Can you confirm the connections to the firewall as separate NICs or an aggregate of some type (switch, etc)?
If it continues, run a few tracert sets from the OM1P side to the 192.168.0.x and vise versa - post them if you can.
Can you confirm the connections to the firewall as separate NICs or an aggregate of some type (switch, etc)?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
My Cisco guy just got here and he said the same thing. He thinks it's a storm. I will go ahead and close this since I am sure he can fix it. Thanks for your help.
ASKER