We help IT Professionals succeed at work.

Unexplainable network issue

Let me give you a full rundown of what has happened, because this is a nightmare that doesn’t seem to end.
Friday night, we took 1 of our networks (10 OM1P’s, shown in diagram) and changed the dashboard to allow flashing to NG firmware. Shortly after, the 192.168.0.x network started shutting down, no ability to connect to the outside world for intermittent periods of time. When I ran wireshark, I found several broadcasts from protocol “Batman” (the protocol associated with NG firmware). We unplugged the backbone to this network and our 192.168.0.x network started working again. We took all 10 points and manually flashed them to NG, and then put them back. Within a few seconds of bringing that network back online, the 192.168.0.x network started dropping again. We then took all 10 OM1P’s and manually flashed them back to r2690 and put them back in place. Within a few seconds, the 192.168.0.x network started dropping again. We then removed all of the OM1P’s but left the network (2, 900MHz backbone panels, and 2 switches) up and everything was fine. We took a working OM1P from another network (r2693) and plugged it in by itself on the network in question and the network started dropping again. We also tried changing the switch with the same results. We also tried removing the points from the network in the dashboard and creating a new network with the same results.
I am out of ideas. Any help from you would be greatly appreciated.
Thanks,
Rick
 EEDiag.pdf
Comment
Watch Question

Author

Commented:
Since this, we tried inserting a router between the second half of the back haul, and the OM1P's, to separate it from the other networks, and it still took down the 192.168.0.x network. I didn't even know this was possible. As far as I know, the network is not in any 4th dimension, and the buildings are not on any Indian burial grounds.
CERTIFIED EXPERT
Top Expert 2005

Commented:
Interesting indeed.

Is there any IPv6 turned on within the new OM1P network?

How are these networks connected to the firewall?  Individual NICs, via switch?

CERTIFIED EXPERT

Commented:
Just for clarity, are these all /24 subnets?

And since it appears to be where the subnets all converge, what is the firewall appliance?

Author

Commented:
i haven't turned on any ipv6 but when i ran a capture, i did see some ipv6 addresses. yes, they are all on /24 right now.

Author

Commented:
sorry, i forgot, everything connect to our firewall through 1 port right now. 1 inside and 1 outside interface right now
CERTIFIED EXPERT
Top Expert 2005

Commented:
Disable all IPv6 bindings to remove that as a possible concern.

If it continues, run a few tracert sets from the OM1P side to the 192.168.0.x and vise versa - post them if you can.

Can you confirm the connections to the firewall as separate NICs or an aggregate of some type (switch, etc)?
CERTIFIED EXPERT
Top Expert 2005
Commented:
If those networks enter a switch before the single firewall connection, are they vLan'd off?  

This switch might be sensing a storm and blocking ports - or the switch may be misconfigured.

Author

Commented:
My  Cisco guy just got here and he said the same thing. He thinks it's a storm. I will go ahead and close this since I am sure he can fix it. Thanks for your help.

Explore More ContentExplore courses, solutions, and other research materials related to this topic.