Network Wide Broadcast Storm with mostly Managed Switches
Posted on 2004-11-09
We have recently diagnosed a broadcast problem on our high speed internet network. It is unclear how long it has been going on for. However, it is bringing our network to an error ridden bog every time it happens.
Here is our setup:
1 HP Procurve central switch.
5 other managed switches connected various ways to the central procurve. Some with fiber and some with copper. (Dell, Netgear, Dlink)
There are about 5 more unmanaged switches connected to some of the 5 managed switches.
So in all there are around 11 or 12 switches on this network.
Here is our problem:
I notced our activity lights constantly flashing on all switches. I hooked a network sniffer onto the network and noticed about 1500-2200 packets going through every second. This is the same no matter what switch I am plugged into.
The only way to stop the storm is to reset the central HP switch and at that point everything seems to go back to normal with somewhere between 15 and 200 packets per second.
There is not a single source for the broadcast storm many dinnferent types of MAC addresses have been identified as culprits. Each packet is identical.
The only way I can describe it is a device on the network sends our a broadcast and it starts to bounce around like a pinball machine and wont stop until it is unplugged.
This doesnt seem to happen with every broadcast just some, sometimes.
I have increased some on the broadcast storm settings on the switches to try and fix the problem. It seems to have helped some. But when the broadcast storm gets rolling it is just as bad as when the switches are set to the default settings.
If you have any questions please let me know......Any help would be greatly appreciated!!!