Network Wide Broadcast Storm with mostly Managed Switches

Posted on 2004-11-09
Last Modified: 2013-12-07
We have recently diagnosed a broadcast problem on our high speed internet network. It is unclear how long it has been going on for. However, it is bringing our network to an error ridden bog every time it happens.

Here is our setup:

1 HP Procurve central switch.
5 other managed switches connected various ways to the central procurve. Some with fiber and some with copper. (Dell, Netgear, Dlink)
There are about 5 more unmanaged switches connected to some of the 5 managed switches.

So in all there are around 11 or 12 switches on this network.  

Here is our problem:
I notced our activity lights constantly flashing on all switches. I hooked a network sniffer onto the network and noticed about 1500-2200 packets going through every second. This is the same no matter what switch I am plugged into.

The only way to stop the storm is to reset the central HP switch and at that point everything seems to go back to normal with somewhere between 15 and 200 packets per second.

There is not a single source for the broadcast storm many dinnferent types of MAC addresses have been identified as culprits. Each packet is identical.

The only way I can describe it is a device on the network sends our a broadcast and it starts to bounce around like a pinball machine and wont stop until it is unplugged.

This doesnt seem to happen with every broadcast just some, sometimes.

I have increased some on the broadcast storm settings on the switches to try and fix the problem. It seems to have helped some. But when the broadcast storm gets rolling it is just as bad as when the switches are set to the default settings.

If you have any questions please let me know......Any help would be greatly appreciated!!!

Question by:Mohonk
    LVL 30

    Assisted Solution

    Could be spanning tree issue.But in any event look at this:

    1 - Is there more than one frame type on the file servers, routers, print servers, etc.? If the answer is yes, then determine if every application and/or protocol on the network can run on one single frame type. Using a single frame type reduces redundant broadcast traffic.

    2 - Is the network using many protocols, such as IPX, TCP/IP, LAT, SNA, NetBEUI, etc...? Is it possible to run all applications using a single protocol? If so, reconfigure the applications to run over this single protocol. Each protocol type requires it's own broadcasts, so minimizing the number of protocol families can lead to fewer broadcasts.

    3 - External print servers and print server cards are known as 'plug-and-play' or 'ease of installation' devices. This simplicity comes with a price. Often, these devices are packaged with all of the major network protocols enabled, and sometimes multiple frame types are enabled. Most print servers have a management console or configuration screen to display what protocols and frame types are enabled. Disable all of the protocols and frame types not used on the network for printing.

    4 - Most network switches default to enabling the spanning tree bridge protocol. Spanning tree is used for fault tolerance if redundant routes exist on the network. Unless your network is extra-mission-critical, it probably does not have redundant routes from every workstation. If possible, disable the spanning tree protocol. Spanning tree prevents loops on a network by sending out a 'hello' frame from each port every 2 seconds, which then gets resolved by every bridge or switch on the network. On a network with many switched nodes, a misconfiguration of the spanning tree protocols can create MANY broadcasts!

    5- Make sure the WAN devices or routers have spoofing and/or filtering enabled. Contact your router manufacturer for specific functionality. The goal is to reduce the amount of broadcasts traversing the LAN and WAN, and to help conserve buffering memory inside the routers.

    6- Have a network baseline analysis performed by an impartial 3rd party. A properly executed analysis will define protocols in use, identify problematic nodes, and give other pertinent information relating to the network's overall performance at all layers.
    LVL 5

    Accepted Solution

    Looks like a duplicate IP or a ambiguous LAN Path (STP is not enabled)

    Steps to try in order to solve it:

    1. Go and check your paths (links) between switches - when u have the loop on the main swtich, you have all the lights blinking madly!
    So go and disconnect each cable 1 by 1, wait for each cable for a few moments - see that the lights stop blinking. In case they have - you found the end of the problematic link. Go with it forward and see where it ends - then check on that switch the same thing.

    By the end of the procedure you will have a LOOP, which means that you connected 2 switches (2 ends ) with more than 1 cable (ambiguous path).

    2. Check for different MAC addresses for the same IP address in arp table : go to several PCs and open a CMD then type "arp -a" in order to print the arp table, then look for same IP records.

    Good luck.

    PS: Source of the broadcast can help you (seen in sniffer) - find that MAC or IP and check the arp table there (arp -a).

    LVL 9

    Assisted Solution

    What kinda OS is on the clients ??..
    Is it mainly using the DNS ?.
    Is WIns installed.....

    Are all clients using the Wins server .....

    How many network connections:
    12*24=300 users ?.. one large IP broadcast domain...
    LVL 4

    Assisted Solution

    We had a similar problem and eventually (many many months) tracked it down to a faulty backplane on the switch.


    Author Comment

    Really thats interesting Beldoran. However we had this problem and put this HP in to try and solve it. So it was happening even with another switch. Something HP tech support told us to try is set all ports to 100Full Duplex instead of auto. So were going to give that a shot.

    Author Comment

    Thanks for all the help. I have narrowed down the problem to the following:

    All broadcast storm packets originated either from the cisco 1100 WAPs or clients using these access points. As i removed these access points from the network the problem slowed down. When I removed all of the access points the problem subsided. I have yet to call Cisco to see if there is a fix. But the one time i plugged an AP back into the network it blew up again.

    I will split the points between all who commented. Thanks

    LVL 9

    Expert Comment

    hi, there

    what you could try is to simply create a second VLAN for the Access Point's then they will be on there on broadcast doamin...

    Featured Post

    Do You Know the 4 Main Threat Actor Types?

    Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

    Join & Write a Comment

    Don’t let your business fall victim to the coming apocalypse – use our Survival Guide for the Fax Apocalypse to identify the risks and signs of zombie fax activities at your business.
    Meet the world's only “Transparent Cloud™” from Superb Internet Corporation. Now, you can experience firsthand a cloud platform that consistently outperforms Amazon Web Services (AWS), IBM’s Softlayer, and Microsoft’s Azure when it comes to CPU and …
    Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
    In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor ( If you're interested in additional methods for monitoring bandwidt…

    746 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    15 Experts available now in Live!

    Get 1:1 Help Now