?
Solved

Network Wide Broadcast Storm with mostly Managed Switches

Posted on 2004-11-09
7
Medium Priority
?
3,033 Views
Last Modified: 2013-12-07
We have recently diagnosed a broadcast problem on our high speed internet network. It is unclear how long it has been going on for. However, it is bringing our network to an error ridden bog every time it happens.

Here is our setup:

1 HP Procurve central switch.
5 other managed switches connected various ways to the central procurve. Some with fiber and some with copper. (Dell, Netgear, Dlink)
There are about 5 more unmanaged switches connected to some of the 5 managed switches.

So in all there are around 11 or 12 switches on this network.  

Here is our problem:
I notced our activity lights constantly flashing on all switches. I hooked a network sniffer onto the network and noticed about 1500-2200 packets going through every second. This is the same no matter what switch I am plugged into.

The only way to stop the storm is to reset the central HP switch and at that point everything seems to go back to normal with somewhere between 15 and 200 packets per second.

There is not a single source for the broadcast storm many dinnferent types of MAC addresses have been identified as culprits. Each packet is identical.

The only way I can describe it is a device on the network sends our a broadcast and it starts to bounce around like a pinball machine and wont stop until it is unplugged.

This doesnt seem to happen with every broadcast just some, sometimes.



I have increased some on the broadcast storm settings on the switches to try and fix the problem. It seems to have helped some. But when the broadcast storm gets rolling it is just as bad as when the switches are set to the default settings.


If you have any questions please let me know......Any help would be greatly appreciated!!!

0
Comment
Question by:Mohonk
7 Comments
 
LVL 30

Assisted Solution

by:pgm554
pgm554 earned 500 total points
ID: 12542662
Could be spanning tree issue.But in any event look at this:

1 - Is there more than one frame type on the file servers, routers, print servers, etc.? If the answer is yes, then determine if every application and/or protocol on the network can run on one single frame type. Using a single frame type reduces redundant broadcast traffic.

2 - Is the network using many protocols, such as IPX, TCP/IP, LAT, SNA, NetBEUI, etc...? Is it possible to run all applications using a single protocol? If so, reconfigure the applications to run over this single protocol. Each protocol type requires it's own broadcasts, so minimizing the number of protocol families can lead to fewer broadcasts.

3 - External print servers and print server cards are known as 'plug-and-play' or 'ease of installation' devices. This simplicity comes with a price. Often, these devices are packaged with all of the major network protocols enabled, and sometimes multiple frame types are enabled. Most print servers have a management console or configuration screen to display what protocols and frame types are enabled. Disable all of the protocols and frame types not used on the network for printing.

4 - Most network switches default to enabling the spanning tree bridge protocol. Spanning tree is used for fault tolerance if redundant routes exist on the network. Unless your network is extra-mission-critical, it probably does not have redundant routes from every workstation. If possible, disable the spanning tree protocol. Spanning tree prevents loops on a network by sending out a 'hello' frame from each port every 2 seconds, which then gets resolved by every bridge or switch on the network. On a network with many switched nodes, a misconfiguration of the spanning tree protocols can create MANY broadcasts!

5- Make sure the WAN devices or routers have spoofing and/or filtering enabled. Contact your router manufacturer for specific functionality. The goal is to reduce the amount of broadcasts traversing the LAN and WAN, and to help conserve buffering memory inside the routers.

6- Have a network baseline analysis performed by an impartial 3rd party. A properly executed analysis will define protocols in use, identify problematic nodes, and give other pertinent information relating to the network's overall performance at all layers.
0
 
LVL 5

Accepted Solution

by:
muhalok earned 500 total points
ID: 12542678
Looks like a duplicate IP or a ambiguous LAN Path (STP is not enabled)

Steps to try in order to solve it:

1. Go and check your paths (links) between switches - when u have the loop on the main swtich, you have all the lights blinking madly!
So go and disconnect each cable 1 by 1, wait for each cable for a few moments - see that the lights stop blinking. In case they have - you found the end of the problematic link. Go with it forward and see where it ends - then check on that switch the same thing.

By the end of the procedure you will have a LOOP, which means that you connected 2 switches (2 ends ) with more than 1 cable (ambiguous path).

2. Check for different MAC addresses for the same IP address in arp table : go to several PCs and open a CMD then type "arp -a" in order to print the arp table, then look for same IP records.

Good luck.

PS: Source of the broadcast can help you (seen in sniffer) - find that MAC or IP and check the arp table there (arp -a).

0
 
LVL 9

Assisted Solution

by:cooledit
cooledit earned 500 total points
ID: 12544442
What kinda OS is on the clients ??..
Is it mainly using the DNS ?.
Is WIns installed.....

Are all clients using the Wins server .....

How many network connections:
12*24=300 users ?.. one large IP broadcast domain...
0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 
LVL 4

Assisted Solution

by:Beldoran
Beldoran earned 500 total points
ID: 12549650
We had a similar problem and eventually (many many months) tracked it down to a faulty backplane on the switch.

Bel
0
 

Author Comment

by:Mohonk
ID: 12551102
Really thats interesting Beldoran. However we had this problem and put this HP in to try and solve it. So it was happening even with another switch. Something HP tech support told us to try is set all ports to 100Full Duplex instead of auto. So were going to give that a shot.
0
 

Author Comment

by:Mohonk
ID: 12644821
Thanks for all the help. I have narrowed down the problem to the following:

All broadcast storm packets originated either from the cisco 1100 WAPs or clients using these access points. As i removed these access points from the network the problem slowed down. When I removed all of the access points the problem subsided. I have yet to call Cisco to see if there is a fix. But the one time i plugged an AP back into the network it blew up again.

I will split the points between all who commented. Thanks

0
 
LVL 9

Expert Comment

by:cooledit
ID: 12681588
hi, there

what you could try is to simply create a second VLAN for the Access Point's then they will be on there on broadcast doamin...
0

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This month, Experts Exchange’s free Course of the Month is focused on CompTIA IT Fundamentals.
How to fix a SonicWall Gateway Anti-Virus firewall blocking automatic updates to apps like Windows, Adobe, Symantec, etc.
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
Internet Business Fax to Email Made Easy - With  eFax Corporate (http://www.enterprise.efax.com), you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, f…

850 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question