Solved

Troubleshooting Broadcast Storm

Posted on 2004-10-18
2
2,165 Views
Last Modified: 2010-08-05
We have a small network, 100+/- clients (almost all xp, 1-2 Win98) with W2K domain controller, couple W2K servers, couple Server 2003 servers, and a linux gateway supplied by our ISP.

The network is all a single subnet with 4 Netgear FSM750s switches. 2 of the switches in the rack with the servers, the other two in different areas of the building. The 2 remote switches are connected to the main server room switch through the GB ports, one fiber (gbic) one copper. (side note.... although we have never had trouble with the main switch till now we spent weeks trying to get the second gbic to work with the fiber run we have in place and were not able to get it going... this despite having the line tested for loss and going through multiple gbic adapters etc, etc hence the one copper backbone)

There are a minimal number of other switches or hubs, and just a single WAP we turn on occasionally to provide wireless connectivity to a conference room.

Everything worked fine untill recently when we added the 4th FSM750s into the rack in the server room. This was done to pick up some extra runs and accomadate new clients. (there was previously a small 'dumb' netgear switch that was connected to the main switch through a patch cable and this new switch replaced that one.)

We first tried to add the new switch to the main switch as a slave, through the rear stacking port. This resulted in us not being able to access the main switch from the web control panel and we were also not able to ping the switch on the network, althoug it appeared to be working normally. We connecet throught the console and the IP was the same as it had been previously (192.168.1.30, a static address in our set aside range for network equipment). Although we could manage the switch through the console, we could not see the slave switch in the console.

After trying different IPs and various other solutions, we disconnected the stacking cable and just connected the new switch to main switch as a temporaty work around. Everything seemed to be working again and we could now access the main switch again from the web control panel. (the two are connected with a patch cable)

After about 24-48 hours, we had a broadcast storm and had to reset the switch. The next day it happened again. We began to troubleshoot and covered all the obvious areas, eliminating what we could. Every day, about once a day, usually over night, the switch becomes unusable due to a broadcast storm.

Things we have tried:

resetting to factory defaults
removing the second switch
turning off port mirroring (our internet filtering software Ifilter)
double checking our anti virus software
looking for kids plugging patch cables into multiple jacks (it is a school)
looking for unauthorized machines

At this point I have begun to use some packet sniffing software to try and see what is actually going on but the problem seems very intermitent. I have noticed a lot of traffic on ports 137-138-139 inside the network but I don't have historical data to compare against to see if this is normal. Traffic does not seem to originate from a single source, I haven't even been able to find out which part of the network it is coming from.

In the past we never had problems with the network, it was always very solid (with the exception of the fiber which we never figured out)

We have recently taken down a norton anti virus server and switched to avg - norton hasn't been removed from all the clients but i can't believe this is the problem.

My questions are:

What is the likely hood that this is a bad switch?
Is there a virus that i may be missing?
what should i be looking for in captured packets?
what could cause a network storm on a regular but widely spaced interval?
0
Comment
Question by:blacksun-networks
2 Comments
 
LVL 15

Accepted Solution

by:
adamdrayer earned 500 total points
Comment Utility
Ports 137-139 are Microsoft NetBIOS broadcasts.  If your computers are using broadcasting to resolve NetBIOS name, this could be thte cause of such a broadcast storm.

I don't know why, but sometimes WINS databases get corrupted.  If your servers have not been rebooted since the problem started, try rebooting all of your servers especially the ones running WINS.

Then you can alter thte clients method of resolving NetBIOS names.  It usually defaults to using WINS, then broadcasts.. but you can change this to ONLY USE WINS.  It's called Node Type 'P'

Set This Registry value:
HKLM\Services\CCS\Services\NetBT\Parameters\DHCPNodeType=2

do this on one client, and then run "ipconfig /all" and see if the Node Type has changed from Hybrid to Peer-to-Peer

if you are convinced that these storms are not caused by NetBT name resolution broadcasting, then you can run http://www.ethereral.com on one of the workstations.  Its a protocol analyzer that not only captures network I/O but also tries to interpret the requests as best it can and tell you what kind it is.
0
 

Author Comment

by:blacksun-networks
Comment Utility
It appears that this was the problem -

We recently installed Faronics "Deep Freeze" which disables users from making changes to the computer. Our machines were set to use windows automatic update without SUS. Becasue the machines never actually 'saved' the update they downloaded it had a cascading effect over a period of a couple weeks so that the port to our ISP on the switch gets overloaded and this must somehow cause the broadcast storm. All traffic on the network was minimal with the exception of the 25 or so machines constantly hammering windows update.
0

Featured Post

Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

Join & Write a Comment

FIPS stands for the Federal Information Processing Standardisation and FIPS 140-2 is a collection of standards that are generically associated with hardware and software cryptography. In most cases, people can refer to this as the method of encrypti…
Resolve DNS query failed errors for Exchange
It is a freely distributed piece of software for such tasks as photo retouching, image composition and image authoring. It works on many operating systems, in many languages.
Illustrator's Shape Builder tool will let you combine shapes visually and interactively. This video shows the Mac version, but the tool works the same way in Windows. To follow along with this video, you can draw your own shapes or download the file…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now