Solved

Troubleshooting Broadcast Storm

Posted on 2004-10-18
2
2,193 Views
Last Modified: 2010-08-05
We have a small network, 100+/- clients (almost all xp, 1-2 Win98) with W2K domain controller, couple W2K servers, couple Server 2003 servers, and a linux gateway supplied by our ISP.

The network is all a single subnet with 4 Netgear FSM750s switches. 2 of the switches in the rack with the servers, the other two in different areas of the building. The 2 remote switches are connected to the main server room switch through the GB ports, one fiber (gbic) one copper. (side note.... although we have never had trouble with the main switch till now we spent weeks trying to get the second gbic to work with the fiber run we have in place and were not able to get it going... this despite having the line tested for loss and going through multiple gbic adapters etc, etc hence the one copper backbone)

There are a minimal number of other switches or hubs, and just a single WAP we turn on occasionally to provide wireless connectivity to a conference room.

Everything worked fine untill recently when we added the 4th FSM750s into the rack in the server room. This was done to pick up some extra runs and accomadate new clients. (there was previously a small 'dumb' netgear switch that was connected to the main switch through a patch cable and this new switch replaced that one.)

We first tried to add the new switch to the main switch as a slave, through the rear stacking port. This resulted in us not being able to access the main switch from the web control panel and we were also not able to ping the switch on the network, althoug it appeared to be working normally. We connecet throught the console and the IP was the same as it had been previously (192.168.1.30, a static address in our set aside range for network equipment). Although we could manage the switch through the console, we could not see the slave switch in the console.

After trying different IPs and various other solutions, we disconnected the stacking cable and just connected the new switch to main switch as a temporaty work around. Everything seemed to be working again and we could now access the main switch again from the web control panel. (the two are connected with a patch cable)

After about 24-48 hours, we had a broadcast storm and had to reset the switch. The next day it happened again. We began to troubleshoot and covered all the obvious areas, eliminating what we could. Every day, about once a day, usually over night, the switch becomes unusable due to a broadcast storm.

Things we have tried:

resetting to factory defaults
removing the second switch
turning off port mirroring (our internet filtering software Ifilter)
double checking our anti virus software
looking for kids plugging patch cables into multiple jacks (it is a school)
looking for unauthorized machines

At this point I have begun to use some packet sniffing software to try and see what is actually going on but the problem seems very intermitent. I have noticed a lot of traffic on ports 137-138-139 inside the network but I don't have historical data to compare against to see if this is normal. Traffic does not seem to originate from a single source, I haven't even been able to find out which part of the network it is coming from.

In the past we never had problems with the network, it was always very solid (with the exception of the fiber which we never figured out)

We have recently taken down a norton anti virus server and switched to avg - norton hasn't been removed from all the clients but i can't believe this is the problem.

My questions are:

What is the likely hood that this is a bad switch?
Is there a virus that i may be missing?
what should i be looking for in captured packets?
what could cause a network storm on a regular but widely spaced interval?
0
Comment
Question by:blacksun-networks
2 Comments
 
LVL 15

Accepted Solution

by:
adamdrayer earned 500 total points
ID: 12347130
Ports 137-139 are Microsoft NetBIOS broadcasts.  If your computers are using broadcasting to resolve NetBIOS name, this could be thte cause of such a broadcast storm.

I don't know why, but sometimes WINS databases get corrupted.  If your servers have not been rebooted since the problem started, try rebooting all of your servers especially the ones running WINS.

Then you can alter thte clients method of resolving NetBIOS names.  It usually defaults to using WINS, then broadcasts.. but you can change this to ONLY USE WINS.  It's called Node Type 'P'

Set This Registry value:
HKLM\Services\CCS\Services\NetBT\Parameters\DHCPNodeType=2

do this on one client, and then run "ipconfig /all" and see if the Node Type has changed from Hybrid to Peer-to-Peer

if you are convinced that these storms are not caused by NetBT name resolution broadcasting, then you can run http://www.ethereral.com on one of the workstations.  Its a protocol analyzer that not only captures network I/O but also tries to interpret the requests as best it can and tell you what kind it is.
0
 

Author Comment

by:blacksun-networks
ID: 12373604
It appears that this was the problem -

We recently installed Faronics "Deep Freeze" which disables users from making changes to the computer. Our machines were set to use windows automatic update without SUS. Becasue the machines never actually 'saved' the update they downloaded it had a cascading effect over a period of a couple weeks so that the port to our ISP on the switch gets overloaded and this must somehow cause the broadcast storm. All traffic on the network was minimal with the exception of the 25 or so machines constantly hammering windows update.
0

Featured Post

Save the day with this special offer from ATEN!

Save 30% on the CV211 using promo code EXPERTS30 now through April 30th. The ATEN CV211 connects a laptop directly to any server allowing you instant access to perform data maintenance and local operations, for quick troubleshooting, updating, service and repair.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
USB on Hyper-V VM Server 2012R2 13 112
Interviewing System Administrator recommended questions 18 100
OpenVPN Speed limitation to only 10 mbps 7 225
forward schedule of change 1 73
Downtime reduced, data recovered by utilizing an Experts Exchange Business Account Challenge The United States Marine Corps employs more than 200,000 active-duty Marines with operations in four continents, all requiring complex networking system…
Resolve DNS query failed errors for Exchange
A short tutorial showing how to set up an email signature in Outlook on the Web (previously known as OWA). For free email signatures designs, visit https://www.mail-signatures.com/articles/signature-templates/?sts=6651 If you want to manage em…
In a recent question (https://www.experts-exchange.com/questions/29004105/Run-AutoHotkey-script-directly-from-Notepad.html) here at Experts Exchange, a member asked how to run an AutoHotkey script (.AHK) directly from Notepad++ (aka NPP). This video…

679 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question