Entire network goes down once primary switch is connected, tried rebooting DC, changing router, and switch. Help!

Hi Everyone,

We are having a big problem on our network.  We are a non-profit organization supporting about 30 users here running various children's tutoring programs, and today the network became completely unstable.

Here is the sitaution, all of a sudden the network slowed to a complete crawl.  Internet connection went up and down, shared printers and file shares started disappearing, etc.  We have a spare router and switch, so the first things we tried was swapping out a new router and switch and rebooting the DC.  This didn't help.

For troubleshooting, one thing I've noticed is that if we plug the server straight into the router (bypass the switch) then it works just fine and wireless clients can connect.  This leads me to believe the problem is in the switch connected computer someplace.  Again, we tried changing the switch itself but that didn't help.  If we boot up the router with just the server plugged into it, it works flawlessly and wireless clients are fine.  THE MINUTE you connect the switch (24 port) into the router, everything goes nuts.  The server and wireless clients begin to drop connections to the network, etc.  

I've checked the event logs on the DC and it does show several events saying that the Domain Controller lost n election to be the master browser.  It says it lost an election to FSERVER, which is itself!  

I'm wondering if perhaps there is a malware infection on one of the machines that could do this, or a shorted out cat5 cable going into the switch or something?  Can anyone provide any help?

Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

snafumasterDirector of Information TechnologyCommented:
The switch in question...have you tried unplugging all the cables and reconnecting one by one?  Is this a managed switch?  Does it have an uplink switch?
what kind of routers and switches are you using?
JsmplyAuthor Commented:
Router one is a Linksys WRT-54G paired with a basic Netgear 24 port switch.  The second spare set is a Linksys WRT120N paired with a D-link standard 24 port switch, nothing fancy.

The primary WRT54G with the Netgear 24 port switch have been running flawlessly for several years.  This just started today with no additions or subtractions from the network.  

Have not tried disconnecting cables one by one yet, that is the next idea.  Several of those 24 ports run downstairs into other hubs and would create quite a mess to sort out as it was never documented when it was run.  

Anyone have any ideas?

From military to cyber security

If you've served in the military or worked in law enforcement, starting a career in cyber security may be a great move for you. Discover what cyber security professionals do, and how former military or law enforcement professionals can get started on this career path.

snafumasterDirector of Information TechnologyCommented:
The cables downstair are the ones I'd be most concerned with.  I have had people in my organization plug the wrong end of a router in to a wall jack...quickly brought the LAN down.  
JsmplyAuthor Commented:
No cables have changed though.  All employees are tutoring volunteers, etc.  No one touches that stuff.  
Sounds like you have some type of loop or broadcast storm that may have been started by a failing piece of hardware somewhere.  Your best best may be to continue as planned with unplugging port by port and see what happens.
snafumasterDirector of Information TechnologyCommented:
On one of the machines that won't network properly, go to the command line and do an ipconfig/all to see from where it is getting its DHCP.  That may quickly shed some light on the issue.
JsmplyAuthor Commented:
Unplugging cables one at a time now.  
As the others have said, your problem is a loop somewhere on the network.   I have had this twice on my networks.  Once was a $50 switch we used to add an extra port to an office.  Switch was bad, network, went down.    The second time a night janitor was moving things and thought he knocked a cable out of the wall, so he plugged it back into a jack.  Problem was the other end was already connected.  Loop caused traffic bringing everything down.

What are the activity lights showing on the switch?   If they are lit solid, that is not likley a normal condition.  Disconnect cables one at a time to see if activity returns to normal.   If the uplink cables are documented, start with them.    Doing this will narrow the problem to one part of your network.  
If your network was well documented you might be able to put a sniffer on a monitor port to find the offending device.  In our case, the traffic tied up the switch and prevented us form setting up the monitor port.

Good Luck

Once you figure it out.  Spend the time to document your network.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
JsmplyAuthor Commented:
Well we got it figured out, turns out we had two problems going on.  

One: A user had an Apple Print Server and AP device plugged in downstairs.  It's one of those devices that you plug into the wall and plug into the wall and then plug in a network cable and it sets up a little wireless network and a USB printer can be connected.  Problem is, they moved the plug from the wall into an APC UPS.  I've seen that cause weird problems with powerline networking equipment.  Sure enough, unplugging that fixed the issue!  

Two: Once we plugged the main switch back in, we started getting issues again.  Turns out, when the network went down this morning, a volunteer saw a cable that was loose near the switch and plugged it in thinking it would fix the problem.  As you guessed . . . that cable was already connected on the other end (it used to be for a printer that was removed) and sure enough that caused problem number two.

Once both of these were removed, restarting the switch seemed to do the trick.  Now we can return the spare equipment (since this is a charity with a limited budget), thanks everyone!

Thanks everyone!  
JsmplyAuthor Commented:
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Server 2003

From novice to tech pro — start learning today.