Network Unresponsive/timeout Issues

Recently our ethernet network began experiencing a problem in which connection timeouts are occuring between the workstations and multiple servers.  We are using mulitiple Win2k server and one Novell 3 server, all over which are losing connectivity.  I had thought the the problem was narrowed down to a bad cisco switch, however after replacing the switch the problem still exists.  I can ping each server and watch as it replys for about 5 minutes strait, then will begin timing out. It will timeout for anywhere from 5 to 60 seconds before "coming back". I have tried just about every network analyzer I can find and do not notice any strange activity .  Any help, including anything suspicious to look for would be helpful.
LVL 1
fakir420Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Robing66066Commented:
Sounds like it might be a broadcast storm brought on by a cabling loop.  Check your connections between switches.  If you have a situation where you can reach the same switch by two different paths, you may have a cabling loop.  (You may not, the lines may be trunked, but you can check this on the switch).  If you find such a loop, remove one of the paths and see if your problem goes away.

Good luck.  
0
ShineOnCommented:
It could also be broadcast storm brought about by a worm like Welchia or MSBlaster.  Are all of your servers and workstations patched to current service packs and security hotfixes?  Do you run an antivirus realtime scanner?  Are the antivirus signature files kept up-to-date?

If the answer to ANY of those questions is "no" then you should suspect an infection and start tracking it down and cleaning it up.
0
Steve JenningsIT ManagerCommented:
Any other switches on the network . . . running spanning tree?

When the ping fails, does the switch show a valid mac address for the machine that's plugged into it? Does the port show UP?

Check for stuff ShineOn and RObing66066 mention.

Good luck,
Steve
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

fakir420Author Commented:
I do not believe it is an RPC worm as I have looked at netowrk traffic and have not seen any signs of this type of traffic.  No changes have been made to cabling so I do not believe it could be a looping issues.  I first beleieved it had something to do with DNS, is there any kind of DNS specific broadcast/attack?  I can also supply a network trace file if needed.
0
rogue_phoenixCommented:


When the timeouts happen, do they occur on the clients simultaneously or each on its own?

are your interfaces/switch ports hard set, or do you rely on autonegotiation?
http://www.cisco.com/warp/public/473/3.html

If it's DNS related 1). it'll show up in the trace, 2).  it'd be a timeout more likely than a broadcast attack. Are there DNS proxies in the network that clients would be going through?

If none of the above leads anywhere, I'd like to see a network diagram and a wide open (no filters) trace (snoop, tcpdump or ethereal are preferable to netmon). Preferably from multiple clients & servers simultaneously, with either sync'd clocks or notes on the time differential so I can tell whats happening on each end, but at least one pair of a client-server mismatch would be the minimum.


0
DRVVCommented:
Just for reference, I just posted a similiar problem on my network (Question Q_20790421 sporadic major packet loss).

It will be interesting to see if the cause of the problem is the same for both of our networks.
0
stevenlewisCommented:
rerun the ping test, and make note of the time, and then check event viewer on the machines to see if anything unusual shows (like services stopping and starting)
0
fakir420Author Commented:
I see also now that on at least one server when doing netstat there is an established connection to unkown.level3.net port 80.  I have read about an exploit that alters registry settings/creates new hosts file, but I did find this on that machine.  I don't know why there would otherwise be an connection or if this has anything to do with the problem.

One software package shows traffic on the MS/TCP Loopback Interace on the servers rise to very high levels at the time of unresponsiveness.  I'm not sure what this shows as I don't really see that much traffic on the ethernet interface.
0
chicagoanCommented:
although incidents.org doesn't show any reports of probes from unknown.level3.net, it did ring a bell with me and there are lots of folks in the group reporting INCOMING connections from unkown.level3.net:80 which certainly should be suspicious.

This has all the earmarks of a worm. What you set a span port up on your switch and does your sniffer have an adaptor it can get into promiscuous mode? Have you set your firewall to log at debug level and analyzed the syslog from that?
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
fakir420Author Commented:
I've posted trace files from two seperate stations (made with Ethereal) in libpcap format.  The compressed file is about 25mb and can be downloaded from http://tdec-fish.dyndns.org/traces.rar
0
chicagoanCommented:
took a look and other than suggesting you ban e-donkey there's not much to go on...
If you could put the sniffer on a span port or the wan segment it might help
 
0
ShineOnCommented:
What're you allowing edonkey for anyway?  You should likewise ban kazaa, WinMX, IRC and any other sharing stuff... beyond being a source of unecessary traffic, it can expose you to legal issues.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Network Analysis

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.