Network traffic Problems - Please help URGENT

We have a reasonably complex network in the office. For structural reasons it goes through a number of switches with cabling going everywhere. We has 2 MS SQL servers, 1 MYSQL server, about 5 ISS servers on Win2003 machines, and 2 Asterisk Voip servers.
We also have 4 20mb ADSL connections.
Just recent the INTERNAL network has become a total traffic jam, with daily total blocks at around 14:10, when I have to switch off all the switches 'cause nothing works any more.
I am a programmer/analyst, etc with about 30 years experience, but I'm NOT a networking expert.
We are located in Rome, and I have phoned around but apart from selling us new switches, no one has solved the problem.
I wondered if, even under some form of financial reimbursement, someone could guide me through the installation of some monitoring software (like WireShark), which could log traffic for a period, then I could send them the log, and they could help me pinpoint the problem.

Any offers pls?
Francesco Facco de Lagarda
francesco@delagada.com
fdl333Asked:
Who is Participating?
Advertise Here
 
BasementCatConnect With a Mentor Commented:
The thing about arp is that it's a layer 2 protocol, meaning it can't be routed - so that traffic can't leave your network and also cannot have been generated outside your network.  I'm really not sure why you're seeing this much arp traffic - it's definitely not normal.  One thing you can try is to make sure that spanning tree is enabled on all of your switches, and that you have no cables with both ends plugged into the same switch - switching loops can sometimes cause floods of broadcast traffic.  Also, what devices have the IPs 192.0.1.57 and 192.0.1.86?  If they're non-critical, try disconnecting them from the network when you start seeing this traffic again and see if it stops the traffic, as they could potentially be the origin of the issue
0
 
evgeny_f31Commented:
You can use Cacti for snmp monitoring of your switches,
if your switches are managed -  they might have basic monitoring built in, or at least you can check the logs for errors, exessive broadcasts...
0
 
ujitnosCommented:
As you have mentioned that at about 14:10 you face this issue, is it possible for you to check if any kind of updates are happening internally, like Antivirus update, MS update etc.

0
Improve Your Query Performance Tuning

In this FREE six-day email course, you'll learn from Janis Griffin, Database Performance Evangelist. She'll teach 12 steps that you can use to optimize your queries as much as possible and see measurable results in your work. Get started today!

REGISTER NOW
 
fdl333Author Commented:
Sorry No, it's not updates ... the whole network locks down solid. I'm sure there's one rogue machine but I cant find it!
Thanks evgeny_f21, I'll look into it right away. How does it differ from WireShark (I dont know either)..?
0
 
BasementCatCommented:
If your switches are locking up due to some activity on the network, I would be willing to bet that you're seeing excessive broadcast traffic.  If that is the case, setting up a machine with Wireshark on it should allow you to pinpoint the source of the traffic, and may not require any special configuration of your switches.

If the problem is not caused by broadcast traffic but is still caused by traffic of some sort, you should still be able to capture it with wireshark and analyze it but it gets somewhat more involved.  In that case, it's important to make the distinction between switches and hubs.

If you're sure you have switches and NOT hubs, you will have to configure at least one of the switches with a "monitor port", which re-broadcasts all of the switch traffic out that port so it can be captured with a packet sniffer.  This requires a managed switch - if your switches are unmanaged, you're out of luck with this approach.

However, if you do have hubs and not switches then the first approach I suggested should work, as hubs broadcast all incoming traffic out all ports, which in a network like yours could cause so many collisions that it simply stops the network.
0
 
fdl333Author Commented:
Thanks BasementCat, that's exactly what I've been thinking. I also  thinki it's broadcast. The switches are some 3COM Managed switches, and they have their own little interface. I installed Cacti as suggested by evgeny_f31 but can't seem to get it to recognize the switches. They are 3Com Baseline Switch 2928-SFP Plus Software Version Release 1101P10.
Unfortunately I know very little about network analysis, I've always been on the software analysis/development and hardware for fun side!

I am quite willing to let you connect to the wireshark machine or even to the switches web management .
0
 
BasementCatCommented:
fdl333, I have actually done this on 3com switches before - it's really quite easy to set up.  If the traffic that is being a problem is actually broadcast traffic, then all you need to do is run wireshark on any machine that is directly connected to a switch that is affected by this problem.  To create a network traffic dump for others to analyze:
Download and install wireshark
From the "capture" menu, choose "options"
Make sure the correct network interface is selected (there probably will only be one)
Choose a location for the capture file under the "capture files" heading (it may get pretty huge)
Under "Name resolution", uncheck all of the checkboxes
Click the "start" button
Do this sometime before you usually see that burst in traffic, and you can stop wireshark once it has gone on for a little while, all you have to do is close the program and confirm if it asks you to save the capture file.  Once you have the capture file, anyone with a copy of wireshark can open it and analyze it.  If you'd like to do a capture I'd be happy to analyze it for you.
0
 
fdl333Author Commented:
BasementCat you're a real angel !
I'll do exactly that today (european time)
Thanks and speak soon..
I played with wireshark yesterday and I noticed that one of my ADSL routers sends about 70 "who is xxx.xxx.xx.xxx. reply to (it's own ip add)" is this normal?
0
 
fdl333Author Commented:
Thanks BasementCat!
I put it on our website 'cause it's 64M and I don't think I can u/l it here
http://www.timoholding.com/fdl/SharkForBasementCat.rar
IF you see a lot of traffic from 192.0.0.23 it's my pc downloading from Giganews ...

Sort this one out and you'll definetly get all the points!

0
 
BasementCatCommented:
Here's what I'm seeing so far:  initially, you're getting a huge flood of packets with the TCP reset flag set from 195.22.200.178:80 to 192.0.1.22 on either port 50126 or 50128, but I don't think that this is the source of your problem - it only seems to occur once.  Next, I'm seeing a flood of ARP requests (several million of them, around 10,000 per second) for 192.0.1.57 and 192.0.1.86 from either 192.0.0.64 (20:fd:f1:08:98:5c) or 192.0.0.66 (20:fd:f1:08:7f:86), this makes up most of that packet capture.  Those mac address ranges seem to belong to 3com europe - I would start by tracking down the devices that have those mac addresses, or those IPs, starting with any 3com equipment.
0
 
fdl333Author Commented:
Yep, last night I captured an enormous file of
"who has (ipaddres) tell (address of two of the 3com switches)"
It stopped when I rebooted one of our routers.
But it's really odd, 'cause we bought the three 3coms (ip's .64, .65 and .66) to try to solve the problem!
Why is there so much "who as xxxx" ARP traffic? II don't understand networking at this level, but it looks like an attack from outside to me
0
 
fdl333Author Commented:
You have ALL been very helpful. Just one last thing. Can anyone tell me HOW to configure Cactii tfor the 3 com switches. They are 3Com Baseline Switch 2928-SFP Plus Software Version Release 1101P10.
0
 
ujitnosCommented:
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.