Solved

Network traffic Problems - Please help URGENT

Posted on 2011-03-01
13
1,212 Views
Last Modified: 2012-05-11
We have a reasonably complex network in the office. For structural reasons it goes through a number of switches with cabling going everywhere. We has 2 MS SQL servers, 1 MYSQL server, about 5 ISS servers on Win2003 machines, and 2 Asterisk Voip servers.
We also have 4 20mb ADSL connections.
Just recent the INTERNAL network has become a total traffic jam, with daily total blocks at around 14:10, when I have to switch off all the switches 'cause nothing works any more.
I am a programmer/analyst, etc with about 30 years experience, but I'm NOT a networking expert.
We are located in Rome, and I have phoned around but apart from selling us new switches, no one has solved the problem.
I wondered if, even under some form of financial reimbursement, someone could guide me through the installation of some monitoring software (like WireShark), which could log traffic for a period, then I could send them the log, and they could help me pinpoint the problem.

Any offers pls?
Francesco Facco de Lagarda
francesco@delagada.com
0
Comment
Question by:fdl333
  • 6
  • 4
  • 2
  • +1
13 Comments
 
LVL 4

Expert Comment

by:evgeny_f31
ID: 35005332
You can use Cacti for snmp monitoring of your switches,
if your switches are managed -  they might have basic monitoring built in, or at least you can check the logs for errors, exessive broadcasts...
0
 
LVL 10

Expert Comment

by:ujitnos
ID: 35005611
As you have mentioned that at about 14:10 you face this issue, is it possible for you to check if any kind of updates are happening internally, like Antivirus update, MS update etc.

0
 

Author Comment

by:fdl333
ID: 35009331
Sorry No, it's not updates ... the whole network locks down solid. I'm sure there's one rogue machine but I cant find it!
Thanks evgeny_f21, I'll look into it right away. How does it differ from WireShark (I dont know either)..?
0
 
LVL 1

Expert Comment

by:BasementCat
ID: 35023438
If your switches are locking up due to some activity on the network, I would be willing to bet that you're seeing excessive broadcast traffic.  If that is the case, setting up a machine with Wireshark on it should allow you to pinpoint the source of the traffic, and may not require any special configuration of your switches.

If the problem is not caused by broadcast traffic but is still caused by traffic of some sort, you should still be able to capture it with wireshark and analyze it but it gets somewhat more involved.  In that case, it's important to make the distinction between switches and hubs.

If you're sure you have switches and NOT hubs, you will have to configure at least one of the switches with a "monitor port", which re-broadcasts all of the switch traffic out that port so it can be captured with a packet sniffer.  This requires a managed switch - if your switches are unmanaged, you're out of luck with this approach.

However, if you do have hubs and not switches then the first approach I suggested should work, as hubs broadcast all incoming traffic out all ports, which in a network like yours could cause so many collisions that it simply stops the network.
0
 

Author Comment

by:fdl333
ID: 35024568
Thanks BasementCat, that's exactly what I've been thinking. I also  thinki it's broadcast. The switches are some 3COM Managed switches, and they have their own little interface. I installed Cacti as suggested by evgeny_f31 but can't seem to get it to recognize the switches. They are 3Com Baseline Switch 2928-SFP Plus Software Version Release 1101P10.
Unfortunately I know very little about network analysis, I've always been on the software analysis/development and hardware for fun side!

I am quite willing to let you connect to the wireshark machine or even to the switches web management .
0
 
LVL 1

Expert Comment

by:BasementCat
ID: 35032703
fdl333, I have actually done this on 3com switches before - it's really quite easy to set up.  If the traffic that is being a problem is actually broadcast traffic, then all you need to do is run wireshark on any machine that is directly connected to a switch that is affected by this problem.  To create a network traffic dump for others to analyze:
From the "capture" menu, choose "options"
Make sure the correct network interface is selected (there probably will only be one)
Choose a location for the capture file under the "capture files" heading (it may get pretty huge)
Under "Name resolution", uncheck all of the checkboxes
Click the "start" button
Do this sometime before you usually see that burst in traffic, and you can stop wireshark once it has gone on for a little while, all you have to do is close the program and confirm if it asks you to save the capture file.  Once you have the capture file, anyone with a copy of wireshark can open it and analyze it.  If you'd like to do a capture I'd be happy to analyze it for you.
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 

Author Comment

by:fdl333
ID: 35033903
BasementCat you're a real angel !
I'll do exactly that today (european time)
Thanks and speak soon..
I played with wireshark yesterday and I noticed that one of my ADSL routers sends about 70 "who is xxx.xxx.xx.xxx. reply to (it's own ip add)" is this normal?
0
 

Author Comment

by:fdl333
ID: 35036150
Thanks BasementCat!
I put it on our website 'cause it's 64M and I don't think I can u/l it here
http://www.timoholding.com/fdl/SharkForBasementCat.rar
IF you see a lot of traffic from 192.0.0.23 it's my pc downloading from Giganews ...

Sort this one out and you'll definetly get all the points!

0
 
LVL 1

Expert Comment

by:BasementCat
ID: 35041217
Here's what I'm seeing so far:  initially, you're getting a huge flood of packets with the TCP reset flag set from 195.22.200.178:80 to 192.0.1.22 on either port 50126 or 50128, but I don't think that this is the source of your problem - it only seems to occur once.  Next, I'm seeing a flood of ARP requests (several million of them, around 10,000 per second) for 192.0.1.57 and 192.0.1.86 from either 192.0.0.64 (20:fd:f1:08:98:5c) or 192.0.0.66 (20:fd:f1:08:7f:86), this makes up most of that packet capture.  Those mac address ranges seem to belong to 3com europe - I would start by tracking down the devices that have those mac addresses, or those IPs, starting with any 3com equipment.
0
 

Author Comment

by:fdl333
ID: 35042229
Yep, last night I captured an enormous file of
"who has (ipaddres) tell (address of two of the 3com switches)"
It stopped when I rebooted one of our routers.
But it's really odd, 'cause we bought the three 3coms (ip's .64, .65 and .66) to try to solve the problem!
Why is there so much "who as xxxx" ARP traffic? II don't understand networking at this level, but it looks like an attack from outside to me
0
 
LVL 1

Accepted Solution

by:
BasementCat earned 500 total points
ID: 35044118
The thing about arp is that it's a layer 2 protocol, meaning it can't be routed - so that traffic can't leave your network and also cannot have been generated outside your network.  I'm really not sure why you're seeing this much arp traffic - it's definitely not normal.  One thing you can try is to make sure that spanning tree is enabled on all of your switches, and that you have no cables with both ends plugged into the same switch - switching loops can sometimes cause floods of broadcast traffic.  Also, what devices have the IPs 192.0.1.57 and 192.0.1.86?  If they're non-critical, try disconnecting them from the network when you start seeing this traffic again and see if it stops the traffic, as they could potentially be the origin of the issue
0
 

Author Comment

by:fdl333
ID: 35174468
You have ALL been very helpful. Just one last thing. Can anyone tell me HOW to configure Cactii tfor the 3 com switches. They are 3Com Baseline Switch 2928-SFP Plus Software Version Release 1101P10.
0
 
LVL 10

Expert Comment

by:ujitnos
ID: 35175337
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

This is an article about my experiences with remote access to my clients (so that I may serve them) and eventually to my home office system via Radmin Remote Control. I have been using remote access for over 10 years and have been improving my metho…
ADCs have gained traction within the last decade, largely due to increased demand for legacy load balancing appliances to handle more advanced application delivery requirements and improve application performance.
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now