Solved

Network traffic Problems - Please help URGENT

Posted on 2011-03-01
13
1,221 Views
Last Modified: 2012-05-11
We have a reasonably complex network in the office. For structural reasons it goes through a number of switches with cabling going everywhere. We has 2 MS SQL servers, 1 MYSQL server, about 5 ISS servers on Win2003 machines, and 2 Asterisk Voip servers.
We also have 4 20mb ADSL connections.
Just recent the INTERNAL network has become a total traffic jam, with daily total blocks at around 14:10, when I have to switch off all the switches 'cause nothing works any more.
I am a programmer/analyst, etc with about 30 years experience, but I'm NOT a networking expert.
We are located in Rome, and I have phoned around but apart from selling us new switches, no one has solved the problem.
I wondered if, even under some form of financial reimbursement, someone could guide me through the installation of some monitoring software (like WireShark), which could log traffic for a period, then I could send them the log, and they could help me pinpoint the problem.

Any offers pls?
Francesco Facco de Lagarda
francesco@delagada.com
0
Comment
Question by:fdl333
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 4
  • 2
  • +1
13 Comments
 
LVL 4

Expert Comment

by:evgeny_f31
ID: 35005332
You can use Cacti for snmp monitoring of your switches,
if your switches are managed -  they might have basic monitoring built in, or at least you can check the logs for errors, exessive broadcasts...
0
 
LVL 10

Expert Comment

by:ujitnos
ID: 35005611
As you have mentioned that at about 14:10 you face this issue, is it possible for you to check if any kind of updates are happening internally, like Antivirus update, MS update etc.

0
 

Author Comment

by:fdl333
ID: 35009331
Sorry No, it's not updates ... the whole network locks down solid. I'm sure there's one rogue machine but I cant find it!
Thanks evgeny_f21, I'll look into it right away. How does it differ from WireShark (I dont know either)..?
0
Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

 
LVL 1

Expert Comment

by:BasementCat
ID: 35023438
If your switches are locking up due to some activity on the network, I would be willing to bet that you're seeing excessive broadcast traffic.  If that is the case, setting up a machine with Wireshark on it should allow you to pinpoint the source of the traffic, and may not require any special configuration of your switches.

If the problem is not caused by broadcast traffic but is still caused by traffic of some sort, you should still be able to capture it with wireshark and analyze it but it gets somewhat more involved.  In that case, it's important to make the distinction between switches and hubs.

If you're sure you have switches and NOT hubs, you will have to configure at least one of the switches with a "monitor port", which re-broadcasts all of the switch traffic out that port so it can be captured with a packet sniffer.  This requires a managed switch - if your switches are unmanaged, you're out of luck with this approach.

However, if you do have hubs and not switches then the first approach I suggested should work, as hubs broadcast all incoming traffic out all ports, which in a network like yours could cause so many collisions that it simply stops the network.
0
 

Author Comment

by:fdl333
ID: 35024568
Thanks BasementCat, that's exactly what I've been thinking. I also  thinki it's broadcast. The switches are some 3COM Managed switches, and they have their own little interface. I installed Cacti as suggested by evgeny_f31 but can't seem to get it to recognize the switches. They are 3Com Baseline Switch 2928-SFP Plus Software Version Release 1101P10.
Unfortunately I know very little about network analysis, I've always been on the software analysis/development and hardware for fun side!

I am quite willing to let you connect to the wireshark machine or even to the switches web management .
0
 
LVL 1

Expert Comment

by:BasementCat
ID: 35032703
fdl333, I have actually done this on 3com switches before - it's really quite easy to set up.  If the traffic that is being a problem is actually broadcast traffic, then all you need to do is run wireshark on any machine that is directly connected to a switch that is affected by this problem.  To create a network traffic dump for others to analyze:
From the "capture" menu, choose "options"
Make sure the correct network interface is selected (there probably will only be one)
Choose a location for the capture file under the "capture files" heading (it may get pretty huge)
Under "Name resolution", uncheck all of the checkboxes
Click the "start" button
Do this sometime before you usually see that burst in traffic, and you can stop wireshark once it has gone on for a little while, all you have to do is close the program and confirm if it asks you to save the capture file.  Once you have the capture file, anyone with a copy of wireshark can open it and analyze it.  If you'd like to do a capture I'd be happy to analyze it for you.
0
 

Author Comment

by:fdl333
ID: 35033903
BasementCat you're a real angel !
I'll do exactly that today (european time)
Thanks and speak soon..
I played with wireshark yesterday and I noticed that one of my ADSL routers sends about 70 "who is xxx.xxx.xx.xxx. reply to (it's own ip add)" is this normal?
0
 

Author Comment

by:fdl333
ID: 35036150
Thanks BasementCat!
I put it on our website 'cause it's 64M and I don't think I can u/l it here
http://www.timoholding.com/fdl/SharkForBasementCat.rar
IF you see a lot of traffic from 192.0.0.23 it's my pc downloading from Giganews ...

Sort this one out and you'll definetly get all the points!

0
 
LVL 1

Expert Comment

by:BasementCat
ID: 35041217
Here's what I'm seeing so far:  initially, you're getting a huge flood of packets with the TCP reset flag set from 195.22.200.178:80 to 192.0.1.22 on either port 50126 or 50128, but I don't think that this is the source of your problem - it only seems to occur once.  Next, I'm seeing a flood of ARP requests (several million of them, around 10,000 per second) for 192.0.1.57 and 192.0.1.86 from either 192.0.0.64 (20:fd:f1:08:98:5c) or 192.0.0.66 (20:fd:f1:08:7f:86), this makes up most of that packet capture.  Those mac address ranges seem to belong to 3com europe - I would start by tracking down the devices that have those mac addresses, or those IPs, starting with any 3com equipment.
0
 

Author Comment

by:fdl333
ID: 35042229
Yep, last night I captured an enormous file of
"who has (ipaddres) tell (address of two of the 3com switches)"
It stopped when I rebooted one of our routers.
But it's really odd, 'cause we bought the three 3coms (ip's .64, .65 and .66) to try to solve the problem!
Why is there so much "who as xxxx" ARP traffic? II don't understand networking at this level, but it looks like an attack from outside to me
0
 
LVL 1

Accepted Solution

by:
BasementCat earned 500 total points
ID: 35044118
The thing about arp is that it's a layer 2 protocol, meaning it can't be routed - so that traffic can't leave your network and also cannot have been generated outside your network.  I'm really not sure why you're seeing this much arp traffic - it's definitely not normal.  One thing you can try is to make sure that spanning tree is enabled on all of your switches, and that you have no cables with both ends plugged into the same switch - switching loops can sometimes cause floods of broadcast traffic.  Also, what devices have the IPs 192.0.1.57 and 192.0.1.86?  If they're non-critical, try disconnecting them from the network when you start seeing this traffic again and see if it stops the traffic, as they could potentially be the origin of the issue
0
 

Author Comment

by:fdl333
ID: 35174468
You have ALL been very helpful. Just one last thing. Can anyone tell me HOW to configure Cactii tfor the 3 com switches. They are 3Com Baseline Switch 2928-SFP Plus Software Version Release 1101P10.
0
 
LVL 10

Expert Comment

by:ujitnos
ID: 35175337
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
ADCs have gained traction within the last decade, largely due to increased demand for legacy load balancing appliances to handle more advanced application delivery requirements and improve application performance.
Internet Business Fax to Email Made Easy - With  eFax Corporate (http://www.enterprise.efax.com), you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, f…
Monitoring a network: how to monitor network services and why? Michael Kulchisky, MCSE, MCSA, MCP, VTSP, VSP, CCSP outlines the philosophy behind service monitoring and why a handshake validation is critical in network monitoring. Software utilized …

729 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question