Troubleshooting network traffic
I have a fairly simple (flat) ethernet network that is being 'killed' by traffic during the work day. The only thing I've been able to identify so far is that our inbound internet traffic (T1) hits about 95% capacity as users login for the day and that level continues constant all day until about 5/6pm when users are logging out for the day. Outbound traffic never exceeds about 50% capacity and never stays that high. overnight, traffic is perfectly normal.
The other serious symptom is that the LAN appears to also be flooded during this same time period. Making the LAN almost un-usable. Any help would be greatly appreciated.
The other serious symptom is that the LAN appears to also be flooded during this same time period. Making the LAN almost un-usable. Any help would be greatly appreciated.
jayasanker
Are you using any kind of firewall/Antivirus in your network??
Do you have any hubs installed or the network is totally switched?
Do you have managed switches to check port statistics?
Do you have managed switches to check port statistics?
Lots of things you could try.
Wireshark is a packet sniffer that will let you snoop on all the packets going through the network, it can be a bit overwhelming but will show you every packet.
The other thing I'd look at is an application called Ntop, this will sniff traffic for you and graph it all up nicely so you can see if a particular host is going mental for some reason. (http://www.ntop.org/).
Also, does the switch have SNMP on it and is it managed, if so you can graph port usage and stuff with something like zenoss/cacti/zabbix etc.
Wireshark is a packet sniffer that will let you snoop on all the packets going through the network, it can be a bit overwhelming but will show you every packet.
The other thing I'd look at is an application called Ntop, this will sniff traffic for you and graph it all up nicely so you can see if a particular host is going mental for some reason. (http://www.ntop.org/).
Also, does the switch have SNMP on it and is it managed, if so you can graph port usage and stuff with something like zenoss/cacti/zabbix etc.
Usually I would say use a process of elimination, and just try and stager the users loggin in to see where the problem could potentially be.
It seems a litter weird, It could be dodgy software or malware, or even a single fault network card. Is a user using/downloading torrents?
I would suggest installing wireshark and running it on your network. You should be able to see what is happening. It will be HEAPS of info for your senario, but it might point you in the right direction to find out what is utilising your bandwidth.
www.wireshark.com
Alternatively to that, you could stay back one night (not sure how many PC's on your network) but one at a time, turn them on and log in. See if you can replicate the problem. Make sure you check the util after each computer you login. This should instantly give you a starting point
It seems a litter weird, It could be dodgy software or malware, or even a single fault network card. Is a user using/downloading torrents?
I would suggest installing wireshark and running it on your network. You should be able to see what is happening. It will be HEAPS of info for your senario, but it might point you in the right direction to find out what is utilising your bandwidth.
www.wireshark.com
Alternatively to that, you could stay back one night (not sure how many PC's on your network) but one at a time, turn them on and log in. See if you can replicate the problem. Make sure you check the util after each computer you login. This should instantly give you a starting point
Dude make sure that none of your clients have p2p software running in the background. Specially torrent apps. I had a network that almost went completely down because of one user that had utorrent. Every time she logged in it opened utorrent automatically and the network went nutz.
Hi,
We had the same problem with our customers and found a solution.
Can you try to do a - ipconfig /flushdns - when the problem occurs and see if it is resolved thereafter ?
you should also
Internet History : 100 Mb Limit
Large Send Offload on network card : Disabled
Bypass Proxcy for local adresses
Your primary DNS should be your DC
The problem is with the DNS cache becaming too old.
We had the same problem with our customers and found a solution.
Can you try to do a - ipconfig /flushdns - when the problem occurs and see if it is resolved thereafter ?
you should also
Internet History : 100 Mb Limit
Large Send Offload on network card : Disabled
Bypass Proxcy for local adresses
Your primary DNS should be your DC
The problem is with the DNS cache becaming too old.
ASKER
Cisco PIX firewall, Symantec EndPoint, HP procurve switches (ports don't show anything abnormal), have wireshark (lost knowing what to look for in packets), currently searching down users with 'crap' installed and finding nothing horrible yet. (lots of Chat clients, WeatherBug and assorted similar crap).
70 users.
70 users.
Still sounds like some kind of software.
Dont think it would be a hardware fault.
There may be a chance that a p2p or torrent client is exhausting your NAT table on the pix. That would cause extremely slow performance if any performance at all. Not sure of the command or even if you can view the nat table inside a pix - but it would clearly indicate a problem
Dont think it would be a hardware fault.
There may be a chance that a p2p or torrent client is exhausting your NAT table on the pix. That would cause extremely slow performance if any performance at all. Not sure of the command or even if you can view the nat table inside a pix - but it would clearly indicate a problem
70 users on a a single T1 doesn't sound too good to me...
zgluffria has a very good point.
t1 is 1544 kbps i believe.
Some math - 1544 / 70 users = 22 kbps per user (in a perfect situation), that probably means 11kbps in real world speeds (duplex/half duplex, actual throughput speeds etc etc)
11 - 22 kbps is not really sufficient by any means. If a couple of users are streaming music, watching youtube... bye bye bandwidth
t1 is 1544 kbps i believe.
Some math - 1544 / 70 users = 22 kbps per user (in a perfect situation), that probably means 11kbps in real world speeds (duplex/half duplex, actual throughput speeds etc etc)
11 - 22 kbps is not really sufficient by any means. If a couple of users are streaming music, watching youtube... bye bye bandwidth
I had a T1 with 15 users on it one time that was very slow... It was an attorney's office so they did whatever they want but you get the point. :)
I use an equivalent to a t2 @ home, and I still complain ?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ADSL speeds can be 10,000kbps down / 750kbps up .. maybe better than that up. Why use T1 at all? It's more expensive and doesn't support what you're doing very well.
ASKER
This may ultimately lead me to resolving my problem(s), but I'm still having problems. Overall, this was the best solution of those offered. Some
"solutions" were merely comments about how awful it was that I had a T1 line and not helpful at all.
"solutions" were merely comments about how awful it was that I had a T1 line and not helpful at all.