can the netstat results help tell what using up my bandwidth
Hi All
We a small network with small business server 2003 and 7 windows xp machines. lately we have been using more that 20gig of bandwidth a month and no one downloads or uploads anything out of the ordinary. i've also noticed that we used nearly 2gig over the weekend and there was no one around.
i've just installed net meter to try and confirm what the isp says we use but it won't tell me what or who is using what. i've gone on the server and run the netstat command but i don't really know how to interpret the results. i also wanted to know if, when I do interpret the results, they would be enough to give me an idea of what is using my bandwidth.
If not what can i do to get an idea of what happening to the bandwidth.
We a small network with small business server 2003 and 7 windows xp machines. lately we have been using more that 20gig of bandwidth a month and no one downloads or uploads anything out of the ordinary. i've also noticed that we used nearly 2gig over the weekend and there was no one around.
i've just installed net meter to try and confirm what the isp says we use but it won't tell me what or who is using what. i've gone on the server and run the netstat command but i don't really know how to interpret the results. i also wanted to know if, when I do interpret the results, they would be enough to give me an idea of what is using my bandwidth.
If not what can i do to get an idea of what happening to the bandwidth.
you can try to start with logging all SYN, FIN & RST TCP (start, end and reset ) packets using f.e. wireshark together with some bandwidth metering software and at moment of peak traffic try to see what might be active.
Traphics graphers like MRTG can keep track of interface statistics and make pictures of them prtg (from i think praessler ) is the same package but in a windows dressing.
Traphics graphers like MRTG can keep track of interface statistics and make pictures of them prtg (from i think praessler ) is the same package but in a windows dressing.
Yes it is Paessler and the product is PRTG Network Monitor.
I use it and it works well. There is a free version with up to 5 monitors and you would probably need more than 5 monitors to accomplish your goal.
Does your main switch have a mirroring port? This copies all traffic to one port so you can analyze it. Any tool that puts the network card into promiscuos mode would allow you to see all traffic as long as it's on a hub or mirrored switch port. Pretty much managed switches have this capability but not the cheaper desktop ones. Switches normally segregate all traffic so you only see broadcasts but nothing else unless it's destined for the port your on.
You could also install a hub or a switch with a mirroring port between your router/firewall and internet connection. This would give you a port with all traffic. Then you could take an old machine and install linux or use any extra machine with a linux live boot CD (like BackTrack) so you can use iptraf which has break downs by IP, port, mac address, protocol, current bandwidth usage, etc. The best part is that it's free GPL software . This would answer your question.
I use it and it works well. There is a free version with up to 5 monitors and you would probably need more than 5 monitors to accomplish your goal.
Does your main switch have a mirroring port? This copies all traffic to one port so you can analyze it. Any tool that puts the network card into promiscuos mode would allow you to see all traffic as long as it's on a hub or mirrored switch port. Pretty much managed switches have this capability but not the cheaper desktop ones. Switches normally segregate all traffic so you only see broadcasts but nothing else unless it's destined for the port your on.
You could also install a hub or a switch with a mirroring port between your router/firewall and internet connection. This would give you a port with all traffic. Then you could take an old machine and install linux or use any extra machine with a linux live boot CD (like BackTrack) so you can use iptraf which has break downs by IP, port, mac address, protocol, current bandwidth usage, etc. The best part is that it's free GPL software . This would answer your question.
How are you connected to your ISP (leased line, ADSL, WiMax, Cable, VSAT, etc.)?
Are you in contact with your ISP?
Just disconnect the cable coming into your premises. Ask the ISP how much data transfer do they see? It should be zero. If not, you can file a report with your ISP.
FYI - To download 2GB of data over a weekend (3 nights or about 60 hours from Friday evening to Monday morning), you need an average data transfer rate of 77Kbps. So, if you are not using it, somebody else is.
Do you have a wireless router ? You can check the access logs.
Are you in contact with your ISP?
Just disconnect the cable coming into your premises. Ask the ISP how much data transfer do they see? It should be zero. If not, you can file a report with your ISP.
FYI - To download 2GB of data over a weekend (3 nights or about 60 hours from Friday evening to Monday morning), you need an average data transfer rate of 77Kbps. So, if you are not using it, somebody else is.
Do you have a wireless router ? You can check the access logs.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Hi
Sorry for the late response. Since we are now over our 10gig threshold cap, we have been downgraded from our "normal" 4Mbps to 128kpbs which I still doubt we are getting, the internet is so slow and the connection has been dropping terribly. I'm yet to try your suggestions but to answer some of your questions.
We have an adsl line, a billion router which like cybervzhn_tech's client I don't have access to, only the ISP has access.
cybervzhn_tech, we are not using exchange for Inbound SMTP, some else handles our mail and we download it to exchange using pop3 connectors, they do not filter the mail so we do get a lot of spam but nothing unusual. On the issue of updates I thought WSUS was supposed to centralise the downloading of updates and ensure that everyone doesn't keep downloading the same thing. I could be wrong.
The other thing that makes me think it might not be updates is that we are, on average downloading and uploading 60mb of data per hour almost every hour so its not like at 11pm we suddenely download so much data but it seems to be more or less the same amount every hour. I've also been having disconnected sessions since I started monitoring this yesterday. I don't know y I'm now getting so many disconnected sessions this never happened before. See the attached document
Login-time.doc
Sorry for the late response. Since we are now over our 10gig threshold cap, we have been downgraded from our "normal" 4Mbps to 128kpbs which I still doubt we are getting, the internet is so slow and the connection has been dropping terribly. I'm yet to try your suggestions but to answer some of your questions.
We have an adsl line, a billion router which like cybervzhn_tech's client I don't have access to, only the ISP has access.
cybervzhn_tech, we are not using exchange for Inbound SMTP, some else handles our mail and we download it to exchange using pop3 connectors, they do not filter the mail so we do get a lot of spam but nothing unusual. On the issue of updates I thought WSUS was supposed to centralise the downloading of updates and ensure that everyone doesn't keep downloading the same thing. I could be wrong.
The other thing that makes me think it might not be updates is that we are, on average downloading and uploading 60mb of data per hour almost every hour so its not like at 11pm we suddenely download so much data but it seems to be more or less the same amount every hour. I've also been having disconnected sessions since I started monitoring this yesterday. I don't know y I'm now getting so many disconnected sessions this never happened before. See the attached document
Login-time.doc
~60 MB per hour ~ 133kbps so yes 128kbps seems to be a threshhold now. Please note that if you do a file transfer that IP adds 32+ bytes per packet overhead, the protocol in question can add some, if you have a PPPover AtM then that adds some overhead too the ISP presents you the bandwidth as seen on their (ATM?) ports (including overhead).
In that case just sample the data going out using wireshark, tcpdump etc. to see what goes out.
If needed get a classic HUB (not a switch switch/hub etc.) and put your network(the line that went into the billion router), a new line to the billion router and a PC with wireshark on that hub and sample the line. You might at first want to filter known "good" traffic like HTTP (TCP/80) DNS (UDP/53) and a few others and then see what is left. Then look at the IP addresses where they come from/go to and investigate those systems for f.e. p2p based protocols (File sharing, but also Skype!!).
Maybe you need to invest in a small ethernet/ethernet firewall and only allow known good traffic. (the mailserver is the only system doing port 25, any PC can access port 80 f.e.) and block all others.
In that case just sample the data going out using wireshark, tcpdump etc. to see what goes out.
If needed get a classic HUB (not a switch switch/hub etc.) and put your network(the line that went into the billion router), a new line to the billion router and a PC with wireshark on that hub and sample the line. You might at first want to filter known "good" traffic like HTTP (TCP/80) DNS (UDP/53) and a few others and then see what is left. Then look at the IP addresses where they come from/go to and investigate those systems for f.e. p2p based protocols (File sharing, but also Skype!!).
Maybe you need to invest in a small ethernet/ethernet firewall and only allow known good traffic. (the mailserver is the only system doing port 25, any PC can access port 80 f.e.) and block all others.
I pressed submit a tad too early...
IMO, you should own the device that sits between your network and the outside world.
For security as well as measuring traffic ammounts, possibly even shaping them to prevent saturation for time sensitive protocols like VOIP.
You cannot trust your ISP to be your gatekeeper. (And too many are too sloppy with modem configurations anyway).
IMO, you should own the device that sits between your network and the outside world.
For security as well as measuring traffic ammounts, possibly even shaping them to prevent saturation for time sensitive protocols like VOIP.
You cannot trust your ISP to be your gatekeeper. (And too many are too sloppy with modem configurations anyway).
I'd disconnect the computers, one-by-one, for 1 hour and check the usage. Make sure the duration sync's up with that in 'Login-time.doc'. It would help us identify an errant PC (bad NIC, etc.).
If the PC's are connected via switch, check the lights/activity. I'd start with the PC's that are more often on than others. Just disconnect them and then connect again (one-by-one) to determine the culprit.
If the PC's are connected via switch, check the lights/activity. I'd start with the PC's that are more often on than others. Just disconnect them and then connect again (one-by-one) to determine the culprit.
If none of your machines are compromised and you have verified no machines are using peer-to-peer file sharing (i.e. bittorrent), then the problem may simply be a lot of spam traffic. Do you have anti-spam of any type on the SBS server or is spam removed by clients? How much spam are your users getting per day?
I was seeing 250-350 MB of mail traffic per day until I configured the mailserver to use DNSRBLs to reject inital connections from known/highly possible spammers and misconfigured mail servers before the mail was even accepted. Almost instantly mail traffic reduced to 140-200 MB per day (reducing mail traffic by at least 100MB per day), most of which was valid email. Initially the clients I mention were getting 40-100 spam emails a day which dropped to 1 or 2 per day.
I've attached a graph showing the dramatic reduction in total email traffic by implementing more aggressive anti-spam controls. Keep in mind when the connection is rejected by the mail server, a very small amount of bandwidth is used compared to accepting the email then deleting it as spam later.
Unfortunately you cannot do this because you are downloading the email from your host (via POP3) not accpeting the inbound SMTP yourself. This means you have to remove the spam after it has been downloaded. See if your mail host offers anti-spam and if they will allow you to enable the feature temporarily to see if it helps.
email-traffic.jpg
I was seeing 250-350 MB of mail traffic per day until I configured the mailserver to use DNSRBLs to reject inital connections from known/highly possible spammers and misconfigured mail servers before the mail was even accepted. Almost instantly mail traffic reduced to 140-200 MB per day (reducing mail traffic by at least 100MB per day), most of which was valid email. Initially the clients I mention were getting 40-100 spam emails a day which dropped to 1 or 2 per day.
I've attached a graph showing the dramatic reduction in total email traffic by implementing more aggressive anti-spam controls. Keep in mind when the connection is rejected by the mail server, a very small amount of bandwidth is used compared to accepting the email then deleting it as spam later.
Unfortunately you cannot do this because you are downloading the email from your host (via POP3) not accpeting the inbound SMTP yourself. This means you have to remove the spam after it has been downloaded. See if your mail host offers anti-spam and if they will allow you to enable the feature temporarily to see if it helps.
email-traffic.jpg
Noci's comment is a really good idea. I would NEVER rely on an ISPs router for security, ever. It's like flying blind, you won't even know there is a wall in your path until you hit it.
Cisco/Linksys makes an excellent business router/firewall called the RV042 that goes for less than $150 that can provide you with much more information than you currently have, as well as increasing your level of security exponentially. On top of that you can subscribe to security updates that will protect your whole network from viruses, expoilts, and worse at the Internet connection before it even enters your network. An added benefit is that you get secure remote access via VPN.
Cisco/Linksys makes an excellent business router/firewall called the RV042 that goes for less than $150 that can provide you with much more information than you currently have, as well as increasing your level of security exponentially. On top of that you can subscribe to security updates that will protect your whole network from viruses, expoilts, and worse at the Internet connection before it even enters your network. An added benefit is that you get secure remote access via VPN.
ASKER
Hi some more information
It definately is not spam coz we do not get that much spam mail atmost maybe 20 emails a day. As for disconnecting the machines to see the usage, I know for sure that all the machines except the server are off over night and on weekends so all the activety I see during those times should be coming from the server. I disconnected all wireless access points when I started noticing this issue so there hasn't been any wireless activity. This weekend with all wireless access points disconnected and no other machines on, this is the activity i had Megabytes sent were 155.92mb and megabytes received were 2369.18mb.
I also got hold of my ISP to ask which ports are open on the router and they tell me that all ports are open by defualt. Initially when we got the router I told them which ports to open but they neglected to close the other ports. I do agree, not having access to your router is not the ideal thing but with this contract they told us that they do not give access to any clients who have threshold uncapped adsl and static IPs.
It definately is not spam coz we do not get that much spam mail atmost maybe 20 emails a day. As for disconnecting the machines to see the usage, I know for sure that all the machines except the server are off over night and on weekends so all the activety I see during those times should be coming from the server. I disconnected all wireless access points when I started noticing this issue so there hasn't been any wireless activity. This weekend with all wireless access points disconnected and no other machines on, this is the activity i had Megabytes sent were 155.92mb and megabytes received were 2369.18mb.
I also got hold of my ISP to ask which ports are open on the router and they tell me that all ports are open by defualt. Initially when we got the router I told them which ports to open but they neglected to close the other ports. I do agree, not having access to your router is not the ideal thing but with this contract they told us that they do not give access to any clients who have threshold uncapped adsl and static IPs.
As said before there are solutions that can sit happily in between a modem/router and your network.
An oridinary PC with 2 net adapters running a firewall distro (see below), A Zyxel Zywall 2, the earlier mentioned Cisco/Linksys RV042 box (and others),
Then YOU have the ability to forward or block traffic at will, (even allow more for a short wile), most modern boxes support QoS, so you can limit certain traffic over others or log/alert when there is a discrepancy.
IPCOP: http://www.ipcop.org/
Smoothwall: http://www.smoothwall.org/
CoyoteLinux: http://coyotelinux.com/
DevilLinux: http://www.devil-linux.org/home/index.php
An oridinary PC with 2 net adapters running a firewall distro (see below), A Zyxel Zywall 2, the earlier mentioned Cisco/Linksys RV042 box (and others),
Then YOU have the ability to forward or block traffic at will, (even allow more for a short wile), most modern boxes support QoS, so you can limit certain traffic over others or log/alert when there is a discrepancy.
IPCOP: http://www.ipcop.org/
Smoothwall: http://www.smoothwall.org/
CoyoteLinux: http://coyotelinux.com/
DevilLinux: http://www.devil-linux.org/home/index.php
ASKER
Yah you were right Cybervzhn tech. Its WSUS the wsuscontent folder grow by 22gig this month alone.
Typically connection info is available at the device that handles the Internet connection (firewall/router) as it's already keeping track of the all active connections to the Internet already. This is handled by an SNMP console app that pulls the stats from the router or firewall via SNMP. The right SNMP management app will break down traffic by port exposing the bandwidth hog from a network standpoint. Some personal or host -based firewalls show network utilization by process (application or service). SNMP managment consoles can range from free (GPL or open source) to thousands of dollars. They will use snmp or agents installed on computers to be monitored.
Finally, you can use Perfmon and Netmon in windows to collect this info but it would only show traffic related to the computers that have the network monitor service installed (in network card config, shows in the box where you would see what services are bound to the adapter like TCP/IP). You also take a perforance hit for collecting those stats.
Keep in mind getting this info from your firewall or router is the best way unless yours doesn't support SNMP or the neccessary counters because it tracks all hosts and can automatically notify you of an outtage or if thresholds you set are overrun. This gives you stats of all traffic related to all computers using the connection.
That kind of utilization sounds like file transfer or something more sinister. Are you sure someone on your network isn't using peer-to-peer file sharing type apps like bittorrent?