Bandwidth on my 100 user network drops from 5Mbps to 312kbps but only during working hours

My network consists of a LAN with 40 users, 50 VPN connections, 1 DNS/DHCP Server running Server 2003, 1 Exchange Server running Exchange 2003, 1 Microsoft SQL 2000 Server.  I have had it all running on a 2X2 connection until my bandwidth dies the other day.  I called my ISP and have them kick my downstream speed to 5Mbps.  They do this and I still can only pull 386kbps.  The odd thing is that the bandwidth drain starts when the workday begins and I regain all speed when it ends.  I thought, ok, must be my connection.  Nope, if you kill the network and plug directly into the cable modem your speeds are great.  So then I went and disconnected each users and server one by one and that did not fix it.  If you kill a server you regain some of the bandwidth but it still crawls back down.  
Our Sonicwall monitors bandwidth and nothing is out of the ordinary there.  We also have software installed on all computers that monitors Bandwidth, and nothing bad is showing there.  Did a virus sweep on everything at that point, but the only thing that showed was a vundo virus on one of my workstations.  I then changed out all of my switches hoping one of them were bad, nothing there.  I then turned my attention the software on the servers.  I checked all of the server logs with no luck.  I was thinking maybe DNS or Active Directory could be messing up but they look fine and act fine otherwise.  I then checked all my queues in Exchange hoping there was something stuck, but they were fine.  I rebooted all of the servers for good measure but that didn’t help either.   The strange part is that it only does this during working hours and if fine otherwise.  I am out of ideas and need something new to try.  You would think it was a user or server on the network, but it all checks out ok.
Who is Participating?
jdogunccConnect With a Mentor Author Commented:
This question solved this problem:

Lack of memory on server leading to lack of space on C:
use a sniffer like wireshark and configure a port on your switch for mirroring to capture all traffic. Run the sniffer and look for patterns of hammering of specific requests or nodes that are generating excessive traffic.
Also do an IP scan of your subnet(s) during and after business hours to see which devices may be shutdown after business hours and then monitor these during business hours for irregularities,
Never miss a deadline with

The revolutionary project management tool is here!   Plan visually with a single glance and make sure your projects get done.

What type of connection is this? do you have a CIR? IT is possible the bandwidth is being used by your neighbors
IS this comcast with the SMC modem?
jdogunccAuthor Commented:
We have wiresharked for hours on end with nothing out of the ordinary showing in the results.  The largest use of bandwitdh from one computer we have isn't enough to kill our alloted bandwidth.  The connection is a Time Warner connection.  We have been running under the current setup for 3 years with no major changes.  It is like there is a phantom device pulling sucking out all of out bandwidth.  Between wireshark and the bandwidth monitoring on the Sonicwall you think if it was a device on the network we would have spotted it by now.  That's why I turned to software instead of hardware.
Maybe the VPN connections (or equipment connected via VPN) are eating up your bandwidth. If the upload bandwidth is smaller than the download bandwidth, a large upload (sending a file from your network to a VPN computer for example) could impact your download bandwidth as well. 50 VPN connections mean a lot, and if they don't connect via the same provider or they don't have some sort of metropolitan traffic benefit (a larger bandwidth with computers from the same area /town/provider). Monitor the traffic from VPN and consider it as internet traffic not as LAN traffic.
Sounds like you have some sort of "Time of Day" policy active.
As you say the link is good if you connected directly to the cable modem then it looks internal.
Check any device that sits between your users and the internet, servers, routers.
If it happens at a specifc time ie 09:00 exactly then test with a large ftp so you can see the speed drop at that time, then - if you can - change the system time on the servers one by one to fool them that the time of day policy should not be active. You should then know which server to look at more closley.
Have you checked the NIC to your outside connection? In event viewer do you see any unusual entries about hardware? Often when it is hardware the errors do not actually point at specific devices. Instead you might see driver errors or even something more obscure.
If you are not sure and you have an extra NIC for your outside connection it might not hurt to swap the device out with a know good card and see what happens. The worst would be no improvement noticed.
Closed, 500 points refunded.
EE Moderator
All Courses

From novice to tech pro — start learning today.