Network Slowdown
We provide IT to a fairly large car dealership. For the past several months they've had the internet slow to a crawl. They pay for 40/40mbs. During the slow down they'll see speedtests of .2-1mb down and 30up. These slow downs are not everyday. And when they happen, it's first thing in the morning. Usually 8am - 1030am and then it just goes back to normal.
We've talked multiple times with the ISP. They say it's not on their end. They have sent us a screenshot showing our network is saturated but no details as to what could be causing it.
We have a sonicwall in place and have recently purchased Bandwidth Monitoring. I've been able to check slow downs twice with the BWM. The top initiators change almost every time i refresh. They are all devices or workstations. Not seeing any servers or a single workstation as the top "bandwidth hog." The top Application in the app flow is "General HTTPS".
We've been able to use the IPs to nslookup the workstations and run some anti-malware, check browsing history. Some have needed malware/PUPs cleaned up but nothing has ultimately resolved the issue. We're running out of things to look for and try. Just looking for tools, suggestions, more tests to try and track this issue down.
We've talked multiple times with the ISP. They say it's not on their end. They have sent us a screenshot showing our network is saturated but no details as to what could be causing it.
We have a sonicwall in place and have recently purchased Bandwidth Monitoring. I've been able to check slow downs twice with the BWM. The top initiators change almost every time i refresh. They are all devices or workstations. Not seeing any servers or a single workstation as the top "bandwidth hog." The top Application in the app flow is "General HTTPS".
We've been able to use the IPs to nslookup the workstations and run some anti-malware, check browsing history. Some have needed malware/PUPs cleaned up but nothing has ultimately resolved the issue. We're running out of things to look for and try. Just looking for tools, suggestions, more tests to try and track this issue down.
ZeropointNRG
40/40 and BMW just doesn't seem right..How many PC's is this? I'm going to assume you've shut down the router for at least a minute each day?
The correct action is to open an issue ticket with your ISP about this.
Also, when doing speed tests, be sure you do this via a computer plugged into an Ethernet connection on the actual incoming router, else you're testing WiFi speed, which might be very different than actual ISP provided speed.
Also, when doing speed tests, be sure you do this via a computer plugged into an Ethernet connection on the actual incoming router, else you're testing WiFi speed, which might be very different than actual ISP provided speed.
You said, "We've talked multiple times with the ISP. They say it's not on their end."
This is easily determined by testing via a hard wired Ethernet connection.
If the problem persists over a hardwired connection, fire your ISP + get another one.
This is easily determined by testing via a hard wired Ethernet connection.
If the problem persists over a hardwired connection, fire your ISP + get another one.
Very suspicious. But many ISPs will always blame someone else unless you're connected directly to their equipment and then show that there are issues with the speeds. Downside is that this will require basically having the dealership cut off from the internet for a period of time.
Which ISP is this?
Which ISP is this?
ASKER
Zero: We have around 140-150 PCs and servers on the network plus client devices like cellphones (these have a 2hour dhcp lease restriction.) PCs and Servers are all connected via ethernet. We have rebooted the router around 1-2 months ago. Also rebooted the sonicwall today.
David: We did connect directly to the sonicwall today and run the test. This came back at 35/35 and by the time we got to it the internet had returned to normal. Last time we opened a ticket, our ISP wanted us to take the entire network down in order to test directly. This is a really busy dealership with other locations under the same umbrella connecting back for central programs.
The problem is noticed even with 90%+ of the network being over Ethernet.
David: We did connect directly to the sonicwall today and run the test. This came back at 35/35 and by the time we got to it the internet had returned to normal. Last time we opened a ticket, our ISP wanted us to take the entire network down in order to test directly. This is a really busy dealership with other locations under the same umbrella connecting back for central programs.
The problem is noticed even with 90%+ of the network being over Ethernet.
We did connect directly to the sonicwall today and run the test. This came back at 35/35 and by the time we got to it the internet had returned to normal.35/35 isn't bad. The other numbers you reported in your question are far into the red flag range.
Last time we opened a ticket, our ISP wanted us to take the entire network down in order to test directly.This is a normal request. You're going to have to schedule some downtime, ideally in the earlier span of when things go slow. 8:30 am would be a less painful time for downtime than 10 am from a business perspective.
ASKER
ISP is LSI/Telesystems.
When we've opened tickets with them in the past, they're able to get to the modem and see that the circuit is good (of course this is something only they see.) They have also sent us a graph twice showing that our "pipe" , which is dedicated only to our store, is saturated. Which is why we started looking internally. What inside the network is saturating the connection. We just can't find a reason.
When we've opened tickets with them in the past, they're able to get to the modem and see that the circuit is good (of course this is something only they see.) They have also sent us a graph twice showing that our "pipe" , which is dedicated only to our store, is saturated. Which is why we started looking internally. What inside the network is saturating the connection. We just can't find a reason.
100/100 is barely enough for my house lol...
@Zero - In comparison to 40/40, 35/35 isn't bad.... haha.
ASKER
Just confirmed that 40/40 is correct. It's worked for a long time up until 5-6 months ago.
lol
Did anything change 5 or 6 months ago?
Have any of your internal systems been getting more traffic than normal?
I am guessing you haven't seen signs if any machines putting out a number of requests? Such as say to update servers?
Have any of your internal systems been getting more traffic than normal?
I am guessing you haven't seen signs if any machines putting out a number of requests? Such as say to update servers?
ASKER
Just looked back through my notes. Things i worked on 5-6 months ago:
Setting up some email forwards for a program call Vinsolutions.
Renewed an Exchange Cert.
Worked on some IP Cameras
Other general troubleshooting but these are the major things. I also notice that it happened a few times middle of Feb then i don't have it noted as happening again until middle of July. Only i really worked on leading up to the mid-July occurrence is some more IP Camera setup.
Setting up some email forwards for a program call Vinsolutions.
Renewed an Exchange Cert.
Worked on some IP Cameras
Other general troubleshooting but these are the major things. I also notice that it happened a few times middle of Feb then i don't have it noted as happening again until middle of July. Only i really worked on leading up to the mid-July occurrence is some more IP Camera setup.
Just for curiosity's sake, is the camera traffic segregated at all?
If you had the Analyzer, you might have an easier time seeing traffic going in and out. Of course this assumes you dont already have something doing this.
If you had the Analyzer, you might have an easier time seeing traffic going in and out. Of course this assumes you dont already have something doing this.
Have you scanned all PCs and servers for infections? That could easily be causing the slowdown.
Does the bandwidth monitoring show the other side of the connection, ie. if there is a common website/service that all your workstations are downloading from?
To further identify HTTPS traffic you may need a transparent proxy service to 'man-in-the-middle' all HTTPS traffic; this would allow you to decrypt the traffic and see the destination service address and identlfy what the traffic is.
To be effective you would block any HTTPS traffic that doesn't flow through the proxy.
N.B. there could be privacy/legal concerns for intercepting traffic which your users may have considered private.
To further identify HTTPS traffic you may need a transparent proxy service to 'man-in-the-middle' all HTTPS traffic; this would allow you to decrypt the traffic and see the destination service address and identlfy what the traffic is.
To be effective you would block any HTTPS traffic that doesn't flow through the proxy.
N.B. there could be privacy/legal concerns for intercepting traffic which your users may have considered private.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.