Missing Network Performance

(Please see the attached image, which shows my client's network topology)

My client uses FIOS as their ISP, with a 150 down / 150 up connection.  This is a new connection that was setup about 2 weeks ago, and on average, we were seeing 155Mbps down, but only 15Mbps up.  We called Verizon about the issue.  They came out, and replaced the router, which "resolved" the problem.  However, here is the problem we are facing:

The Verizon tech's certified laptop reached 155 down / 160 up when connected to the FIOS router.
Our server reached 155 down / 40 up while connected to the FIOS router
The Verizon tech's certified laptop also reached 155 down / 160 up when connected to the wireless (5GHz connected @ 450Mbps)
Our laptop reached 155 down / 40 up also when connected to the wireless (5GHz connected @ 450Mbps)

The problem is that since Verizon's laptop was able to achieve the advertised speed, the upload speed problem is therefore our responsibility.  However, I don't know how to troubleshoot the problem.  I have tested the server and 4 workstations within the office.  Whether they were directly connected to the router, or via wireless, the upload speed is always the same.  The only anomaly is the Verizon laptop.  The server is running SBS 2011 and is linked at 1Gb.  The workstations are either Win7 64bit, or WinXP 32bit.

The managed network switch is controlled by a VoIP service.  I unplugged the VoIP box and switch from the network during testing, but it had no impact on performance.

Is there some special way that Verizon setup their laptop?

Thank you

RobertSystems AdministratorAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Bryant SchaperCommented:
How are you testing, typically I have seen providers use a specific site to test, normally hosted on their network.  Verizon may use an internal speedtest server to verify speeds then have a hands off attitude when you use an internet site sense they don't control the internet.  

I really question the "Verizon" certified laptop, in the past it us always a third party contractor that seems to come out and they have some crap chromebook or something.
Trent SmithCommented:
If you plug the connection directly into another laptop what speeds do you get?

Directly connect the cable to the server and test again.

Are you using a firewall?  If so what type?  The provider is only required to provide the speed to their box and anything past that point is your responsibility.

On wireless you will always get a lower speed.
I suppose that you are talking about a test to some place like speedtest.net. It could well be that Verizon was going to their own site as Bryant Shaper suggested. In that case, you would have to test to the same location to get their results.

If you place a computer free from any anti-malware whatsoever and which has no firewall whatsoever, and completely remove your external firewall from the network, do you still get the low upload speeds?

The likelihood is that they (Verizon) have a throttle placed somewhere in their system and that throttle is accidentally bypassed when they make the test. I have had a similar issue with a different provider. They were supposed to have provided a larger upload bandwidth and for a few weeks said that all was well. It took a few weeks of complaining for them to apologetically explain that there was a station between that was still holding to the lower upload speed. Make the tests with no restrictions and when that shows that the shortfall still exists, go back and ask them to look into it again.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Amazon Web Services

Are you thinking about creating an Amazon Web Services account for your business? Not sure where to start? In this course you’ll get an overview of the history of AWS and take a tour of their user interface.

RobertSystems AdministratorAuthor Commented:
Hello, and thank you for the responses.  My apologies for the delayed response.

@Bryant: We were originally using speedtest.verizon.net, but it apparently caps at 30 tests from the same IP daily, so we continuted using speedtest.net.  The Verizon tech was using a certified Windows 7 laptop, since he didn't have admin rights to the machine.

@Trent: The laptop was directly connected to the FIOS router, and got 155down/160up.  I used the same plug and connected the server (SBS 2011) to the FIOS router, and got 155down/40up.  The server is using the built-in Windows Firewall.

@awed1: The Verizon tech was onsite at our location, and was connecting using the same network hardware as us.  It was freaky!  His laptop getts 155down/160up, and our server get 155down/40up.

Is it possible that there could be a group policy on the server that would cap performance?  Our previous ISP was TWC.  I'm wondering if the previous admin put something in place to manage network performance.
rdege, It is possible that there are some QoS settings in play that limit it, but did you remove (disable) the internal firewall on your server when you made the test? Also, did you look at your server's NIC settings to make sure that it doesn't set any limits or go to sleep etc.

Do you have a laptop that you can connect to the same port, turn of its internal firewall and make the speedtest.

If the laptop gets the same results as the server, the server is not at fault and the likelihood is that the Verizon Tech was using a different site. - cause to call them back to investigate further and prove the viability of the network speed using the same destination as you do.

If the laptop has good results, the problem is in the server and you could start looking at the NIC firmware etc. Any QoS settings etc. and all of the Firewall rules.
RobertSystems AdministratorAuthor Commented:
@awed1 - I did some research, and found that many people have upload limit problems with FIOS.  A lot of people suggested tweaking settings in their NIC drivers (specifically Large Send Offload).  I applied the changes, but it did not resolve the problem.

I plan on adding a linux/Windows dual-boot workstation to the LAN to test the speed and will go from there.  I will report my findings later this week.
Thanks for the update. It will be interesting to hear what you find out. B.
RobertSystems AdministratorAuthor Commented:
I connected my own workstation to the client network.  The workstation tripled booted Win7 / Win8 / Ubuntu.  All three achieved full FIOS speeds without issue, so I was able to eliminate this as a FIOS bandwidth issue.

The problem ended up being related to the TrendMicro internet security software.  I opened the Windows Resource Monitor -> Network tab, and monitored the network activity while running a speedtest.  I saw two executables spike in network activity; chrome.exe and TM_proxy.exe.

Uninstalling Trend Micro resolved the problem.
rdege, I has been a pressured season, and I just read your comment. Good job finding that it was TrendMicro.  We switched away from them and went back to Symantec - which seems to be less troublesome than its past versions were. They always say somewhere along the troubleshooting path to disable the AV software. Thanks for posting the solution. I'm glad that you found it. Did the next AV software work better?
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Network Analysis

From novice to tech pro — start learning today.