Slow SMB over WAN?
If you need additional info please let me know. Thanks for taking a look at our post.
The problem slow SMB over WAN. Latency avg 27ms Wiresharks shows the Windows Size is small 8012 for SMB traffic. Bandwidth 100 MBPS both up and down. Speed out to the Internet is great both up and down.
Site to Site VPN setup between two Fortigates.
Copying and opening files between the two sites is slow. Branch Office has a mix of Win7 and Win10 on the Desktops. Fileserver is running Win2016. I have three remote sites with the same problem. Wireshark tells us we're running SMB 2.0.
I'm asking for suggestions to increase the file transfer speed over the WAN.
Fortinet's TAC Level 3 has checked and double checked the Fortigates for misconfiguration errors.
We have done things like reduce the MTU size on the Fortigates. Ran continuous pings on both the private and public interfaces looking for dropped packets.
Checked for mismatched duplexing. Updated the Fortigates Firmware, Attached laptops directly to the Fortigates and copied files between them, ran Iperf both TCP and UDP between them, etc, etc.
We've read document after document on slow SMB over WAN. Performance tuning for SMB. - https://www.apachelounge.com/download/contr/Perf-tun-srv-2016.pdf
We've added the registry key for SizReqBuf, and set it to ffff.
Windows Size Scaling is mentioned.
https://www.auvik.com/franklymsp/blog/tcp-window-size/? This link mentions Windows Scaling is enabled by default in all recent implementations.
How to fix TCP windowing
The TCP window size is controlled by the end devices, not by the routers, switches, or firewalls that happen to be in the middle. The devices actively and dynamically negotiate the window size throughout the session.
But as I mentioned earlier, the TCP mechanism was designed for network bandwidth that’s orders of magnitude slower than what we have today. So some implementations still enforce a maximum window size of 64KB. You can get around this by enabling windows scaling, which allows windows of up to 1GB.
Windows scaling was introduced in RFC 1323 to solve the problem of TCP windowing on fast, reliable networks. It’s available as an option in any modern TCP implementation. The only question is whether it’s been enabled properly.
In all recent Microsoft Windows implementations, windows scaling is enabled by default. You ‘ll find places on the Internet telling you to change registry values to increase your window size, but depending on the Windows version you’re using, these changes will have no effect. The values may no longer even exist. Bottom line, you don’t need to fix TCP windowing in Windows, either clients or servers.
A few links we've read.
https://blogs.technet.microsoft.com/neilcar/2004/10/26/smbcifs-performance-over-wan-links/
https://support.microsoft.com/en-us/help/224829/description-of-windows-2000-and-windows-server-2003-tcp-features
https://social.technet.microsoft.com/Forums/en-US/c5ac8ffd-71a1-4357-8fc5-119edab186e6/tcp-window-size-problems?forum=winserverPN
https://searchnetworking.techtarget.com/definition/Server-Message-Block-Protocol
https://support.microsoft.com/en-us/help/2696547/detect-enable-disable-smbv1-smbv2-smbv3-in-windows-and-windows-server
The problem slow SMB over WAN. Latency avg 27ms Wiresharks shows the Windows Size is small 8012 for SMB traffic. Bandwidth 100 MBPS both up and down. Speed out to the Internet is great both up and down.
Site to Site VPN setup between two Fortigates.
Copying and opening files between the two sites is slow. Branch Office has a mix of Win7 and Win10 on the Desktops. Fileserver is running Win2016. I have three remote sites with the same problem. Wireshark tells us we're running SMB 2.0.
I'm asking for suggestions to increase the file transfer speed over the WAN.
Fortinet's TAC Level 3 has checked and double checked the Fortigates for misconfiguration errors.
We have done things like reduce the MTU size on the Fortigates. Ran continuous pings on both the private and public interfaces looking for dropped packets.
Checked for mismatched duplexing. Updated the Fortigates Firmware, Attached laptops directly to the Fortigates and copied files between them, ran Iperf both TCP and UDP between them, etc, etc.
We've read document after document on slow SMB over WAN. Performance tuning for SMB. - https://www.apachelounge.com/download/contr/Perf-tun-srv-2016.pdf
We've added the registry key for SizReqBuf, and set it to ffff.
Windows Size Scaling is mentioned.
https://www.auvik.com/franklymsp/blog/tcp-window-size/? This link mentions Windows Scaling is enabled by default in all recent implementations.
How to fix TCP windowing
The TCP window size is controlled by the end devices, not by the routers, switches, or firewalls that happen to be in the middle. The devices actively and dynamically negotiate the window size throughout the session.
But as I mentioned earlier, the TCP mechanism was designed for network bandwidth that’s orders of magnitude slower than what we have today. So some implementations still enforce a maximum window size of 64KB. You can get around this by enabling windows scaling, which allows windows of up to 1GB.
Windows scaling was introduced in RFC 1323 to solve the problem of TCP windowing on fast, reliable networks. It’s available as an option in any modern TCP implementation. The only question is whether it’s been enabled properly.
In all recent Microsoft Windows implementations, windows scaling is enabled by default. You ‘ll find places on the Internet telling you to change registry values to increase your window size, but depending on the Windows version you’re using, these changes will have no effect. The values may no longer even exist. Bottom line, you don’t need to fix TCP windowing in Windows, either clients or servers.
A few links we've read.
https://blogs.technet.microsoft.com/neilcar/2004/10/26/smbcifs-performance-over-wan-links/
https://support.microsoft.com/en-us/help/224829/description-of-windows-2000-and-windows-server-2003-tcp-features
https://social.technet.microsoft.com/Forums/en-US/c5ac8ffd-71a1-4357-8fc5-119edab186e6/tcp-window-size-problems?forum=winserverPN
https://searchnetworking.techtarget.com/definition/Server-Message-Block-Protocol
https://support.microsoft.com/en-us/help/2696547/detect-enable-disable-smbv1-smbv2-smbv3-in-windows-and-windows-server
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
If you want to test the whole setup..the easiest method should be to bypass just about everything and connect a laptop on each router ( I mean the router that connects directly to the Internet)...setup a FTP server/Client ....Filezilla (?) and then test the connection by in turn assigning one laptop as the Server and the other as the client...just send some files and observe speed and latency (i reckon that you have good Internet connection because here in poor Greece my lowest was 50 ms...now its around 90-100ms and it can go up to 300+) ...write down the numbers and compare them to your current setup...
If you get a match then your setup is good...if not you need to check by moving in reverse...the laptop instead of the router connects a level up...(e.g router --> firewall --> switch ...)
If you get a match then your setup is good...if not you need to check by moving in reverse...the laptop instead of the router connects a level up...(e.g router --> firewall --> switch ...)
In stead of FTP use something like netio or iperf3 for assessing network speeds that cuts the disk bandwidth from the network equation.
ASKER
I don't mean to sound repetitive, regarding Fortinet, we did what they asked us to do.
@Dr. Klahn - Reducing MTU was suggested by a Fortinet. We have average speed of 355 KBPS transferring files using SMB.
@WiReDWolf - The Windows 2016 file server is a VMWare guest. Can you edit those settings on the NIC for just that VMWare guest?
@John Tsioumpris - We're confident that the circuit between the two points is good. We have reached out to the ISP's and asked them to check it out. We also have access to a Dashboard the ISP provides to monitor a number of things such as bandwidth and network interface utilization. We have attached a laptop to the ISP router and ran speed tests out to the Internet. We have not attached a laptop on each end as you suggest. We have attached a laptop directly into the Firewalls and transferred files between them. We used Wireshark to capture the packet flow. We've also used Fortinet's packet capture tool and saved the output to a file for Fortinet's engineers to review.
@noci - We have used Iperf3 to measure speed while having the laptops attached directly to the Fortigates. We used Wireshark to capture the output to a file for review. "NOCI" I'm not familiar with. Iperf the best speeds we measured was around 5MBPS. I need to go back and review my notes to be sure. At the moment I'm working from memory. Also note at the branch office the upload speed is 20MBPS, I've read that the max speed you can expect is based on what the upload speed is.
The thing that is the common denominator is the Window size we see in Wireshark. For SMB traffic it's quite small.
Two questions: How is that Window size determined? Is there a way to increase it for the SMB protocol?
Reading about SMB it seems the devices on each end (the workstation and the file server) determine the Window size when setting up the initial TCP\IP session. Check me if I'm wrong on this point.
The Windows Sliding scale is used to increase the size. And the Windows Sliding scale is enabled by default. Check me on this point if I'm wrong.
Thank you all for your input. I look forward to hearing back from you.
@Dr. Klahn - Reducing MTU was suggested by a Fortinet. We have average speed of 355 KBPS transferring files using SMB.
@WiReDWolf - The Windows 2016 file server is a VMWare guest. Can you edit those settings on the NIC for just that VMWare guest?
@John Tsioumpris - We're confident that the circuit between the two points is good. We have reached out to the ISP's and asked them to check it out. We also have access to a Dashboard the ISP provides to monitor a number of things such as bandwidth and network interface utilization. We have attached a laptop to the ISP router and ran speed tests out to the Internet. We have not attached a laptop on each end as you suggest. We have attached a laptop directly into the Firewalls and transferred files between them. We used Wireshark to capture the packet flow. We've also used Fortinet's packet capture tool and saved the output to a file for Fortinet's engineers to review.
@noci - We have used Iperf3 to measure speed while having the laptops attached directly to the Fortigates. We used Wireshark to capture the output to a file for review. "NOCI" I'm not familiar with. Iperf the best speeds we measured was around 5MBPS. I need to go back and review my notes to be sure. At the moment I'm working from memory. Also note at the branch office the upload speed is 20MBPS, I've read that the max speed you can expect is based on what the upload speed is.
The thing that is the common denominator is the Window size we see in Wireshark. For SMB traffic it's quite small.
Two questions: How is that Window size determined? Is there a way to increase it for the SMB protocol?
Reading about SMB it seems the devices on each end (the workstation and the file server) determine the Window size when setting up the initial TCP\IP session. Check me if I'm wrong on this point.
The Windows Sliding scale is used to increase the size. And the Windows Sliding scale is enabled by default. Check me on this point if I'm wrong.
Thank you all for your input. I look forward to hearing back from you.
You are correct. The upload speed is a limiting factor. So if you have a DSL 100/10 then 10Mbps is the upload speed. and would be the limit for that site to end data or the other side to receive data from here. (obviously the connection must be shared between several users).
The speed might also be limited by the capturing while measuring with iperf.
The windows size should be increased with no retransmits & lost packets. The window size should decrease when a lot of retransmits are needed. (it is the receiver buffer size and when the sender should expect an ACK).
Some explanation in this video: https://www.youtube.com/watch?v=2PJVHvthrNU
Even if the windows size IS say 64K even then if a packet is sent of only 100 Bytes then those will be acked after a short while..
And please mind your units... m = milli, M = Mega, B = Bytes, b=bits datacomm speeds are measured in bits per second (bps) for short. MBPS != mbps as (1000000 BYTES per second != 0.001 bit per second) about factor ~10 Billion or 10 orders of magnitude in difference.
The speed might also be limited by the capturing while measuring with iperf.
The windows size should be increased with no retransmits & lost packets. The window size should decrease when a lot of retransmits are needed. (it is the receiver buffer size and when the sender should expect an ACK).
Some explanation in this video: https://www.youtube.com/watch?v=2PJVHvthrNU
Even if the windows size IS say 64K even then if a packet is sent of only 100 Bytes then those will be acked after a short while..
And please mind your units... m = milli, M = Mega, B = Bytes, b=bits datacomm speeds are measured in bits per second (bps) for short. MBPS != mbps as (1000000 BYTES per second != 0.001 bit per second) about factor ~10 Billion or 10 orders of magnitude in difference.
To exclude fragmentation, you can use
If you see the MTU being decreased, e.g. because of the MTU change in the FortiGate, reduce the client MTU with
mturoute -t target.ip.addr.essIf you see the MTU being decreased, e.g. because of the MTU change in the FortiGate, reduce the client MTU with
netsh interface ipv4 set subinterface LAN mtu=1400 store=persistent
To answer the question asked of me about VMware guests - the answer is definitely. Which VMware NIC driver are you using? If using the Intel E1000 you really need to change that to the VMXNET3 driver.
I agree with noci as well - you keep saying MBPS as opposed to Mbps, which is a huge difference. Let's make sure we're talking about the same thing.
Also, Site to SIte VPN can use different levels of encryption. There is going to be a noticeable overhead regardless simply because the traffic is having to be encrypted/decrypted at each end. Assuming you're using IPSEC, have you investigated which protocols will grant the greater speeds? 3DES vs AES for example?
There appear to be a lot of variables here making it difficult to pin down exactly where you may have a speed issue.
I agree with noci as well - you keep saying MBPS as opposed to Mbps, which is a huge difference. Let's make sure we're talking about the same thing.
Also, Site to SIte VPN can use different levels of encryption. There is going to be a noticeable overhead regardless simply because the traffic is having to be encrypted/decrypted at each end. Assuming you're using IPSEC, have you investigated which protocols will grant the greater speeds? 3DES vs AES for example?
There appear to be a lot of variables here making it difficult to pin down exactly where you may have a speed issue.
ASKER
@noci - I'll mind the units. Does the Windows size scaling happen automatically?
@Qlemo - Thanks for the command. I'll give it a go.
@WiReDWolf - We're using the VMXNET3 driver. Thanks for the confirmation on configuring the nic's. We are using IPSEC, and we've tried different encryption protocols. We have not noted any major difference in the file transfer speeds. 3DES and SHA1 is what we started with, currently using AES256 and SHA256.
We've been advised to open a ticket with MS about the Window size. We're being told that if we can increase the size of the Window we'll get better throughput.
Thanks again.
@Qlemo - Thanks for the command. I'll give it a go.
@WiReDWolf - We're using the VMXNET3 driver. Thanks for the confirmation on configuring the nic's. We are using IPSEC, and we've tried different encryption protocols. We have not noted any major difference in the file transfer speeds. 3DES and SHA1 is what we started with, currently using AES256 and SHA256.
We've been advised to open a ticket with MS about the Window size. We're being told that if we can increase the size of the Window we'll get better throughput.
Thanks again.
ASKER
@WiReDWolf - after disabling the unneeeded features on the nic. Do you need to reboot to implement the new nic config?
Thanks
Thanks
No, not that I remember. There is a brief network disruption while the driver resets. That's about it.
ASKER
Thank to all of you.
This can only make things worse. The Fortigates should be set to the standard MTU size of 1500. Reducing MTU size causes packet fragmentation. Time is lost sending multiple packets where one would do and the reverse is also true; time can be lost reassembling multiple packets into one.
https://en.wikipedia.org/wiki/Maximum_transmission_unit
Bandwidth 100 MBPS both up and down.
LAN speed or WAN speed? LAN speed machts nicht when the packets go onto the internet where they are at the mercy of your ISP and network weather. Packets can't go onto the WAN faster than WAN speed, and contrariwise packets can't come off the WAN faster than WAN speed. 100 Mbps would be pretty fast for a WAN connection.
What are the speeds you're observing?