Enabling Jumbo Frames makes 10Gbe a bit slower

K B
K B used Ask the Experts™
on
I have a switch that supports jumbo frames (9014) and my 10 Gbe NICS support the exact same (9014).
I can ping 10.20.30.100 -f -l 8000 without a problem but my speeds tend to be slower with Jumbo Frames enabled.
I am just doing a file copy from a 2012R2 Server to a 2012R2 SMB 3.0 file share.
I use SMBmultichannelConstraint to assure that it is using the correct NIC.

I get as high as 9.3 Gbps without Jumbo Frames enabled but I only get about 7.1 Gbps WITH Jumbo Frames enabled.

Any ideas Experts?
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
This is quite normal, there is no guarantee, that enabling jumbo frames, will make traffic faster.

You may want to experiment, with the size you have defined, e.g. try an MTU of 4000 or 8000, rather than 9014.

Jumbo Frames are frames which are larger than 1500.

Also you may find that the bottle neck could be elsewhere, or you are not full optimizing the link.

Commented:
Normally, the only way jumbo frame could slow transfer speed is if the error rate on the link is high.
That said, File copy is a rather high level test, you should test link speed with a dedicated utility to see if the problem is effectively at the network stack or elsewhere.
Don JohnstonInstructor
Top Expert 2015

Commented:
Ping is not a good performance indicator for transfer speed.

I get as high as 9.3 Gbps without Jumbo Frames enabled but I only get about 7.1 Gbps WITH Jumbo Frames enabled.

How are you determining this speed?
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

K B

Author

Commented:
File copy and performance tab of task manager.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
and also it depends on the physical switch, in a recent conversation with Dell Networking, they advised us, that MTU size, should match the recommended figure designed for the Dell physical switch.

In this case with Dell, they suggested a figure of 9018! was the magic MTU for this Dell switch.
K B

Author

Commented:
on an empty 5.5 TB Raid 6 Array as the SMB 3.0 destination for the file transfer..

I copy one folder with 21.2 GB (64 Files, 5 Folders) I get anywhere between 9 Gbps to 4 Gbps.
if I copy the same folder (renamed) I get slower speeds

If I copy one 22 GB file it is fairly high 9 GB to a quick dip to 3.8 GB
If I copy it (renamed) it is consistently slower at  9 GB but mainly 7 GB rates.

With such a huge array.. why does the file transfer rate decrease when I add files?  We are only talking 22 GB out of 5.5 TB.
K B

Author

Commented:
Is Task Manager's performance tab not reliable?
"Batchelor", Developer and EE Topic Advisor
Top Expert 2015
Commented:
Did you wait between copies, to allow the RAID to write all data and clear its cache?

Task Manager is "reliable" if you look at it for a longer period (more than a few seconds). But you cannot measure network performance by copying files - too many factors are determining the end speed. You need network I/O performance tools, which consist of a client and a server part. The server is just confirming the packets, dismissing any content.
K B

Author

Commented:
I used LANBench which has both Client & Server... Worked well! thank you!
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
So what was the outcome ?

All working well ? No problems  with jumbo frames ?
Qlemo"Batchelor", Developer and EE Topic Advisor
Top Expert 2015

Commented:
Has the performance degradation been a red herring? It would also be interessing if you changed the MTU, and what the rates are with and without Jumbo frames, just for the files.
K B

Author

Commented:
I am realizing that I had the connections for the test set to more than one.. When I just use 1 connection it is a steady 6Gbps.  arggh

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial