Enabling Jumbo Frames makes 10Gbe a bit slower

I have a switch that supports jumbo frames (9014) and my 10 Gbe NICS support the exact same (9014).
I can ping -f -l 8000 without a problem but my speeds tend to be slower with Jumbo Frames enabled.
I am just doing a file copy from a 2012R2 Server to a 2012R2 SMB 3.0 file share.
I use SMBmultichannelConstraint to assure that it is using the correct NIC.

I get as high as 9.3 Gbps without Jumbo Frames enabled but I only get about 7.1 Gbps WITH Jumbo Frames enabled.

Any ideas Experts?
K BAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
This is quite normal, there is no guarantee, that enabling jumbo frames, will make traffic faster.

You may want to experiment, with the size you have defined, e.g. try an MTU of 4000 or 8000, rather than 9014.

Jumbo Frames are frames which are larger than 1500.

Also you may find that the bottle neck could be elsewhere, or you are not full optimizing the link.
Normally, the only way jumbo frame could slow transfer speed is if the error rate on the link is high.
That said, File copy is a rather high level test, you should test link speed with a dedicated utility to see if the problem is effectively at the network stack or elsewhere.
Don JohnstonInstructorCommented:
Ping is not a good performance indicator for transfer speed.

I get as high as 9.3 Gbps without Jumbo Frames enabled but I only get about 7.1 Gbps WITH Jumbo Frames enabled.

How are you determining this speed?
Your Guide to Achieving IT Business Success

The IT Service Excellence Tool Kit has best practices to keep your clients happy and business booming. Inside, you’ll find everything you need to increase client satisfaction and retention, become more competitive, and increase your overall success.

K BAuthor Commented:
File copy and performance tab of task manager.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
and also it depends on the physical switch, in a recent conversation with Dell Networking, they advised us, that MTU size, should match the recommended figure designed for the Dell physical switch.

In this case with Dell, they suggested a figure of 9018! was the magic MTU for this Dell switch.
K BAuthor Commented:
on an empty 5.5 TB Raid 6 Array as the SMB 3.0 destination for the file transfer..

I copy one folder with 21.2 GB (64 Files, 5 Folders) I get anywhere between 9 Gbps to 4 Gbps.
if I copy the same folder (renamed) I get slower speeds

If I copy one 22 GB file it is fairly high 9 GB to a quick dip to 3.8 GB
If I copy it (renamed) it is consistently slower at  9 GB but mainly 7 GB rates.

With such a huge array.. why does the file transfer rate decrease when I add files?  We are only talking 22 GB out of 5.5 TB.
K BAuthor Commented:
Is Task Manager's performance tab not reliable?
Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
Did you wait between copies, to allow the RAID to write all data and clear its cache?

Task Manager is "reliable" if you look at it for a longer period (more than a few seconds). But you cannot measure network performance by copying files - too many factors are determining the end speed. You need network I/O performance tools, which consist of a client and a server part. The server is just confirming the packets, dismissing any content.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
K BAuthor Commented:
I used LANBench which has both Client & Server... Worked well! thank you!
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
So what was the outcome ?

All working well ? No problems  with jumbo frames ?
Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
Has the performance degradation been a red herring? It would also be interessing if you changed the MTU, and what the rates are with and without Jumbo frames, just for the files.
K BAuthor Commented:
I am realizing that I had the connections for the test set to more than one.. When I just use 1 connection it is a steady 6Gbps.  arggh
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.