Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17


Jumbo Frames in Vmware

Posted on 2014-03-28
Medium Priority
Last Modified: 2014-04-04
I believe in some cases Jumbo frames needs to be set up to maximum of 9000 in Vmware.
First  I need to know in which case it needs to be changed from default 1500 to 9000.
I also need to know if it also  requires changes of the MTU on the physical switches and routers.

Thank you
Question by:jskfan
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
  • 2
  • +1
LVL 23

Assisted Solution

Mysidia earned 1200 total points
ID: 39963214
Well;   to use Jumbo frames in vSphere  on a network that leaves the vSphere host...

  o Your switches need to support this, AND  you need to increase the Layer 2  MTU on all switchports  related to a network on which you will be using Jumbo frames.

The general recommendation here is to set the MTU  on all Layer 2 physical switchports (and port channels) to the absolute maximum MTU supported  -----  you can do this safely, even on ports you are not using jumbo frames on, and the switch configuration is simplified.

Because the MTU configured on Layer 2 switchports,  ONLY decides which frames can be forwarded ----  Layer 2 switching elements will not generate new frames on the network.

So the switch MTU that applies to physical ports is completely transparent to the hosts ------   as long as the MTU / frame size configured  on your Layer 2  switches is as large or larger  than the frames you are using ----  this is completely transparen to hosts.

After all  Layer 2 switching infrastructure (Including virtual switches in all vSphere hosts) is ready to support large frames,   then you need to look at all devices such as router ports, virtual machines and VMkernel ports assigned Layer 3 addresses on the same subnet / same VLAN.

Essentially:  The Layer 3  MTU  of all  devices on any particular VLAN  (subnet) must match.

If a network interface has an IP address assigned to it:  the Layer 3   MTU of that interface must agree with all the other hosts in the same LAN.

If that MTU is  9000,  then the host will emit and accept jumbo frames.

  o  Your network cards need to support Jumbo frames.   Current 10-gigabit server NICs from Intel and Broadcom support these.

      However,  the combination of  Jumbo frames and Hardware-based iSCSI offload,
      may be unsupported  on some hardware.

 There are two places in vSphere itself where this setting occurs.

o  MTU Setting on the Vswitch;  on a standard vSwitch this is found under:
    Host > Configuration  > Networking
      vSwitch Properties
      Select the vSwitch > Edit

The MTU configured on the vSwitch is  the maximum  Layer 2  MTU used by vSphere.

Next there is the Layer 3  MTU, which is set on the VMkernel port group, and on the router port/Layer 3 interface of the switch  or router being used as default gateway:
o MTU Setting on the VMKernel port groups.
    Host > Configuration  > Networking

Author Comment

ID: 39963860
Most of the environments, they leave the MTU to the default 1500.
I am not sure in which case jumbo frames need to be changed to 9000.

it sounds like a lot of work in the Network side, because you will have to go to each switch in the network and change all ports MTU to 9000…
if an environment has 5000 or 10000 users, imagine how many switches are used and how long it will take to Network Administrators to change all switch ports MTU.
LVL 23

Assisted Solution

Mysidia earned 1200 total points
ID: 39964156
"Most of the environments, they leave the MTU to the default 1500. I am not sure in which case jumbo frames need to be changed to 9000."

Well.. the answer is ALL environments where the MTU on the switchports is not already greater than 1500.    If the MTU on the switch ports is 1500,  then there are definitely not jumbo frames going through that port.

"if an environment has 5000 or 10000 users, imagine how many switches are used and how long it will take to Network Administrators to change all switch ports MTU."

It doesn't really matter how many users there are.

First of all:  Jumbo frames are primarily the most useful for a few purposes, such as access to ISCSI storage,  Vmotion,  Fault tolerance, Storage replication, AND to make efficient usage of 10-Gigabit and other high-throughput networks.

Smaller packets are less efficient in terms of CPU usage on your hosts, and, ultimately latency of high volume traffic.

Generally; I  recommend segmenting desktop computers on a LAN which does not use Jumbo frames.

Until we have 10-Gigabit to the desktop ---- jumbo frames do not have a large number of applications for desktop users,  who generally do not even need a full gigabit.

It is already best practice that all these special functions be on separate VLANS.     So the Layer 2 path between all ESXi hosts and  each other, storage, routers,  trunk ports between switches, and any new network ports that are turned up in the future need to be Jumbo-enabled,
throughout the VLAN where these large MTUs are assigned to VMKernel ports and storage,  but not necessarily users.

This is not a heavy burden.

Second....  it is generally just a few commands on each switch to make all ports Jumbo frame ready. It can be scripted.

So all Layer 2 networks can be Jumbo enabled ----- that is fine, as long as all your switches support it.

blah(config)#system mtu jumbo 9216
blah(config)#system mtu 9216
blah(config)# int range GigabitEthernet0/1 - 48
blah(config-if-range)# mtu 9216
blah(config-if-range)#write memory

Note that an interface range command is used, and then the MTU is changed for   numerous ports with one command

Note that MTU on your switches needs to be  ~22 bytes (at least)  larger than the Layer 3 MTU;  so we generally config.  Force10 switches to use 9022 MTU  when using MTU of 9000 on VMkernel ports and on the firewall interface for the storage LAN.

Anything larger than 1500 on an IP interface is  Jumbo.
9000 is generally  as high as you can go in vSphere,  and only if your switches support 9022 MTU.

But 4000 byte frames, etc, are still considered jumbo frames.
Complete VMware vSphere® ESX(i) & Hyper-V Backup

Capture your entire system, including the host, with patented disk imaging integrated with VMware VADP / Microsoft VSS and RCT. RTOs is as low as 15 seconds with Acronis Active Restore™. You can enjoy unlimited P2V/V2V migrations from any source (even from a different hypervisor)

LVL 62

Assisted Solution

gheist earned 400 total points
ID: 39964756
it has to be changed to 9000 on all virtual switches. That has no implications at all on physical or virtual connections.
LVL 23

Assisted Solution

Mysidia earned 1200 total points
ID: 39964779
@gheist: eh?  Of COURSE there are implications on physical and virtual connections!
It affects whether network connectivity will be broken or not.

The moment you increase the MTU on your VMkernel portgroup listed on your vSwitch [which is separate from the virtual switch's MTU setting] -- to actually use Jumbo frames  on that virtual network;  any jumbo frames being sent from that vmkernel IP address will be silently discarded by any  Virtual or Physical switchport  that has not yet been configured to accept those packets,     And by any other IP interface  that has not yet had its own MTU increased to match.

Every physical switchport and  virtual switch  that will have a    VLAN or Port group   on it    supporting a network which will use jumbo frames needs to be configured with an adequate MTU,   and it is recommended to do this first -----  before you start  changing MTUs on vSphere Vmkernel ports and router ports,  or virtual machines.ernel ports.

Adequate MTU on most physical switches is calculated based on the Layer 3 MTU -- you add 22 bytes worth of  frame and VLAN headers,  which gives you a minimum Layer2 MTU of  9022 to support  a  Layer 3 MTU of 9000  on your routers / VMkernel ports.

On the other hand: for vSphere's  Virtual switch...  the Virtual switch MTU just needs to be greater than or larger than the   VMkernel port group MTU.

It is a  switch vendor specific matter --- whether the specified MTU  on your physical switchports  includes  (or does not include)  the additional 22  bytes for  802.1q tag and Ethernet frame headers ------   on most physical switches,  you  need to increase MTU further to account for them:   on  the VMware virtual switch - these are allowed bytes  not included in the MTU value.

The exact procedure for enabling this functionality on vSphere itself, depends on if you are using the vSphere .NET client or Web client,  and also....   if you are using  Distributed vSwitches  VS  a Standard vSwitch.

In the standard vSphere client, on a standard vSwitch  you click a host,  go to  Configuration > Networking

Pick the blue properties button above one of the vSwitches.
Then from within this dialog box, you can edit the MTU on the entire vSwitch, by selecting the top entry, to PREPARE the vSwitch for jumbo frames.

If ready to  start using the Jumbo frames for vSphere traffic;  the various VMkernel ports  on that vSwitch are listed below,  and each one has an independent MTU.

So vSphere can have Jumbos activated on  some VMkernel port groups, but not others.

This way you can easily choose to  use Jumbo frames on the Vmotion network and the Storage network,  but  leave the Management network   with standard sized frames.

As for Virtual machines themselves.... there is no  MTU setting for each individual portgroup;  it is the network interface within the guest operating system that has a configured MTU.

I do not recommend using Jumbo frames on virtual machine networks (however), and.... particularly not networks with large numbers of VMs.

A procedure for enabling JUMBOs on the vSphere Distributed vSwitch is discussed here:
LVL 62

Expert Comment

ID: 39964782
Nobody generates jumbo frames right away. If you have MTU at 9000 you have the chance to test connections one by one. If you have vswitch MTU at 1500 you have to make global cable cut some day later.
LVL 123

Accepted Solution

Andrew Hancock (VMware vExpert / EE MVE^2) earned 400 total points
ID: 39966443
I would add, you may want to test Jumbo Frames (or any frame larger than 1500, could be considered a jumbo frame!).

and when testing, you will need to ensure, all your physical network switches and storage area network (NFS or iSCSI) are also changed and compatible with Jumbo Frame.

Usually 9,000 or larger!

Sometimes, you may find an increase in performance, or decrease in performance for storage.

see my EE Article

HOW TO: Enable Jumbo Frames on a VMware vSphere Hypervisor (ESXi 5.0) host server using the VMware vSphere Client

Author Closing Comment

ID: 39978989
Thank you Guys!
LVL 62

Expert Comment

ID: 39979214
You cannot have more than 9000 in vswitches

Featured Post

The Ideal Solution for Multi-Display Applications

Check out ATEN’s VS1912 12-Port DP Video Wall Media Player at InfoComm 2017. Kerri describes how easy it is to design creative video walls in asymmetric layouts and schedule detailed playlists ahead of time with its advanced scheduling feature.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

HOW TO: Upload an ISO image to a VMware datastore for use with VMware vSphere Hypervisor 6.5 (ESXi 6.5) using the vSphere Host Client, and checking its MD5 checksum signature is correct.  It's a good idea to compare checksums, because many installat…
Giving access to ESXi shell console is always an issue for IT departments to other Teams, or Projects. We need to find a way so that teams can use ESXTOP for their POCs, or tests without giving them the access to ESXi host shell console with a root …
Teach the user how to install and configure the vCenter Orchestrator virtual appliance Open vSphere Web Client: Deploy vCenter Orchestrator virtual appliance OVA file: Verify vCenter Orchestrator virtual appliance boots successfully: Connect to the …
This video shows you how to use a vSphere client to connect to your ESX host as the root user. Demonstrates the basic connection of bypassing certification set up. Demonstrates how to access the traditional view to begin managing your virtual mac…

721 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question