Solved

Jumbo Frames in Vmware

Posted on 2014-03-28
9
1,118 Views
Last Modified: 2014-04-04
I believe in some cases Jumbo frames needs to be set up to maximum of 9000 in Vmware.
First  I need to know in which case it needs to be changed from default 1500 to 9000.
I also need to know if it also  requires changes of the MTU on the physical switches and routers.

Thank you
0
Comment
Question by:jskfan
  • 3
  • 3
  • 2
  • +1
9 Comments
 
LVL 23

Assisted Solution

by:Mysidia
Mysidia earned 300 total points
ID: 39963214
Well;   to use Jumbo frames in vSphere  on a network that leaves the vSphere host...

  o Your switches need to support this, AND  you need to increase the Layer 2  MTU on all switchports  related to a network on which you will be using Jumbo frames.

The general recommendation here is to set the MTU  on all Layer 2 physical switchports (and port channels) to the absolute maximum MTU supported  -----  you can do this safely, even on ports you are not using jumbo frames on, and the switch configuration is simplified.

Because the MTU configured on Layer 2 switchports,  ONLY decides which frames can be forwarded ----  Layer 2 switching elements will not generate new frames on the network.

So the switch MTU that applies to physical ports is completely transparent to the hosts ------   as long as the MTU / frame size configured  on your Layer 2  switches is as large or larger  than the frames you are using ----  this is completely transparen to hosts.

After all  Layer 2 switching infrastructure (Including virtual switches in all vSphere hosts) is ready to support large frames,   then you need to look at all devices such as router ports, virtual machines and VMkernel ports assigned Layer 3 addresses on the same subnet / same VLAN.

Essentially:  The Layer 3  MTU  of all  devices on any particular VLAN  (subnet) must match.

If a network interface has an IP address assigned to it:  the Layer 3   MTU of that interface must agree with all the other hosts in the same LAN.

If that MTU is  9000,  then the host will emit and accept jumbo frames.



  o  Your network cards need to support Jumbo frames.   Current 10-gigabit server NICs from Intel and Broadcom support these.

      However,  the combination of  Jumbo frames and Hardware-based iSCSI offload,
      may be unsupported  on some hardware.

 There are two places in vSphere itself where this setting occurs.

o  MTU Setting on the Vswitch;  on a standard vSwitch this is found under:
    Host > Configuration  > Networking
      vSwitch Properties
      Select the vSwitch > Edit

The MTU configured on the vSwitch is  the maximum  Layer 2  MTU used by vSphere.


Next there is the Layer 3  MTU, which is set on the VMkernel port group, and on the router port/Layer 3 interface of the switch  or router being used as default gateway:
o MTU Setting on the VMKernel port groups.
    Host > Configuration  > Networking
0
 

Author Comment

by:jskfan
ID: 39963860
Most of the environments, they leave the MTU to the default 1500.
I am not sure in which case jumbo frames need to be changed to 9000.

it sounds like a lot of work in the Network side, because you will have to go to each switch in the network and change all ports MTU to 9000…
if an environment has 5000 or 10000 users, imagine how many switches are used and how long it will take to Network Administrators to change all switch ports MTU.
0
 
LVL 23

Assisted Solution

by:Mysidia
Mysidia earned 300 total points
ID: 39964156
"Most of the environments, they leave the MTU to the default 1500. I am not sure in which case jumbo frames need to be changed to 9000."

Well.. the answer is ALL environments where the MTU on the switchports is not already greater than 1500.    If the MTU on the switch ports is 1500,  then there are definitely not jumbo frames going through that port.

"if an environment has 5000 or 10000 users, imagine how many switches are used and how long it will take to Network Administrators to change all switch ports MTU."

It doesn't really matter how many users there are.

First of all:  Jumbo frames are primarily the most useful for a few purposes, such as access to ISCSI storage,  Vmotion,  Fault tolerance, Storage replication, AND to make efficient usage of 10-Gigabit and other high-throughput networks.

Smaller packets are less efficient in terms of CPU usage on your hosts, and, ultimately latency of high volume traffic.

Generally; I  recommend segmenting desktop computers on a LAN which does not use Jumbo frames.

Until we have 10-Gigabit to the desktop ---- jumbo frames do not have a large number of applications for desktop users,  who generally do not even need a full gigabit.


It is already best practice that all these special functions be on separate VLANS.     So the Layer 2 path between all ESXi hosts and  each other, storage, routers,  trunk ports between switches, and any new network ports that are turned up in the future need to be Jumbo-enabled,
throughout the VLAN where these large MTUs are assigned to VMKernel ports and storage,  but not necessarily users.

This is not a heavy burden.

Second....  it is generally just a few commands on each switch to make all ports Jumbo frame ready. It can be scripted.

So all Layer 2 networks can be Jumbo enabled ----- that is fine, as long as all your switches support it.

blah#
blah(config)#system mtu jumbo 9216
blah(config)#system mtu 9216
blah(config)# int range GigabitEthernet0/1 - 48
blah(config-if-range)# mtu 9216
blah(config-if-range)#^Z
blah(config-if-range)#write memory

Note that an interface range command is used, and then the MTU is changed for   numerous ports with one command

Note that MTU on your switches needs to be  ~22 bytes (at least)  larger than the Layer 3 MTU;  so we generally config.  Force10 switches to use 9022 MTU  when using MTU of 9000 on VMkernel ports and on the firewall interface for the storage LAN.


Anything larger than 1500 on an IP interface is  Jumbo.
9000 is generally  as high as you can go in vSphere,  and only if your switches support 9022 MTU.

But 4000 byte frames, etc, are still considered jumbo frames.
0
 
LVL 61

Assisted Solution

by:gheist
gheist earned 100 total points
ID: 39964756
it has to be changed to 9000 on all virtual switches. That has no implications at all on physical or virtual connections.
0
6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

 
LVL 23

Assisted Solution

by:Mysidia
Mysidia earned 300 total points
ID: 39964779
@gheist: eh?  Of COURSE there are implications on physical and virtual connections!
It affects whether network connectivity will be broken or not.

The moment you increase the MTU on your VMkernel portgroup listed on your vSwitch [which is separate from the virtual switch's MTU setting] -- to actually use Jumbo frames  on that virtual network;  any jumbo frames being sent from that vmkernel IP address will be silently discarded by any  Virtual or Physical switchport  that has not yet been configured to accept those packets,     And by any other IP interface  that has not yet had its own MTU increased to match.

Every physical switchport and  virtual switch  that will have a    VLAN or Port group   on it    supporting a network which will use jumbo frames needs to be configured with an adequate MTU,   and it is recommended to do this first -----  before you start  changing MTUs on vSphere Vmkernel ports and router ports,  or virtual machines.ernel ports.

Adequate MTU on most physical switches is calculated based on the Layer 3 MTU -- you add 22 bytes worth of  frame and VLAN headers,  which gives you a minimum Layer2 MTU of  9022 to support  a  Layer 3 MTU of 9000  on your routers / VMkernel ports.


On the other hand: for vSphere's  Virtual switch...  the Virtual switch MTU just needs to be greater than or larger than the   VMkernel port group MTU.

It is a  switch vendor specific matter --- whether the specified MTU  on your physical switchports  includes  (or does not include)  the additional 22  bytes for  802.1q tag and Ethernet frame headers ------   on most physical switches,  you  need to increase MTU further to account for them:   on  the VMware virtual switch - these are allowed bytes  not included in the MTU value.


The exact procedure for enabling this functionality on vSphere itself, depends on if you are using the vSphere .NET client or Web client,  and also....   if you are using  Distributed vSwitches  VS  a Standard vSwitch.

In the standard vSphere client, on a standard vSwitch  you click a host,  go to  Configuration > Networking

Pick the blue properties button above one of the vSwitches.
Then from within this dialog box, you can edit the MTU on the entire vSwitch, by selecting the top entry, to PREPARE the vSwitch for jumbo frames.

If ready to  start using the Jumbo frames for vSphere traffic;  the various VMkernel ports  on that vSwitch are listed below,  and each one has an independent MTU.

So vSphere can have Jumbos activated on  some VMkernel port groups, but not others.

This way you can easily choose to  use Jumbo frames on the Vmotion network and the Storage network,  but  leave the Management network   with standard sized frames.


As for Virtual machines themselves.... there is no  MTU setting for each individual portgroup;  it is the network interface within the guest operating system that has a configured MTU.

I do not recommend using Jumbo frames on virtual machine networks (however), and.... particularly not networks with large numbers of VMs.





A procedure for enabling JUMBOs on the vSphere Distributed vSwitch is discussed here:
http://kb.vmware.com/selfservice/microsites/search.dolanguage=en_US&cmd=displayKC&externalId=1038827

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007654
0
 
LVL 61

Expert Comment

by:gheist
ID: 39964782
Nobody generates jumbo frames right away. If you have MTU at 9000 you have the chance to test connections one by one. If you have vswitch MTU at 1500 you have to make global cable cut some day later.
0
 
LVL 118

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 100 total points
ID: 39966443
I would add, you may want to test Jumbo Frames (or any frame larger than 1500, could be considered a jumbo frame!).

and when testing, you will need to ensure, all your physical network switches and storage area network (NFS or iSCSI) are also changed and compatible with Jumbo Frame.

Usually 9,000 or larger!

Sometimes, you may find an increase in performance, or decrease in performance for storage.

see my EE Article


HOW TO: Enable Jumbo Frames on a VMware vSphere Hypervisor (ESXi 5.0) host server using the VMware vSphere Client
0
 

Author Closing Comment

by:jskfan
ID: 39978989
Thank you Guys!
0
 
LVL 61

Expert Comment

by:gheist
ID: 39979214
You cannot have more than 9000 in vswitches
0

Featured Post

What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Suggested Solutions

When we have a dead host and we lose all connections to the ESXi, and we need to find a way to move all VMs from that dead ESXi host.
This article will show you how to create an ISO CD-ROM/DVD-ROM image (*.iso), and MD5 checksum signature, for use with VMware vSphere Hypervisor 6.5 (ESXi 6.5). It's a good idea to compare checksums, because many installations fail because of a corr…
Teach the user how to edit .vmx files to add advanced configuration options Open vSphere Web Client: Edit Settings for a VM: Choose VM Options -> Advanced: Add Configuration Parameters:
Teach the user how to install and configure the vCenter Orchestrator virtual appliance Open vSphere Web Client: Deploy vCenter Orchestrator virtual appliance OVA file: Verify vCenter Orchestrator virtual appliance boots successfully: Connect to the …

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now