Ethernet topology for mixing different MTUs

>>What is the simplest network configuration that will let the machines with Gigabit NICs use Jumbo Frames to talk to each other, but still let all devices communicate?

GigE NIC----(jumbo frames)--------10/100/1000 switch-------------100Mb Nic

or

Gig E NIC ----(Jumbo Frames) ------GigE Switch ---------Gig Uplink-------Switch with Gig Uplink & 100Mb ports ----100Mb PC

You could put all your GigE machines in their own VLAN, and they can only talk to each other.  Then you could put all your slower machines in their own vlan, and they could only talk to each other.  You would need a router to connect the two VLANs, or a Layer 3 switch.

You PROBABLY don't "need" jumbo frames.  You PROBABLY don't "need" GigE.  Without knowing what you're doing, I would question whether this is just something nice to have, something that sounds cool, or something you need.

Normally the devices will talk to each other ... no matter if they make use of the jumbo frames or not. Devices that don't support these frames will fragment them into smaller ones. As this procedure consumes time (that you wanted to fasten with the big frames) you obviously try to send packets with equal or less the "path MTU" (what is the maximum size to get through without fragmentation).

Now you may set a DF (disable fragmentation) bit, to force network equipement to pass through the big frames ... but normally you may get (1) an error (destination unreachable) or (2) the network equipement just ignores your request (it has the "ignore DF" turned on - without you would be able to lock the internet ;-) )

So i don't see the problem in communication ... but to optimize MTU you will have to know, which devices support these frames and set them inbetween the path

Holger

But ... I don't think a regular 10/100 NIC will communicate to a GigE port doing jumbo frames - won't the regular 10/100 will see the jumbo frames as giants and discard them?

I'm copying lots of huge files between fast machines, with fast disks.  It currently takes quite a bit of time, while disk-to-disk transfers within a machine are easily 5 times faster.  The two main machines have CSA GigE controllers that support jumbo frames.  It's true that 1500 byte MTUs aren't going to kill me, but I'd like to figure out how one can enable jumbo frames, in my situation.

The solution needs to work for both SMB and NFS (over UDP).  So I think a solution relying on path MTU discovery, at the TCP level, won't work.

To those claiming you can just plug everything into a jumbo frame-capable switch, how do you explain this?

   The various network adapters support different MTU sizes, so make
   sure that you use the same MTU size for all the devices on the
   network. For example, you can not have a Gigabit Ethernet adapter
   using jumbo frame mode with a MTU size of 9000 bytes, while other
   adapters on the network use the default MTU size of 1500 bytes.
   10/100 Ethernet adapters do not support jumbo frame mode, so they
   are not compatible with this Gigabit Ethernet option. You also have
   to configure Ethernet switches to use jumbo frames, if jumbo frames
   are supported on your Ethernet switch.

   It is important to select the MTU size of the adapter early in the
   network setup so you can properly configure all the devices and
   switches. Also, many AIX tuning options are dependent upon the
   selected MTU size.

   (source:
   http://publib.boulder.ibm.com/infocenter/pseries/index.jsp?topic=/com.ibm.aix.doc/aixbman/prftungd/netperf3.htm)


This, too:

   If you are using Jumbo Ethernet frames, ensure that:

       * all end stations on a given LAN[3] have the same maximum
         transmission unit (MTU) setting;
       * intermediate stations such as switch ports in your LAN have an
         MTU equal to or greater than the end station's MTU.


   (source: http://docs.hp.com/en/AB545-90001/ch01s08.html)


I'm told that ethernet doesn't allow the switch to do frame fragmentation.  Is there any mechanism for the sender to learn about the MRUs of each node?  Otherwise, I simply don't see how one can have mixed MTUs on a layer 2 switch, when you can't rely on TCP's path MTU discovery.

Well, I've ordered a layer 2 unmanaged GigE switch with jumbo frame support (MTU 9000 bytes).  On that switch, I'll have PCs with 9000 byte MTU, 3000 MTU, and 1500 MTU.  We'll see what I can ping from what.

>>The various network adapters support different MTU sizes, so make
   sure that you use the same MTU size for all the devices on the
   network. For example, you can not have a Gigabit Ethernet adapter
   using jumbo frame mode with a MTU size of 9000 bytes, while other
   adapters on the network use the default MTU size of 1500 bytes.
   10/100 Ethernet adapters do not support jumbo frame mode, so they
   are not compatible with this Gigabit Ethernet option.

That's what I was saying, with my comment, "But ... I don't think a regular 10/100 NIC will communicate to a GigE port doing jumbo frames - won't the regular 10/100 will see the jumbo frames as giants and discard them?"

pseudocyber, I agree with your comment about any jumbo frames being received by a 1500 MTU NIC being discarded.  What I didn't know was whether there was any way a sender would know the path MTU, at the ethernet level.  My other hope was that the switch would fragment packets destined for a smaller MTU device.  As far as I can tell, neither of these is the case.


However, put simply, my question was:

    What's the simplest way to work around this?


If I'm right that you can't solve this problem at layer 2, I'm (eventually) planning to try putting 2 NICs in a Linux PC and using it as a bridge.  If anyone could comment on that approach, it would help.

Video, I'm not exactly sure.  I would think that the traffic would hit a jumbo capable switch and the switch would put the traffic in a transmit queue.  Then when it goes out the port it would see that it's a smaller mtu and fragment it.  Haven't really messed with it much.

MTU is used at layer 3 - switches operate at lower layers and may only forward packets like they receive them .. so they may not fragment big packets

Okay, I've tried sending different sizes of ICMP packets w/ the don't fragment bit set (via ping), via an unmanaged gigabit ethernet switch, between hosts with different MTU limits.  Like I thought, they didn't get through, when they exceeded the recipient's MTU.

I think my original question was under-specified (I didn't say that some traffic was not TCP/IP, so Irmoore's comment isn't exactly wrong).  I appreciate the effort pseudocyber put into getting more information.  So, I'm splitting the points between them.

Sorry, holger, but none of your comments seem entirely accurate.