Nexus to UCS - seeing "jumbo packets" in sho int but shouldn't

amigan_99
amigan_99 used Ask the Experts™
on
I have a Nexus 6k connecting to UCS Fabric Interconnect. What's puzzling to me is that there are are "jumbo packet" - millions of them counted in the sho interface commands. But the MTU on the Nexus side is 1500. And the same thing is true if I log onto the FI - I see jumbo packet counters highly incremented and yet the interfaces are all set to 1500 MTU. Any thought as to how I could be seeing these big counts of jumbo "packets" (Frames would have been a better term no?) when I don't think I have jumbo framing configured on either side of these links?

RTR01# sho int Eth 1/3
Ethernet1/3 is up
Dedicated Interface
  Belongs to Po2
  Hardware: 1000/10000 Ethernet, address: 002a.5cc2.4aca (bia 002a.5cc2.4aca)
  Description: TO-MY-UCS01-A
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
  reliability 255/255, txload 89/255, rxload 4/255
  Encapsulation ARPA
  Port mode is trunk
  full-duplex, 10 Gb/s, media type is 10G
  Beacon is turned off
  Input flow-control is off, output flow-control is off
  Rate mode is dedicated
  Switchport monitor is off
  EtherType is 0x8100
  Last link flapped 4d02h
  Last clearing of "show interface" counters 17w1d
  1 interface resets
  30 seconds input rate 109418408 bits/sec, 19662 packets/sec
  30 seconds output rate 3958742016 bits/sec, 346160 packets/sec
  Load-Interval #2: 5 minute (300 seconds)
    input rate 166.25 Mbps, 24.08 Kpps; output rate 3.51 Gbps, 309.55 Kpps
  RX
    411181255573 unicast packets  8610632 multicast packets  43547045 broadcast packets
    411233413250 input packets  451753729928599 bytes
    258996359356 jumbo packets  0 storm suppression bytes
    0 runts  0 giants  0 CRC  0 no buffer
    0 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  11403888 input discard
    0 Rx pause
  TX
    861157080800 unicast packets  181835706 multicast packets  84446029 broadcast packets
    861423362536 output packets  1048056776519017 bytes
    654515246211 jumbo packets
    0 output error  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble 2159 output discard
    0 Tx pause
 
 
RTR01# sho int po2
port-channel2 is up
vPC Status: Up, vPC number: 2
  Hardware: Port-Channel, address: 002a.5bb2.4acb (bia 002a.5bb2.4acb)
  MTU 1500 bytes, BW 20000000 Kbit, DLY 10 usec
  reliability 255/255, txload 91/255, rxload 13/255
  Encapsulation ARPA
  Port mode is trunk
  full-duplex, 10 Gb/s
  Input flow-control is off, output flow-control is off
  Switchport monitor is off
  EtherType is 0x8100
  Members in this channel: Eth1/3, Eth1/4
  1 interface resets
  30 seconds input rate 1173067696 bits/sec, 124425 packets/sec
  30 seconds output rate 7362936056 bits/sec, 652549 packets/sec
  Load-Interval #2: 5 minute (300 seconds)
    input rate 1.10 Gbps, 117.78 Kpps; output rate 7.15 Gbps, 629.37 Kpps
  RX
    839473500372 unicast packets  18197725 multicast packets  101243659 broadcast packets
    839592941756 input packets  918753054159174 bytes
    527307331519 jumbo packets  0 storm suppression bytes
    0 runts  0 giants  0 CRC  0 no buffer
    0 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  22919002 input discard
    0 Rx pause
  TX
    2126666792856 unicast packets  271549068 multicast packets  138066875 broadcast packets
    2127076408784 output packets  2672598826144131 bytes
    1683451122508 jumbo packets
    0 output error  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble 2159 output discard
    0 Tx pause
  1 interface resets
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Dr. KlahnPrincipal Software Engineer
Commented:
Attach a network sniffer, examine some of the offending packets, and find out where they are coming from.  With that information, what to do about the problem should be more clear.
Distinguished Expert 2018

Commented:
But the MTU on the Nexus side is 1500
Not necessarily so.
"show interface" command on Nexus 6000 is always displaying 1500 as the MTU for L2 ports, no matter which MTU size is actually configured.


To verify configured MTU issue command:
show queuing interface <interface>

On Nexus 6000 (and some others) MTU size is configured under QoS:

policy-map type network-qos jumbo
  class type network-qos class-default
          mtu 9216
system qos
   service-policy type network-qos jumbo

For more details please read article: Configure and Verify Maximum Transmission Unit on Cisco Nexus Platforms
amigan_99Network Engineer

Author

Commented:
Here is a sample from one of the Nexus ports:

1# show queuing interface e1/3
Ethernet1/3 queuing information:
  TX Queuing
    qos-group  sched-type  oper-bandwidth
        0       WRR            100
 
  RX Queuing
    qos-group 0
    q-size: 100160, HW MTU: 1500 (1500 configured)
    drop-type: drop, xon: 0, xoff: 0
    Statistics:
        Pkts received over the port             : 2980090300
        Ucast pkts sent to the cross-bar        : 2830427525
        Mcast pkts sent to the cross-bar        : 2593195503
        Ucast pkts received from the cross-bar  : 829757361
        Pkts sent to the port                   : 2093176636
        Pkts discarded on ingress               : 11903394
        Per-priority-pause status               : Rx (Inactive), Tx (Inactive)
Python 3 Fundamentals

This course will teach participants about installing and configuring Python, syntax, importing, statements, types, strings, booleans, files, lists, tuples, comprehensions, functions, and classes.

amigan_99Network Engineer

Author

Commented:
I went through all the UCS QOS policies and vNIC Templates and all of those have 1500 MTU set. Just can not figure out how Jumbos are incrementing on either side of the 6k-FI link.

For interest I did look at the 10Gbps links attached to our storage system and there no jumbos are seen tx nor rx.
Distinguished Expert 2018

Commented:
Could be (if hosts are not experiencing delays during establishing sessions) that interfaces are configured as trunk (EtherType is 0x8100) that those jumbo packets are actually mini jumbos because of additional 4 bytes added for Dot1q header.
Ethernet frame of size 1518 bytes are counted as normal size packets and 1522 bytes (dot1q frames) can be counted as baby giant frames, but that should be verified with command "sh int counters" (there was similar IOS behavior previously).

You can look into more specifics by issuing:
 sh int <interface> counters detail all

MTU for Nexus 6000
The Cisco Nexus device is a Layer 2 switch, and it does not support packet fragmentation. A maximum transmission unit (MTU) configuration mismatch between ingress and egress interfaces may result in packets being truncated.

When configuring MTU, follow these guidelines:

    MTU is specified per system class. The system class allows a different MTU for each class of traffic but they must be consistent on all ports across the entire switch. You cannot configure MTU on the interfaces.

    Fibre Channel and FCoE payload MTU is 2158 bytes across the switch. As a result, the rxbufsize for Fibre Channel interfaces is fixed at 2158 bytes. If the Cisco Nexus device receives an rxbufsize from a peer that is different than 2158 bytes, it will fail the exchange of link parameters (ELP) negotiation and not bring the link up.

    Enter the system jumbomtu command to define the upper bound of any MTU in the system. The system jumbo MTU has a default value of 9216 bytes. The minimum MTU is 2158 bytes and the maximum MTU is 9216 bytes.

    The system class MTU sets the MTU for all packets in the class. The system class MTU cannot be configured larger than the global jumbo MTU.

    The FCoE system class (for Fibre Channel and FCoE traffic) has a default MTU of 2158 bytes. This value cannot be modified.

    The switch sends the MTU configuration to network adapters that support DCBX.
amigan_99Network Engineer

Author

Commented:
I don't see any mini-jumbos - but that's an interesting line of thought on the mini-jumbo.
system jumbomtu is not in the config (sho run | i jumbo). I didn't know of the command
 sh int <interface> counters detail all - that's interesting too.

53. Rx Packets from 1519 to 1548 bytes: = 0
61. Tx Packets from 1519 to 1548 bytes: = 0


 sh int <interface> counters detail all

Ethernet1/3
  64 bit counters:
   0.                      rxHCTotalPkts = 412961914695
   1.                       txHCTotalPks = 878499335066
   2.                    rxHCUnicastPkts = 412909215208
   3.                  rxHCMulticastPkts = 8673509
   4.                  rxHCBroadcastPkts = 44025978
   5.                         rxHCOctets = 453386918744653
   6.                    txHCUnicastPkts = 878231238623
   7.                  txHCMulticastPkts = 182587806
   8.                  txHCBroadcastPkts = 85508636
   9.                         txHCOctets = 1071484599719086
  10.                 rxTxHCPkts64Octets = 1381771483
  11.            rxTxHCpkts65to127Octets = 225716034523
  12.           rxTxHCpkts128to255Octets = 50878226796
  13.           rxTxHCpkts256to511Octets = 17059204464
  14.          rxTxHCpkts512to1023Octets = 19091736931
  15.         rxTxHCpkts1024to1518Octets = 48249231515
  16.         rxTxHCpkts1519to1548Octets = 0
  17.                    rxHCTrunkFrames = 412961036201
  18.                    txHCTrunkFrames = 878487091453
  19.                     rxHCDropEvents = 0
  19.                    InLayer3Unicast = 0
  20.              InLayer3UnicastOctets = 0
  21.                  InLayer3Multicast = 0
  22.            InLayer3MulticastOctets = 0
  23.                   OutLayer3Unicast = 0
  24.             OutLayer3UnicastOctets = 0
  25.                 OutLayer3Multicast = 0
  26.           OutLayer3MulticastOctets = 0
  27.                     InLayer3Routed = 0
  28.               InLayer3RoutedOctets = 0
  29.                    OutLayer3Routed = 0
  30.              OutLayer3RoutedOctets = 0
  31.              InLayer3AverageOctets = 0
  32.             InLayer3AveragePackets = 0
  33.             OutLayer3AverageOctets = 0
  34.            OutLayer3AveragePackets = 0
 
  All Port Counters:
   0.                        Rx Packets: = 412961914695
   1.                          Rx Bytes: = 453386918744653
   2.                  No Buffer Errors: = 0
   3.              Rx Broadcast Packets: = 44025978
   4.              Rx Multicast Packets: = 8673509
   5.                Rx Unicast Packets: = 412909215208
   6.                  Rx Jumbo Packets: = 259513533404
   7.                       Runt Errors: = 0
   8.              Rx Storm Suppression: = 0
   9.                      Input Errors: = 0
  10.                  Input CRC Errors: = 0
  11.                        ECC Errors: = 0
  12.                    Overrun Errors: = 0
  13.                    Ignored Errors: = 0
  14.                   Watchdog Errors: = 0
  15.              tx broadcast packets: = 85508636
  16.              tx multicast packets: = 182587806
  17.                tx unicast packets: = 878231238623
  18.                  tx jumbo packets: = 669571510647
  19.                          Rx Pause: = 0
  20.                    Dribble Errors: = 0
  21.               If Down Drop Errors: = 0
  22.             Bad Etype Drop Errors: = 0
  23.             Bad Proto Drop Errors: = 0
  24.                        tx packets: = 878499335066
  25.                          tx bytes: = 1071484599719086
  26.                   Underrun Errors: = 0
  27.                     Output Errors: = 0
  28.                  Collision Errors: = 0
  29.                            Resets: = 0
  30.                     Babble Errors: = 0
  31.             Late Collision Errors: = 0
  32.                   Deferred Errors: = 0
  33.               Lost Carrier Errors: = 0
  34.                 No Carrier Errors: = 0
  35.                          Tx Pause: = 0
  36.           Single Collision Errors: = 0
  37.            Multi-Collision Errors: = 0
  38.           Excess Collision Errors: = 0
  39.                     Jabber Errors: = 0
  40.                Short Frame Errors: = 0
  41.              Input Discard Errors: = 12552463
  42.          Bad Encapsulation Errors: = 0
  43.                           txErrors = 0
  44.                     Symbol Errors: = 0
  45.             Output Dropped Errors: = 2159
  46.                            SQETest = 0
  47.     Rx Packets from 0 to 64 bytes: = 737009
  48.   Rx Packets from 65 to 127 bytes: = 86867695274
  49.  Rx Packets from 128 to 255 bytes: = 21969093168
  50.  Rx Packets from 256 to 511 bytes: = 6546593354
  51. Rx Packets from 512 to 1023 bytes: = 7549608451
  52. Rx Packets from 1024 to 1518 bytes: = 30514654037
  53. Rx Packets from 1519 to 1548 bytes: = 0
  54.                  Rx Trunk Packets: = 412961036201
  55.     Tx Packets from 0 to 64 bytes: = 1381034474
  56.   Tx Packets from 65 to 127 bytes: = 138848339249
  57.  Tx Packets from 128 to 255 bytes: = 28909133628
  58.  Tx Packets from 256 to 511 bytes: = 10512611110
  59. Tx Packets from 512 to 1023 bytes: = 11542128480
  60. Tx Packets from 1024 to 1518 bytes: = 17734577478
  61. Tx Packets from 1519 to 1548 bytes: = 0
  62.                  Tx Trunk Packets: = 878487091453
  63.                  Output BPDU Lost: = 0
  64.                  Output COS0 Lost: = 0
  65.                  Output COS1 Lost: = 0
  66.                  Output COS2 Lost: = 0
  67.                  Output COS3 Lost: = 0
  68.                  Output COS4 Lost: = 0
  69.                  Output COS5 Lost: = 0
  70.                  Output COS6 Lost: = 0
  71.                  Output COS7 Lost: = 0
amigan_99Network Engineer

Author

Commented:
With sh hardware internal bigsur port eth 1/3 - I was able to see more packet size classifications:

RX_PKT_SIZE_IS_64              | 1126302
RX_PKT_SIZE_IS_65_TO_127       | 46055060306
RX_PKT_SIZE_IS_128_TO_255      | 26291129766
RX_PKT_SIZE_IS_256_TO_511      | 8349480103
RX_PKT_SIZE_IS_512_TO_1023     | 9051470200
RX_PKT_SIZE_IS_1024_TO_1518    | 2624797938
RX_PKT_SIZE_IS_1519_TO_2047    | 1830263373  <<<

So perhaps the mini jumbo is in play but slightly larger?
Distinguished Expert 2018

Commented:
I am sorry the above imprecision, actual technical term is "Baby Giant" not mini jumbo (corrected it in post already).

Baby giant frames refer to Ethernet frame size up to 1600 bytes
amigan_99Network Engineer

Author

Commented:
I finally got a packet capture. So the lions share of these packets are iSCSI with 1514 byte frames. The MTU is set to 1500.

iSCSI is TCP based. What's the likely end impact of this traffic traversing the link with frames larger than the 1500 MTU?
Distinguished Expert 2018
Commented:
Since there is no fragmentation on 2 oversized packets should be either dropped or truncated (but I am not sure if it's the case here, since I would expect more errors listed under interface)
amigan_99Network Engineer

Author

Commented:
I've a change request in to permit jumbo framing at both sides.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial