Avatar of amigan_99
amigan_99
Flag for United States of America asked on

TCP Sliding Window and iSCSI

When you are running iSCSI, is TCP sliding window an important consideration? The situation is is Cisco UCS fabric interconnect to a Nexus 5k switch. The switch frequently drops packets inbound from the UCS and this appears to be an issue iSCSI frames from UCS being 1514 bytes which the interface on the Nexus is 1500 and jumbo framing is not enabled. I don't know why the vast majority of frames make it on through yet a significant number (in the millions) are dropped.
The port channel spikes up to about 10Gbps and most of that will be iSCSI. So the connection initiator to target works for the most part. I've planned to enable the jumbo frames as recommended by Cisco so that the 1514 iSCSI will be better processed and not dropped.

But my question is this: With iSCSI, are TCP conversations lengthy or very brief? To what degree would some dropped frames (.003%) in-path cause an issue for iSCSI TCP conversation? Or would this percentage just be noise that TCP connection orientedness should just deal with?
StorageNetworking ProtocolsCiscoTCP/IP

Avatar of undefined
Last Comment
Predrag Jovic

8/22/2022 - Mon
ASKER CERTIFIED SOLUTION
Dr. Klahn

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
GET A PERSONALIZED SOLUTION
Ask your own question & get feedback from real experts
Find out why thousands trust the EE community with their toughest problems.
SOLUTION
noci

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
GET A PERSONALIZED SOLUTION
Ask your own question & get feedback from real experts
Find out why thousands trust the EE community with their toughest problems.
SOLUTION
Predrag Jovic

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
amigan_99

ASKER

These were great thoughtful answers. Thank you!


I have the change planned as you describe Pedrag. Thank you for confirming it's non-impacting. Cisco said as much as well. Still with iSCSI traffic flowing through it I'll be holding my breath as I apply the policy! I wish they'd put that on another switch. But that's what I've inherited. Shared switch, share trunks in fact.

noci

Changing MTU on the switch will not hurt in any way or form on communication (buffers will get larger, also less buffers may be available). 


Another thing is changing MTU on the IP stack. The IP MTU of all systems in the same broadcast domain (aka LAN) needs to be the same, or things will grind to a halt.

So if iSCSI is on a separate (V)LAN then it can be changed.

Predrag Jovic

Another thing is changing MTU on the IP stack. The IP MTU of all systems in the same broadcast domain (aka LAN) needs to be the same, or things will grind to a halt.
Not really, at least for TCP traffic (I have no knowledge if there is some control mechanism for UDP traffic so I will consider only TCP traffic here - also I am deliberately ignoring the fact that MTU configured on host is not the same one MTU configured on Cisco switches)...

With all that said - the only requirement for switching equipment (the same VLAN) is that size of switch MTU configured on switches along path needs to be equal or bigger than MTU on hosts. Hosts are negotiating MSS during 3 way handshake, since  MSS + 40 = MTU end hosts will be aware of each other's MTU therefore MTU on hosts can be different.
I started with Experts Exchange in 2004 and it's been a mainstay of my professional computing life since. It helped me launch a career as a programmer / Oracle data analyst
William Peck
noci

MTU != MSS.

Unix interface MTU = 1500 (ethernet) is net payload....  ==> switch packetsize = 1514.  (or 1518 with VLAN tags).


For any device: if a packet is larger than then MTU it is ignored. (and only counted in the statistic counter: packets too large).

TCP may have MSS negotiation UDP does not. So all UDP frames that are 1501+ bytes will be lost. 


You will experience blocked sessions as at sometime one side decides to send something larger and then to wait for the result that never comes.

On switches you need enough to capture any packet... 


Also if the endpoint communicate both with MTU = 9000, and there is some intermediate having less somewhere (Switch,router..) then MSS will be around 8960 range.

No communication will happen though.. (unless packets actualy are below 1500 bytes.. 

You mean you need to determine the PMTU (path largest MTU that still works) that should be determined BEFORE starting MSS negotiation in TCP.


The same issue will happen with devices tunnelling traffic, and some traffic having the don;t fragment bit set... been seen it. Some 1500 byte packets were not delivered across a PPPoE connection causing random breakdowns of Phone calls. Some people could call, some could not. (no Invites arriving) due to too many options and some path info in the SIP packets.

Predrag Jovic

If MSS + 40 = MTU  (I did not want to go into details what 40 is in this case)
then obviously
MTU != MSS

Regarding
Unix interface MTU = 1500 (ethernet) is net payload....  ==> switch packetsize = 1514.  (or 1518 with VLAN tags).

For any device: if a packet is larger than then MTU it is ignored.
I wrote:
I am deliberately ignoring the fact that MTU configured on host is not the same one MTU configured on Cisco switches
Meaning MTU = 1500 on host L3 interface is not equal to MTU = 1500 on switch for L2 transport, but MTU = 1500 on SVI is the same one MTU = 1500 on host. I did not want to go into details and still not planning to do that. :)
TCP may have MSS negotiation UDP does not. So all UDP frames that are 1501+ bytes will be lost.
My explanation is ignoring discussion regarding situation when hosts are sending L2 frames bigger than what switch can accept:
With all that said - the only requirement for switching equipment (the same VLAN) is that size of switch MTU configured on switches along path needs to be equal or bigger than MTU on hosts.
Translation:
If switch configured L2 frame size is equal or bigger than what will be sent by end hosts in the traffic will not be dropped by switch.
If switch configured MTU is 9126 bytes than 5000 or 9000 bytes frames send by host will not be dropped (when all overhead is added).

So I guess we both agree on that.

Regarding:
UDP does not negotiate MSS
This only means that UDP has no negotiation integrated into protocol itself, however application can still negotiate MSS if needed/wanted, just as UDP has no reliability, but can be implemented into application,
You mean you need to determine the PMTU (path largest MTU that still works) that should be determined BEFORE starting MSS negotiation in TCP.


The same issue will happen with devices tunnelling traffic, and some traffic having the don;t fragment bit set... been seen it. Some 1500 byte packets were not delivered across a PPPoE connection causing random breakdowns of Phone calls. Some people could call, some could not. (no Invites arriving) due to too many options and some path info in the SIP packets.
No. PMTUD is related to L3 - there is no PMTDU in strict L2 (VLAN) so it is irrelevant for the topic.   since in previous post I am discussing strictly Layer 2 - this specific sentence
The IP MTU of all systems in the same broadcast domain (aka LAN) needs to be the same, or things will grind to a halt.
I am stating that MTU on different hosts/systems in the same VLAN doesn't have to be the same as long as switch can support bigger L2 frames than L2 frames that are being send by end hosts. And I was explaining that on example of  TCP since question is related to TCP behavior.
1. switches along path can be configured with MTU 4000, 5000, 9000 as long as end hosts L2 frames + any potential header is smaller than 4000 (the smallest configured MTU on switched path).
2. If different hosts in the same VLAN are configured with different MTU values in the case of TCP there will be no issues as long as statement 1 is satisfied. UDP could be more complicated regarding hosts, but I am network guy and that's why I stated
I will consider only TCP traffic here

Best practice is to configure all switches in domain to support the same MTU value, but TCP can still work with different MTU sizes if few simple rules are satisfied...