Winsock issue with MTU


We have a windows service using the winsock2 libraries. It uses the WSASend and TransmitFile functions to send data to another windows service of the same kind. This service has been working well for a few years.

This windows service is used to transfer files from a sub site to a head site and vice versa, the link between the sites has recently been encrypted using IPSEC over GRE, this has caused a reduction of the MTU on the link. However, some packets sent by this service have a Do not fragment flag set. This then causes the router to send an ICMP message (as part of Path MTU Discovery) to the server asking it to break down the size of the packet so that the router can send the smaller packets.

When looking at the traffic, the packets are never broken down and sent back to the router. This means that the service at the remote site never gets these packets. As far as the windows service is concerned, the WSASend and TransmitFile methods return success.

We want to avoid dropping the MTU of the server that the service runs on, as that will also affect communications on the local network.

Is it the responsibility of the windows service to deal with the ICMP message, or can we setup the server it runs on to handle the ICMP message? Either way, how do we make it so that it works?

Thanks in advance.
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

If you can reduce the MTU on one of the peers, the advertized MSS will also reduce the packet size sent by the other peer. So, if you can do that for one of each pair of peers, it would be an "easy fix" (really a workaround).

I'll assume you can't do that either though for the rest of this post ...

You say that changing the MTU would impact the local network, but that's only the case if the problematic traffic is sent using the same network device that is connected to the local network. The MTU can be set separately per network device, so if you use a different one for the local network, it will not be impacted by a change of the MTU on the other device.

If that doesn't help either, then we should try to find out the actual problem. I don't know the inner workings of Winsock specifically, so I don't know how it handles the ICMP "Destination Unreachable - fragmentation needed and DF set" message, but let's first make sure that the ICMP message actually arrives at the server (ie. that it isn't filtered out somewhere along the way). Can you verify that with a packet sniffer on the server ? If the ICMP packet doesn't arrive, then make sure it's not filtered out on the network, and it might just start working again ...

Assuming it does arrive ... we'll have to dive into WinSock heh. But I would first check the other options I mentioned ...
>>>> However, some packets sent by this service have a Do not fragment flag set.
It would recommend to either removing that flag or send smaller packages.

Handling the ICMP message is hardly a solution cause if the packet couldn't be fragmented how should a handler do it anyway?
WinSock can't change the packet size (due to the DF flag), but the server application apparently doesn't receive an error either (since WSASend and TransmitFile return succesfully).

This makes me think that either :

(a) the ICMP message hasn't arrived at the server
(b) WinSock doesn't know how to handle it
(c) WinSock knows how to handle it, but discards it for some reason
(d) WinSock handles it correctly, and there is a way to get the information in the application - but I don't know WinSock sufficiently to know how ...
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

neotechnologyAuthor Commented:
We have run a packet sniffer in a test environment, and the results show that the router connected to the local area network sends a Path MTU Discovery ICMP message back to our sending server service, but the server never splits that packet and resends it. In the test environment the server is directly attached to the router sending the ICMP packet, and windows firewall has been enabled.
We already have a workaround in place by adjusting the MTU of all packets passing through the Cisco router. What we want to know is path MTU discovery handled naturally in the winsock or is it supposed to be handled within our code (or not at all...).
>>>> What we want to know is path MTU discovery handled naturally in the winsock or is it supposed to be handled within our code (or not at all...).

Neither the receiving server nor the sending server where the ICMP message was sent to, can handle the issue due to the DF flag. The router has the problem that ICMP messages were not allowed to generate additional messages. So, the only way to handle it would be to let fail the initial call what seems impossible to me as the sending action already was terminated and the ICMP messages were on a quite different layer.

From all that, I would assume (not really know) that the ICMP has not the purpose to get directly handled but only is a protocol message, which could be evaluated by other systems. I am thinking of mails with a wrong receiver address. The original mail goes out without error but you finally got a return mail which tells that the mail could not be delivered.  Following that thought, the sending service should generate some kind of error request passed to the sending client (which would need to open an interface for that) as response of the ICMP message. Then, the client could split the failed message and send it again. Or, maybe easier, you only write the ICMP to some kind of error log, and would need to change your sending process so that those issues couldn't occur in future.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
>> and the results show that the router connected to the local area network sends a Path MTU Discovery ICMP message back to our sending server service

But is it also received ?
Is it blocked by the firewall ?
Or does it get discarded somewhere on the network ? (in a different router)

>> but the server never splits that packet and resends it.

The thing is that the packet was marked as unsplittable (the DF flag was set - I assume you did that using IP_DONTFRAGMENT), so there's not much that WinSock can do ... It is not allowed to split the packet, because you told it not to :)

So, this pretty much has to be handled by your code ... Only, to be able to do that, you should get some kind of indication when the ICMP "Destination Unreachable - fragmentation needed and DF set" message is received (assuming it is received - see above). I'll see if I can find something on how WinSock handles that.
Oops, seems Alex already covered the second part of my post :)
>>>> Oops, seems Alex already covered the second part of my post :)
Not so good, that a different view on it would be redundant.

What I don't know is whether the original send call already has returned (in that case the Winsock has no chance to handle it) or whether the ICMP message was still within the process of sending the original message (but only was ignored til now).

Maybe neotechnology could answer that from their protocols.
This MSDN topic claims that WinSock DOES support PMTUD (even though I assumed before that it didn't, based on your observations) :

It needs the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters to have a value EnablePMTUDiscovery that is set to 1.

I didn't find any information about when this support was added, and I don't know which version of WinSock (or Windows for that matter) you have. So, you might want to find out whether your specific WinSock version supports it.

My first bet would still be that WinSock doesn't actually receive the "Destination Unreachable - fragmentation needed and DF set" message though.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Programming Languages-Other

From novice to tech pro — start learning today.