Can't RDP over an IPSec VPN tunnel

MTU limitations sound like a plausible cause. Note that the IPSec encapsulation will increase the size of any packet sent over the tunnel by 73, so if the MTU of either side doesn't take this into account, you'll get fragmentation errors pretty regularly. https://rezanayem.wordpress.com/2008/12/14/adjustingtuning-tcp-mtu-for-remote-desktop-connection/ explains how to adjust the MTU for RDP. I would drop it to something that allows sufficient room to meet the MTU of both sides of the tunnel with the additional space required by the IPSec encapsulation.

Thanks for responding Adam.  Where do you make that change?  On the Terminal Server end?

I would think it needs to be set on both sides. I don't think the PPPoE settings are necessary, but you can change those as well, if you wish.

I've just tried adjusting both client ends.  Made the RDP connection process slower but didn't fix the issue.

Try to enable PMTU then the optimal size will be found out during connecting.

I'm not sure what you mean by that noci.

MPTU is Path MTU discovery. A method that uses the don't fragment bit and decreases packet size until it fits to connect to a remote station.
Thist way local traffic will still used maximum size (1500) and tunneled traffic will get reduced MTU until it does fit.

It's not much help, but I would be surprised if it is an MTU issue.  If MTU is an issue, which is not that common, it is usually a problem with the internet part of the route.  You have no problem client to other PCs over the Internet/VPN.

I assume there is no problem connecting to the terminal server from the local LAN?

Yes there are no issues from the home office laptop to other servers on the main office network.  What makes it look like MTU issues is the fact you can connect to other servers, just not the Terminal Server, as the Terminal server has additional encryption or encapsulation on its packets that other servers don't.

We've almost eliminated everything else as being the cause.   The ISP is going to send someone onsite to check for causes of the speed issues.  I've checked and the firewall is off on the Terminal server.  I've disabled AV to test.  I've checked the users who're allowed to connect to the server, checked the RDP settings in the Settings Host Configurator.  Nothing is blocking this user.

When we RDP to the other servers the credential window pops up instantly and within seconds of entering the correct credentials, we're logged into the server.  When RDPing into the Terminal Server, the credential window takes 3-5 secs to appear and once the credentials are entered, it quickly goes through initiating, and connecting (?) and then waits on securing the remote connection for up to two minutes before erroring out.

Theres nothing in event logs on either the client nor the Terminal Server to suggest whats going wrong.  Wireshark indicates that there are numerous 'retransmission' messages being sent, which might be suggestive of packet loss associated with an incompatible MTU settings on the connection somewhere.

Myself and my colleagues are truely stumped on this one.  When we set up an IPSEC VPN tunnel at a Branch office if normally just works.

Can the user connect if on the same LAN  as the TS ?

Are you using a self-signed certificate or 3rd party?  It can hang at "connecting" if issues with the certificate.  On the LAN it may use the self signed, auto-generated one, which wouldn't work remotely unless installed on the connecting device.

Terminal server also make Large packets for data traveling back from TS -> client. (Screen images or parts of it).
The encryption of the RDP protocol would fit WITHIN the MTU, the ones on the tunnel won't.

@Rob - yes they can RDP to any other server on the same network as the RDP server.
Its a self signed certificate on the Terminal Server, the one that auto updates every 6 months.  I have checked that and its fine and previously tried installing it on the client to no avail.  I don't think this is the issue.

@noci.  Thats my belief also, but I'm not sure how to fix it.

I have no windows systems so here the links i could find o the issue of setting MTU & PMTU discovery.

https://technet.microsoft.com/en-us/library/cc958871.aspx
https://technet.microsoft.com/en-us/library/cc938197.aspx
https://serenity-networks.com/how-to-change-the-mtu-in-windows-server-2008-2012/

I hope this helps.

If PMTU doesn't work out then a value of 1400  is safe to use.

I am still a little skeptical of MTU issues.  MTU is usually a bottle neck at the router, but you can RDP to other devices.  I know it has been suggested TS handles MTU sizes differently.

However, just to toss in some other ideas:  Is the remote computer a member of the domain?  Domain PCs, like those on the LAN have the cert added automatically.

Also, is it possible to do a test using port forwarding rather than the VPN?  If the problem persists it might further indicate a certificate or other problem.

Hi @noci,

I've encountered many many articles similar to these that show me how to do what they describe.  Unfortunately I'm not sure what the magic combo of settings are that I need.
I know how to make the changes, I just don't know what those changes are.

@Rob.  Yes it is a member of the domain.   I will try setting up a portforward to the server directly.

Set up a port forward on the Host domain router and was able to RDP into the Terminal Server without issue.

That proves I am wrong, and it's not a certificate issue.

Have had my manager having a look at the issue this afternoon.  Attached is the maximum size of packets we can send (in ping) from either end.  The issue seems to be with traffic travelling down the tunnel to the remote end.  At the Host Office the router is a Cisco.  At the REmote office it is a Draytek.

Googling I see newer server versions, especially in Hyper-V environment can use packets with an MTU larger than 1500.  That alone can cause problems such as you have described and as outlined in the article below.  The VPN likely uses 1492 (years ago it was 1400 over VPN) and default settings of systems should automatically adjust for that but I am guessing the server does not, thus explaining the difference between PCs and the server.  At the bottom of the article it outlines how to change the MTU.  I would change to 1300.  If it works, then gradualy increase to a max of 1492.
https://blogs.technet.microsoft.com/askpfeplat/2014/12/01/psa-incorrect-mtu-size-causes-connectivity-issues-with-windows-server-2012-and-windows-server-2012-r2/

Perhaps you have already seen the article.