Can't RDP over an IPSec VPN tunnel

I have an IPSec VPN tunnel going between a main office and a home office (Cisco router at the main office end and Draytek at the home office end).  I am wanting the user to be able to log into the Terminal Server down the tunnel from home to the main office.  From her computer I can RDP to any other server but I can't RDP to the Terminal server.  It gets stuck on 'Securing remote Connection' after entering the credentials for up to 2 mins before eventually erroring out with a non-descript general 'Can't connect' error.  We've tried on a different laptop (Win 10 vs Win7, and wired and wireless) and have replaced the home office router with another model Draytek but the issue has remained the same.

After A LOT of googling and a little bit of Wiresharking, and trial and error I think the issue is down to MTU issues but I'm not an expert in this field and I'm trying to learn all I can.

My testing with 'ping -f -l' I've found:
  • Terminal Server at the main office can ping with a limit of 1472 to the router at the main office and out to Google (4.2.2.2)
  • Terminal Server cannot ping the home office router at 1472 - its too big.  I cut it down to 1400 and the first ping timed out and then was too big
  • On the laptop at the home office end I can ping with a limit of 1472 to the home office router, to Google, AND to the router at the main office end.

Another interesting and likely related symptom is that the internet at the home office end is very slow.  Its meant to be Ultra Fast Broadband (fibre), but on various Ookla speed tests I've done at different times, I've gotten anywhere from 12Mbps up to 50Mbps, although most of the time its at the lower end of that scale.  I have a call logged with the ISP to check that theres nothing wrong at their end.

Am I barking up the right tree with chasing the MTU tail?  What else should I look at?  How do I solve this one?
Karen BrownEngineerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Adam BrownSr Solutions ArchitectCommented:
MTU limitations sound like a plausible cause. Note that the IPSec encapsulation will increase the size of any packet sent over the tunnel by 73, so if the MTU of either side doesn't take this into account, you'll get fragmentation errors pretty regularly. https://rezanayem.wordpress.com/2008/12/14/adjustingtuning-tcp-mtu-for-remote-desktop-connection/ explains how to adjust the MTU for RDP. I would drop it to something that allows sufficient room to meet the MTU of both sides of the tunnel with the additional space required by the IPSec encapsulation.
Karen BrownEngineerAuthor Commented:
Thanks for responding Adam.  Where do you make that change?  On the Terminal Server end?
Adam BrownSr Solutions ArchitectCommented:
I would think it needs to be set on both sides. I don't think the PPPoE settings are necessary, but you can change those as well, if you wish.
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

Karen BrownEngineerAuthor Commented:
I've just tried adjusting both client ends.  Made the RDP connection process slower but didn't fix the issue.
nociSoftware EngineerCommented:
Try to enable PMTU then the optimal size will be found out during connecting.
Karen BrownEngineerAuthor Commented:
I'm not sure what you mean by that noci.
nociSoftware EngineerCommented:
MPTU is Path MTU discovery. A method that uses the don't fragment bit and decreases packet size until it fits to connect to a remote station.
Thist way local traffic will still used maximum size (1500) and tunneled traffic will get reduced MTU until it does fit.
Rob WilliamsCommented:
It's not much help, but I would be surprised if it is an MTU issue.  If MTU is an issue, which is not that common, it is usually a problem with the internet part of the route.  You have no problem client to other PCs over the Internet/VPN.

I assume there is no problem connecting to the terminal server from the local LAN?
Karen BrownEngineerAuthor Commented:
Yes there are no issues from the home office laptop to other servers on the main office network.  What makes it look like MTU issues is the fact you can connect to other servers, just not the Terminal Server, as the Terminal server has additional encryption or encapsulation on its packets that other servers don't.

We've almost eliminated everything else as being the cause.   The ISP is going to send someone onsite to check for causes of the speed issues.  I've checked and the firewall is off on the Terminal server.  I've disabled AV to test.  I've checked the users who're allowed to connect to the server, checked the RDP settings in the Settings Host Configurator.  Nothing is blocking this user.

When we RDP to the other servers the credential window pops up instantly and within seconds of entering the correct credentials, we're logged into the server.  When RDPing into the Terminal Server, the credential window takes 3-5 secs to appear and once the credentials are entered, it quickly goes through initiating, and connecting (?) and then waits on securing the remote connection for up to two minutes before erroring out.

Theres nothing in event logs on either the client nor the Terminal Server to suggest whats going wrong.  Wireshark indicates that there are numerous 'retransmission' messages being sent, which might be suggestive of packet loss associated with an incompatible MTU settings on the connection somewhere.

Myself and my colleagues are truely stumped on this one.  When we set up an IPSEC VPN tunnel at a Branch office if normally just works.
Rob WilliamsCommented:
Can the user connect if on the same LAN  as the TS ?

Are you using a self-signed certificate or 3rd party?  It can hang at "connecting" if issues with the certificate.  On the LAN it may use the self signed, auto-generated one, which wouldn't work remotely unless installed on the connecting device.
nociSoftware EngineerCommented:
Terminal server also make Large packets for data traveling back from TS -> client. (Screen images or parts of it).
The encryption of the RDP protocol would fit WITHIN the MTU, the ones on the tunnel won't.
Karen BrownEngineerAuthor Commented:
@Rob - yes they can RDP to any other server on the same network as the RDP server.
Its a self signed certificate on the Terminal Server, the one that auto updates every 6 months.  I have checked that and its fine and previously tried installing it on the client to no avail.  I don't think this is the issue.

@noci.  Thats my belief also, but I'm not sure how to fix it.
nociSoftware EngineerCommented:
I have no windows systems so here the links i could find o the issue of setting MTU & PMTU discovery.

https://technet.microsoft.com/en-us/library/cc958871.aspx
https://technet.microsoft.com/en-us/library/cc938197.aspx
https://serenity-networks.com/how-to-change-the-mtu-in-windows-server-2008-2012/

I hope this helps.

If PMTU doesn't work out then a value of 1400  is safe to use.
Rob WilliamsCommented:
I am still a little skeptical of MTU issues.  MTU is usually a bottle neck at the router, but you can RDP to other devices.  I know it has been suggested TS handles MTU sizes differently.

However, just to toss in some other ideas:  Is the remote computer a member of the domain?  Domain PCs, like those on the LAN have the cert added automatically.

Also, is it possible to do a test using port forwarding rather than the VPN?  If the problem persists it might further indicate a certificate or other problem.
Karen BrownEngineerAuthor Commented:
Hi @noci,

I've encountered many many articles similar to these that show me how to do what they describe.  Unfortunately I'm not sure what the magic combo of settings are that I need.
I know how to make the changes, I just don't know what those changes are.
Karen BrownEngineerAuthor Commented:
@Rob.  Yes it is a member of the domain.   I will try setting up a portforward to the server directly.
Karen BrownEngineerAuthor Commented:
Set up a port forward on the Host domain router and was able to RDP into the Terminal Server without issue.
Rob WilliamsCommented:
That proves I am wrong, and it's not a certificate issue.
Karen BrownEngineerAuthor Commented:
Have had my manager having a look at the issue this afternoon.  Attached is the maximum size of packets we can send (in ping) from either end.  The issue seems to be with traffic travelling down the tunnel to the remote end.  At the Host Office the router is a Cisco.  At the REmote office it is a Draytek.

MTU max transfer packets from source to endpoints.
Rob WilliamsCommented:
Googling I see newer server versions, especially in Hyper-V environment can use packets with an MTU larger than 1500.  That alone can cause problems such as you have described and as outlined in the article below.  The VPN likely uses 1492 (years ago it was 1400 over VPN) and default settings of systems should automatically adjust for that but I am guessing the server does not, thus explaining the difference between PCs and the server.  At the bottom of the article it outlines how to change the MTU.  I would change to 1300.  If it works, then gradualy increase to a max of 1492.
https://blogs.technet.microsoft.com/askpfeplat/2014/12/01/psa-incorrect-mtu-size-causes-connectivity-issues-with-windows-server-2012-and-windows-server-2012-r2/

Perhaps you have already seen the article.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Internet Protocol Security

From novice to tech pro — start learning today.