Windows 2003 Server to Server L2TP VPN troubleshooting and correct setup

Posted on 2006-05-23
Medium Priority
Last Modified: 2010-08-05
I have been unsuccessful in setting up a L2TP VPN between two windows 2003 standard servers.  I did make sure that the proper ports are open and such and was also able to generically test the connection from my PC with a PPTP VPN connection.  The PPTP worked w/o problems.

The issue that I have run into is that the server is not responding to the remote server or client workstation when I switch to L2TP.  And the problem with trying to fix this is that I have read so many different docs on setting up L2TP VPNs that I have been confused to the point that I am mixing up methods and can't determine the proper configuration.

For starters, these are NOT active directory computers which makes things instantly more difficult.

Assume that Routing and Remote access is installed correctly and let's focus on just the security issues involved with L2TP.  The same SSL cert is installed on both machines and is from a trusted root provider.  This eliminates the setup of my own CAs from what I have read.  In the RRAS properties I have allowed EAP and MS-CHAP v2.  Both the Authentication and Accounting provider is Windows.

The RAS policy is setup for Encryption to allow any method and the Authentication is setup to allow EAP with PEAP configured to use the SSL cert installed and MS-CHAP v2 is also allowed.

The VPN adapter has been setup to use optional encryption and smart card/or/other cert with the use simple cert selection box checked.

The Dial -In Account was created and setup correctly.

The Error: A Demand Dial connection to the remote interface VPN_NY_U15197371 on port VPN3-241 was successfully initiated but failed to complete successfully because of the  following error: The remote computer did not respond. For further assistance, click More Info or search Help and Support Center for this error number.

The Error Number: 20111

Thanks in advance for your help!

Question by:jdraggi
  • 7
  • 5
LVL 78

Accepted Solution

Rob Williams earned 2000 total points
ID: 16747976
You are a brave man John  :-)
I haven't done this as it can be quite difficult as you have found, especially without Active Directory accounts, I would assume. Since you are not getting a response from the remote site, I thought I would verify you have all the right ports open/forwarded:
L2TP over IPSec
  To allow IKE forward UDP port 500.
  To allow IPSec NAT-T forward port UDP 4500.
  To allow L2TP forward port UDP 1701.
  Enable IPSec protocols 50 ESP & 51 AH  pass-through (not ports 50 & 51).  May be called VPN or L2TP and IPSec pass-through on most routers)

Have you considered a site to site hardware VPN solution? If you install 2 VPN routers it offloads the service to the routers, and increases security and performance. Now-a-days you can do so for less than $200 US per site with a very nice Linksys RV042 or a Cisco PIX 501 for about double that. They are very stable and much easier to configure.
Just a thought,

Author Comment

ID: 16748114
Yeah, this is turning out to be the hardest network configuration that I have ever done.

You got one... forgot to update the VPN CLNT IP to the remote SERVER IP

The external router is open on ports >1024 and port 500 is cool...  BTW:  Protocol control is something that I don't have at the one data center which is why I had to move from PPTP to L2TP because GRE was not supported...  :(

my IPSEC policy now allows
PPTP TCP any_source to 1723 coming from any_IP to my IP
L2TP UDP any_source to 1701 coming from any_IP to my IP
ISAKMP UDP any_source to 500 coming from any_IP to my IP
VPN CLNT ANY_protocol any_source to any destination coming from AUTH_IP to My_IP

all of which are mirrored... so we're set... new error...

The new error is... the security policy for the connection was not found so I'm checking into that now...


LVL 78

Expert Comment

by:Rob Williams
ID: 16748164
>>"Protocol control is something that I don't have at the one data center which is why I had to move from PPTP to L2TP"
Doesn't support PPTP but does support L2TP and IPSec? odd.

>>"The new error is... the security policy for the connection was not found so I'm checking into that now"
That actually sounds like progress. At least there is some handshaking going on.

A Cyber Security RX to Protect Your Organization

Join us on December 13th for a webinar to learn how medical providers can defend against malware with a cyber security "Rx" that supports a healthy technology adoption plan for every healthcare organization.


Author Comment

ID: 16748190
Yes, it's weird...  imediate error that GRE protocol not supported.  I think that if I turned off their firewall I would be fine but I just like the extra layer.  The other datacenter supports PPTP w/o problems hence the testing with PPTP to confirm that something was working.

Yes, security log is showing talking however it looks like the Peer is using the wrong SSL cert...  Which I have a few installed on that server...  Trying to determine how to modify this choice now...

I think that I am going to delete the 2 default RAS policy objects and start a new one.



Author Comment

ID: 16748260
The Main Mode IKE established then

The Quick Mode fails...  The SSL certs don't seem to apply at this point because they are doing their job...

Here is the current error:

IKE security association negotiation failed.
Data Protection Mode (Quick Mode)

Source IP Address SERVER_IP
Source IP Address Mask
Destination IP Address SERVER_IP2
Destination IP Address Mask
Protocol 17
Source Port 1701
Destination Port 1701
IKE Source Port 500
IKE Destination Port 500
Peer Private Addr

 Peer Identity:
Certificate based Identity.

  Failure Point:

 Failure Reason:
No policy configured

 Extra Status:
Processed third (ID) payload
Initiator.  Delta Time 1
 0x0 0x0
LVL 78

Expert Comment

by:Rob Williams
ID: 16748282
I read more than 200 pages on this, over a 2 month period last year, while in a waiting room. When done I said that was a waste of time, I'll never try that. :-)  So I don't envy you.
Definitely is policy or certificate related. Though you haven't gone far enough to tell, once connected you are aware the subnets at the 2 sites need to be different ??

Author Comment

ID: 16748336
yeah, different subnets won't be a problem...  

The biggest problem is that the errors that I am getting are so freaking general that they are almost worthless.  So is the problem really on the peer side or on the client side...?

I've been working on this for a few days now...  I think I just need to walk away for a bit and come back.


Author Comment

ID: 16752131
ok, new weird problem...

I got the L2TP connection working from any client to Server 2

>>> but <<<

I can't get any client to connect to connect to Server 1

>>> Odd item <<<

Server 1 can connect to Server 2 w/o problems.  Settings are identical including user accounts on both systems.

Any ideas?
LVL 78

Expert Comment

by:Rob Williams
ID: 16754082
Earlier you mentioned;  "The other datacenter supports PPTP w/o problems hence the testing with PPTP to confirm that something was working."
Is there a difference in the router/firewalls such that some protocols may being blocked at one site? What make and models are the routers?
Also there were some changes to IPSec with NAT-T with XP and server 2003. Are both servers 2003 ? or one 2000?

Author Comment

ID: 16754389
Well...  I am not sure what the models are just that it's cisco xyz equipment.  My guess on the PPTP issue is that they have that protocol actually turned off because the error was immediate and stated that the protocol was not supported.

Nevertheless... The configuration of the servers is identical, both windows 2003, etc...

The problem with the sever/center that I am having problems with was the IPsec rules...  It did not have port 4500 configured...

Currently, the only difference is that the server that is working fine does NOT have IPsec rules setup and this other server does...  I have been trying to figure out what needs to be in the IPsec rules besides the items that you have listed.  At this point it seems like I have some dynamic security rule not being applied. (hence no security response)  Unfortunately I don't have a working IPsec policy that supports L2TP to compare against.  :(

OH... also disabled the firewall all together at the datacenter w/problems to try and narrow down problems.


Author Comment

ID: 16755673
Rob, turns out that the IPsec policy was off... in my haste/tiredness I found that I was allowing incomming communications to UDP 4500 instead of allowing outgoing UDP traffice 4500 to any port.

Thanks so much for your help and support!

Now that I have the lingo down L2TP is pretty easy the problem was just too many variables to account for at the same time.


Link on MS website:

In summary:
L2TP over IPSec
  To allow IKE forward UDP port 500 incoming to server from any port
  To allow IPSec NAT-T forward port UDP 4500 outgoing to clients on any port
  To allow L2TP forward port UDP 1701 incoming to server from any port
LVL 78

Expert Comment

by:Rob Williams
ID: 16755850
John, thanks for the update/summary. That is a good MS article, I haven't seen that one. Glad to hear you were able to get it working, there are a lot of variables, with certificates, ports, policies and filters. I have ended up going with VPN routers at all sites, except for temporary PPTP's for short term access. They are easier to configure and offload one service from the server. I must give this a go at least on a couple of Virtual Machines just to get more familiar with it.

Thanks for the points, though I don't think I helped at all except for moral support.
Cheers !

Featured Post

What Security Threats Are We Predicting for 2018?

Cryptocurrency, IoT botnets, MFA, and more! Hackers are already planning their next big attacks for 2018. Learn what you might face, and how to defend against it with our 2018 security predictions.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This is an article about my experiences with remote access to my clients (so that I may serve them) and eventually to my home office system via Radmin Remote Control. I have been using remote access for over 10 years and have been improving my metho…
Will you be ready when the clock on GDPR compliance runs out? Is GDPR even something you need to worry about? Find out more about the upcoming regulation changes and download our comprehensive GDPR checklist today !
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
Windows 10 is mostly good. However the one thing that annoys me is how many clicks you have to do to dial a VPN connection. You have to go to settings from the start menu, (2 clicks), Network and Internet (1 click), Click VPN (another click) then fi…
Suggested Courses
Course of the Month13 days, 10 hours left to enroll

750 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question