Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium


EAP\L2TP VPN cannot connect - Error 792 and 789

Posted on 2004-10-18
Medium Priority
Last Modified: 2011-08-18
I am having problems configuring Windows 2003 server and Windows XP Pro SP2 client, to connect over VPN using EAP and L2TP
They are on the same LAN.

When connecting from client after about a minute I get:

In routing and remote access \server\properties\security\authentication provider:

If I choose windows authentication

I get error message:

Error 792: The L2TP connection attempt failed because security negotiation time out.

If I choose radius authentication

I get this error message:

Error 789: the l2tp connection attempt failed because the security layer encountered a processing error during initial negotiantions with the remote computer.

I have looked at MS knowledge base id:326751

Which suggests but says it not recommended:
1. Add the ProhibitIpSec registry value
2. Manually configure an IPSec policy on both the Windows 2000 Server-based computers
It says applies to Microsoft Windows 2000 Enterprise Edition x2
So, thats not me... right?

Do I really have to do all this or is there a more sutiable solution?

What is best, radius or windows authentication? I think windows is more suited for my needs, but whichever will work essentially.
Question by:Damester
  • 7
  • 6
LVL 23

Expert Comment

by:Tim Holman
ID: 12341195
Does it work if you disable the XP SP2 firewall ?

Author Comment

ID: 12342570
No, I checked that its permanently off.
LVL 23

Expert Comment

by:Tim Holman
ID: 12345474
You can use IPSEC over L2TP -


Otherwise, troubleshooting steps are here:


The timeouts suggest that port 500 is being blocked along the way.
Evaluating UTMs? Here's what you need to know!

Evaluating a UTM appliance and vendor can prove to be an overwhelming exercise.  How can you make sure that you're getting the security that your organization needs without breaking the bank? Check out our UTM Buyer's Guide for more information on what you should be looking for!


Author Comment

ID: 12347626

I know the server is seeing my client as I found this in the event log:

IKE security association establishment failed because peer sent invalid proposal.
Mode: Key Exchange Mode (Main Mode)

Source IP Address
Source IP Address Mask
Destination IP Address
Destination IP Address Mask
Protocol 0
Source Port 0
Destination Port 0
IKE Local Addr
IKE Peer Addr
IKE Source Port 500
IKE Destination Port 500
Peer Private Addr

Phase I Diffie-Hellman Group
 Expected value:
 Received value:

After reading the lower article: http://support.microsoft.com/default.aspx?scid=kb;en-us;314831,
I think IKE is not really what I want and need to change this to PKE. According to the article IKE uses computer certificates but I only installed a user certificate on the client, do you think this the problem.
LVL 23

Expert Comment

by:Tim Holman
ID: 12348153
> IKE security association establishment failed because peer sent invalid proposal.

This error means that you need to ensure both ends are set to use Group 2 DH.  Verify your IKE / Phase 1 parameters match both on the server AND the client.
In VPN terminology, this is a pretty basic issue to sort out - don't give up yet !

Author Comment

ID: 12348181
Okay thanks,

errmm.... how do you do that?

Could you give me a bit of an explanation on where to look for this as I'm not familiar.
LVL 23

Expert Comment

by:Tim Holman
ID: 12369267
There are some how to guides here -


(esp. this one - http://www.pctechnicians.ca/help/VPN3k.html)

It may be worth double checking everything from scratch again, as things are mismatching whereas they shouldn't be unless a config mistake has been made?

Author Comment

ID: 12492643

 The how to guides are not very specific and are outdated. The first link, I could only find one valid article and it only breifley explained how to setup EAP, nothing I did'nt know already.

The second link gave a scenario for setting up VPN when you have two network cards installed.
I do not have this, is it nessecary to have two NIC's for this setup?
I ignored this and followed the setup as closely as possible but the only thing different was setting up a static address pool which did'nt help.

I belive you were correct in your previous statement:

need to ensure both ends are set to use Group 2 DH.  Verify your IKE / Phase 1 parameters match both on the server AND the client.

If this means check the pre shared key is the same on the client IPSEC properties and the RRAS properties, I have and no luck:

I get a slightly different message if I enable radius authentication but still complains of the same thing effectively.

IKE peer authentication failed.
 Peer Identity:
Preshared key ID.
Peer IP Address:
so on and  so on

This confirms I have not got my IKE setup correctly.

I would prefer to not use IKE, so I turned it off and on client when trying to connect I get:

Error 786: The L2TP connection attempt failed because there is no valid machine certificate on your computer for security authentication

Do you know any articles that describe how to create or disrtibute a machine certficate as I can only see user certificate when I use the web enrollment service.

hope you can help, thanks in advance


Author Comment

ID: 12493508
From searching the web for configuring machince certificates for L2TP VPN I have found many articles talking about ISA server.
Do I need to install and configure ISA server for VPN EAP\L2TP client connections, or is this just for addded firewall security?
If so, do I need to install the ISA development kit or can I use the built in ISA on Windows 2003 server?

Does anyone know any good tutorials for setting up machine certificates for VPN EAP\L2TP?
LVL 23

Expert Comment

by:Tim Holman
ID: 12493559
Have you tried with PPTP ?


I'm not sure what the problem is with your L2TP setup, but PPTP is a lot easier to setup and get running, so maybe go for PPTP and then look at L2TP later ?

Author Comment

ID: 12494046
I have setup PPTP with VPN and its in place, elsewhere

I would like to learn and implement L2TP.
LVL 23

Accepted Solution

Tim Holman earned 1500 total points
ID: 12496264
I'm finding it a bit difficult to find resources too, although you could try this GIACs paper for size ?


Don't worry about RADIUS authentication.  The auth failure in question is IKE failure (which is either certificates, pre-shared keys).

Author Comment

ID: 12685683
I've been so busy with other work and I've spent way to much time trying to resolve this problem.

I have decided to go on a VPN course so I understand the problem fully.

Thanks for your time and help Tim you probably gave the correct answer for my problems
 I just dont understand enough about VPN to put your solutions into practice.

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Using Windows 2008 RRAS, I was able to successfully VPN into the network, but I was having problems restricting my test user from accessing certain things on the network.  I used Google in order to try to find out how to stop people from accessing c…
If you’re involved with your company’s wide area network (WAN), you’ve probably heard about SD-WANs. They’re the “boy wonder” of networking, ostensibly allowing companies to replace expensive MPLS lines with low-cost Internet access. But, are they …
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
Windows 10 is mostly good. However the one thing that annoys me is how many clicks you have to do to dial a VPN connection. You have to go to settings from the start menu, (2 clicks), Network and Internet (1 click), Click VPN (another click) then fi…

581 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question