EAP\L2TP VPN cannot connect - Error 792 and 789

Posted on 2004-10-18
Last Modified: 2011-08-18
I am having problems configuring Windows 2003 server and Windows XP Pro SP2 client, to connect over VPN using EAP and L2TP
They are on the same LAN.

When connecting from client after about a minute I get:

In routing and remote access \server\properties\security\authentication provider:

If I choose windows authentication

I get error message:

Error 792: The L2TP connection attempt failed because security negotiation time out.

If I choose radius authentication

I get this error message:

Error 789: the l2tp connection attempt failed because the security layer encountered a processing error during initial negotiantions with the remote computer.

I have looked at MS knowledge base id:326751;en-us;326751&Product=win2000

Which suggests but says it not recommended:
1. Add the ProhibitIpSec registry value
2. Manually configure an IPSec policy on both the Windows 2000 Server-based computers
It says applies to Microsoft Windows 2000 Enterprise Edition x2
So, thats not me... right?

Do I really have to do all this or is there a more sutiable solution?

What is best, radius or windows authentication? I think windows is more suited for my needs, but whichever will work essentially.
Question by:Damester
  • 7
  • 6
LVL 23

Expert Comment

by:Tim Holman
ID: 12341195
Does it work if you disable the XP SP2 firewall ?

Author Comment

ID: 12342570
No, I checked that its permanently off.
LVL 23

Expert Comment

by:Tim Holman
ID: 12345474
You can use IPSEC over L2TP -;en-us;265112

Otherwise, troubleshooting steps are here:;en-us;314831

The timeouts suggest that port 500 is being blocked along the way.
Connect further...control easier

With the ATEN CE624, you can now enjoy a high-quality visual experience powered by HDBaseT technology and the convenience of a single Cat6 cable to transmit uncompressed video with zero latency and multi-streaming for dual-view applications where remote access is required.


Author Comment

ID: 12347626

I know the server is seeing my client as I found this in the event log:

IKE security association establishment failed because peer sent invalid proposal.
Mode: Key Exchange Mode (Main Mode)

Source IP Address
Source IP Address Mask
Destination IP Address
Destination IP Address Mask
Protocol 0
Source Port 0
Destination Port 0
IKE Local Addr
IKE Peer Addr
IKE Source Port 500
IKE Destination Port 500
Peer Private Addr

Phase I Diffie-Hellman Group
 Expected value:
 Received value:

After reading the lower article:;en-us;314831,
I think IKE is not really what I want and need to change this to PKE. According to the article IKE uses computer certificates but I only installed a user certificate on the client, do you think this the problem.
LVL 23

Expert Comment

by:Tim Holman
ID: 12348153
> IKE security association establishment failed because peer sent invalid proposal.

This error means that you need to ensure both ends are set to use Group 2 DH.  Verify your IKE / Phase 1 parameters match both on the server AND the client.
In VPN terminology, this is a pretty basic issue to sort out - don't give up yet !

Author Comment

ID: 12348181
Okay thanks,

errmm.... how do you do that?

Could you give me a bit of an explanation on where to look for this as I'm not familiar.
LVL 23

Expert Comment

by:Tim Holman
ID: 12369267
There are some how to guides here -

(esp. this one -

It may be worth double checking everything from scratch again, as things are mismatching whereas they shouldn't be unless a config mistake has been made?

Author Comment

ID: 12492643

 The how to guides are not very specific and are outdated. The first link, I could only find one valid article and it only breifley explained how to setup EAP, nothing I did'nt know already.

The second link gave a scenario for setting up VPN when you have two network cards installed.
I do not have this, is it nessecary to have two NIC's for this setup?
I ignored this and followed the setup as closely as possible but the only thing different was setting up a static address pool which did'nt help.

I belive you were correct in your previous statement:

need to ensure both ends are set to use Group 2 DH.  Verify your IKE / Phase 1 parameters match both on the server AND the client.

If this means check the pre shared key is the same on the client IPSEC properties and the RRAS properties, I have and no luck:

I get a slightly different message if I enable radius authentication but still complains of the same thing effectively.

IKE peer authentication failed.
 Peer Identity:
Preshared key ID.
Peer IP Address:
so on and  so on

This confirms I have not got my IKE setup correctly.

I would prefer to not use IKE, so I turned it off and on client when trying to connect I get:

Error 786: The L2TP connection attempt failed because there is no valid machine certificate on your computer for security authentication

Do you know any articles that describe how to create or disrtibute a machine certficate as I can only see user certificate when I use the web enrollment service.

hope you can help, thanks in advance


Author Comment

ID: 12493508
From searching the web for configuring machince certificates for L2TP VPN I have found many articles talking about ISA server.
Do I need to install and configure ISA server for VPN EAP\L2TP client connections, or is this just for addded firewall security?
If so, do I need to install the ISA development kit or can I use the built in ISA on Windows 2003 server?

Does anyone know any good tutorials for setting up machine certificates for VPN EAP\L2TP?
LVL 23

Expert Comment

by:Tim Holman
ID: 12493559
Have you tried with PPTP ?

I'm not sure what the problem is with your L2TP setup, but PPTP is a lot easier to setup and get running, so maybe go for PPTP and then look at L2TP later ?

Author Comment

ID: 12494046
I have setup PPTP with VPN and its in place, elsewhere

I would like to learn and implement L2TP.
LVL 23

Accepted Solution

Tim Holman earned 500 total points
ID: 12496264
I'm finding it a bit difficult to find resources too, although you could try this GIACs paper for size ?

Don't worry about RADIUS authentication.  The auth failure in question is IKE failure (which is either certificates, pre-shared keys).

Author Comment

ID: 12685683
I've been so busy with other work and I've spent way to much time trying to resolve this problem.

I have decided to go on a VPN course so I understand the problem fully.

Thanks for your time and help Tim you probably gave the correct answer for my problems
 I just dont understand enough about VPN to put your solutions into practice.

Featured Post

Netscaler Common Configuration How To guides

If you use NetScaler you will want to see these guides. The NetScaler How To Guides show administrators how to get NetScaler up and configured by providing instructions for common scenarios and some not so common ones.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
azure vpn connection 3 66
Windwos Server 2012 R2 NPS for Cisco ASA VPN Authentication 2 70
Allowing Multicast in the firewall 2 53
Anyconnect landing page login failed 2 27
Preface Having the need * to contact many different companies with different infrastructures * do remote maintenance in their network required us to implement a more flexible routing solution. As RAS, PPTP, L2TP and VPN Client connections are no…
One of the Top 10  common Cisco VPN problems are not-matching shared keys. This is an easy one to fix, but not always easy to notice, see the case below. A simple IPsec tunnel between fast Ethernet interfaces of routers SW1 (f1/1) and R1(f0/0). …
After creating this article (, I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
Windows 10 is mostly good. However the one thing that annoys me is how many clicks you have to do to dial a VPN connection. You have to go to settings from the start menu, (2 clicks), Network and Internet (1 click), Click VPN (another click) then fi…

856 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question