EAP\L2TP VPN cannot connect - Error 792 and 789

I am having problems configuring Windows 2003 server and Windows XP Pro SP2 client, to connect over VPN using EAP and L2TP
They are on the same LAN.

When connecting from client after about a minute I get:

In routing and remote access \server\properties\security\authentication provider:

If I choose windows authentication

I get error message:

Error 792: The L2TP connection attempt failed because security negotiation time out.

If I choose radius authentication

I get this error message:

Error 789: the l2tp connection attempt failed because the security layer encountered a processing error during initial negotiantions with the remote computer.

I have looked at MS knowledge base id:326751

Which suggests but says it not recommended:
1. Add the ProhibitIpSec registry value
2. Manually configure an IPSec policy on both the Windows 2000 Server-based computers
It says applies to Microsoft Windows 2000 Enterprise Edition x2
So, thats not me... right?

Do I really have to do all this or is there a more sutiable solution?

What is best, radius or windows authentication? I think windows is more suited for my needs, but whichever will work essentially.
Who is Participating?
Tim HolmanConnect With a Mentor Commented:
I'm finding it a bit difficult to find resources too, although you could try this GIACs paper for size ?


Don't worry about RADIUS authentication.  The auth failure in question is IKE failure (which is either certificates, pre-shared keys).
Tim HolmanCommented:
Does it work if you disable the XP SP2 firewall ?
DamesterAuthor Commented:
No, I checked that its permanently off.
Improved Protection from Phishing Attacks

WatchGuard DNSWatch reduces malware infections by detecting and blocking malicious DNS requests, improving your ability to protect employees from phishing attacks. Learn more about our newest service included in Total Security Suite today!

Tim HolmanCommented:
You can use IPSEC over L2TP -


Otherwise, troubleshooting steps are here:


The timeouts suggest that port 500 is being blocked along the way.
DamesterAuthor Commented:

I know the server is seeing my client as I found this in the event log:

IKE security association establishment failed because peer sent invalid proposal.
Mode: Key Exchange Mode (Main Mode)

Source IP Address
Source IP Address Mask
Destination IP Address
Destination IP Address Mask
Protocol 0
Source Port 0
Destination Port 0
IKE Local Addr
IKE Peer Addr
IKE Source Port 500
IKE Destination Port 500
Peer Private Addr

Phase I Diffie-Hellman Group
 Expected value:
 Received value:

After reading the lower article: http://support.microsoft.com/default.aspx?scid=kb;en-us;314831,
I think IKE is not really what I want and need to change this to PKE. According to the article IKE uses computer certificates but I only installed a user certificate on the client, do you think this the problem.
Tim HolmanCommented:
> IKE security association establishment failed because peer sent invalid proposal.

This error means that you need to ensure both ends are set to use Group 2 DH.  Verify your IKE / Phase 1 parameters match both on the server AND the client.
In VPN terminology, this is a pretty basic issue to sort out - don't give up yet !
DamesterAuthor Commented:
Okay thanks,

errmm.... how do you do that?

Could you give me a bit of an explanation on where to look for this as I'm not familiar.
Tim HolmanCommented:
There are some how to guides here -


(esp. this one - http://www.pctechnicians.ca/help/VPN3k.html)

It may be worth double checking everything from scratch again, as things are mismatching whereas they shouldn't be unless a config mistake has been made?
DamesterAuthor Commented:

 The how to guides are not very specific and are outdated. The first link, I could only find one valid article and it only breifley explained how to setup EAP, nothing I did'nt know already.

The second link gave a scenario for setting up VPN when you have two network cards installed.
I do not have this, is it nessecary to have two NIC's for this setup?
I ignored this and followed the setup as closely as possible but the only thing different was setting up a static address pool which did'nt help.

I belive you were correct in your previous statement:

need to ensure both ends are set to use Group 2 DH.  Verify your IKE / Phase 1 parameters match both on the server AND the client.

If this means check the pre shared key is the same on the client IPSEC properties and the RRAS properties, I have and no luck:

I get a slightly different message if I enable radius authentication but still complains of the same thing effectively.

IKE peer authentication failed.
 Peer Identity:
Preshared key ID.
Peer IP Address:
so on and  so on

This confirms I have not got my IKE setup correctly.

I would prefer to not use IKE, so I turned it off and on client when trying to connect I get:

Error 786: The L2TP connection attempt failed because there is no valid machine certificate on your computer for security authentication

Do you know any articles that describe how to create or disrtibute a machine certficate as I can only see user certificate when I use the web enrollment service.

hope you can help, thanks in advance

DamesterAuthor Commented:
From searching the web for configuring machince certificates for L2TP VPN I have found many articles talking about ISA server.
Do I need to install and configure ISA server for VPN EAP\L2TP client connections, or is this just for addded firewall security?
If so, do I need to install the ISA development kit or can I use the built in ISA on Windows 2003 server?

Does anyone know any good tutorials for setting up machine certificates for VPN EAP\L2TP?
Tim HolmanCommented:
Have you tried with PPTP ?


I'm not sure what the problem is with your L2TP setup, but PPTP is a lot easier to setup and get running, so maybe go for PPTP and then look at L2TP later ?
DamesterAuthor Commented:
I have setup PPTP with VPN and its in place, elsewhere

I would like to learn and implement L2TP.
DamesterAuthor Commented:
I've been so busy with other work and I've spent way to much time trying to resolve this problem.

I have decided to go on a VPN course so I understand the problem fully.

Thanks for your time and help Tim you probably gave the correct answer for my problems
 I just dont understand enough about VPN to put your solutions into practice.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.