L2TP VPN - error 810 with non-domain clients

Hi ,

I have successfully setup a L2TP VPN server with MS TMG 2010. The clients that are part of the domain can successfully establish the connection.

I have my own CA that I use to generate the certificates.

When I generate the exact same certificate with the same CA for a client that is not part of any domain then I get the 810 error.

Specifically, the client I am testing is on Windows 7. It belongs to a WORKGROUP (Windows default) and it is freshly installed with nothing else on it. It is called "burgvpc-PC".
I have test with another non-domain client - same problem.

I made sure that the CA certificate is in the clients stores (in Trusted Root and Intermediate Certification Authorities - for both use and computer stores).

I also have the client certificate in the computer store.

In the VPN trace logs on the client, the only relevant information I get is "RASDiag: Mapping to new errorcode: 810 (ERROR_VPN_BAD_CERT) instead of 786 (ERROR_OKLEY_NO_CERT)"
There is not information why Windows considers it a bad certificate.

I have also tried the subject name for the certificate to be both "burgvpc-PC" and "burgvpc-PC.WORKGROUP" - both give the same error.

Does anyone have any idea how the certificate should be different when the client is not part of the domain?
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

If it is a windows 7 client, you will have three event log entries, one in application, one in system and one in security. The security entry for category  IPSec-xxx should contain information, why the connection fails and on what stage.

Keep in mind thyt L2TP uses a certificate, which comes from the TMG, and the user has a certificate or uses UserID and password, dependend from the settings.

Inside the VPN settings, there is an option "Include windows logon domain" as well as the setting for MSChap2 to automatically use the logon information if available.

You have to seperate the setting:
L2TP (uses TMG cert)
Cert base client authentication (uses user cert)
Enforced cleint certificate (offered by client, bui set and enforced by TMG)

If NAP is active on TMG, also check the NAP settings for limitations.
salt-eitAuthor Commented:

I do not have anything in the system log, just the application and security log (check below).

The computers that are members of the domain can connect successfully, so that should rule out any issues with the TMG Server L2TP certificate.

The "Include windows logon domain" is turned off and also the "automatically use the logon information".

The NAP is not limiting anything at this stage or giving any errors.

Is there any option other than VPN tracing options that can give me more detail on the L2TP negotiations?

Log Name:      Security
Source:        Microsoft-Windows-Security-Auditing
Date:          8/2/2012 1:39:20 PM
Event ID:      5061
Task Category: System Integrity
Level:         Information
Keywords:      Audit Failure
User:          N/A
Computer:      burgvpc-PC
Cryptographic operation.

      Security ID:            SYSTEM
      Account Name:            BURGVPC-PC$
      Account Domain:            WORKGROUP
      Logon ID:            0x3e7

Cryptographic Parameters:
      Provider Name:      Microsoft Software Key Storage Provider
      Algorithm Name:      Not Available.
      Key Name:      le-IPSec!0028Offlinerequest!0029-Client-NFSICustom01-560f5200-defb-4af8-b90c-8f412e103469
      Key Type:      User key.

Cryptographic Operation:
      Operation:      Open Key.
      Return Code:      0x80090016

Log Name:      Application
Source:        RasClient
Date:          8/2/2012 1:36:39 PM
Event ID:      20227
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      burgvpc-PC
CoId={02AF5DD7-008A-46B4-AD18-8B3954C0583B}: The user burgvpc-PC\burgvpc dialed a connection named NFSI VPN IPSec which has failed. The error code returned on failure is 810.

Log Name:      Application
Source:        RasClient
Date:          8/2/2012 1:36:38 PM
Event ID:      20221
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      burgvpc-PC
CoId={02AF5DD7-008A-46B4-AD18-8B3954C0583B}: The user burgvpc-PC\burgvpc has started dialing a VPN connection using a per-user connection profile named NFSI VPN IPSec. The connection settings are:
Dial-in User = vpntest
VpnStrategy = L2TP
DataEncryption = Require maximum
PrerequisiteEntry =
AutoLogon = No
UseRasCredentials = Yes
Authentication Type = CHAP/MS-CHAPv2
Ipv4DefaultGateway = Yes
Ipv4AddressAssignment = By Server
Ipv4DNSServerAssignment = By Server
Ipv6DefaultGateway = Yes
Ipv6AddressAssignment = By Server
Ipv6DNSServerAssignment = By Server
IpDnsFlags =
IpNBTEnabled = Yes
UseFlags = Private Connection
ConnectOnWinlogon = No
IPsec authentication for L2TP = Machine certificate.
Yes sure there are,

From the TMG site, you have some possiblities to check, what is going on. First some additional steps:
1.) First make sure, the TMG is up to date SP2, RU2 is the last one, which is build 7.0.9193.540

2.) Check your certificates on the TMG, the TMG does not take necessarily the cert you assign, it sometimes take the first one who fits. Just had the case with a client, used PSK and cert based L2TP and regognized, the TMG sends not the certificate I configured in NAP

I had two certs, one with serverauthentication one with IPSec, assigned the server authentication and TMG send the IPSec, which was nowhere assigned. But you should be able to identifiy the cert thumbprint on the client.

According to this, also make sure, what the client has as certificates. To much can disturb the conection in the same way.

3.) Check if there are differences between domain members and not domain members, i.e policies, which influences the local windows firewall. For testing, disable all fireus and antivirus products.

4.) The client:
If I use L2TP / IPSec, I get for PSK as well as cert based authentication two security events for
IPSec fast modus and IPSec main modus.
The last one shows the authentification of the machines, so either PSK or cert, and if with cert, it shows me the thrumprint of the TMG cert, the cleint gets.

Be patient, the log has some delay....,

On the TMG site, there are several options....
1.) Event log
2.) NAP Accountig log (you have to enable the log in NAP (i.e. file based - IAS Format)
3.) The protocol log in TMG (Log & Reports)
4.) The Diagnostic Logging in TMG (Troubleshooting)
5.) The TMG Data Packager (TMG BPA unter Tools)

The last one installs also MS Network Monitor, the is hardcore but logs everything

But I guess the problem seems to be on the client as P2TP seems to fail.
I just test, which logs you can see in different combinations.
So come back in a moment
Price Your IT Services for Profit

Managed service contracts are great - when they're making you money. Yes, you’re getting paid monthly, but is it actually profitable? Learn to calculate your hourly overhead burden so you can master your IT services pricing strategy.

And another thing to check...
The user you use has dial in rihgt via AD, right?
OK, to bring some light in the darkness....

TMG has the settings to enable / disable PPTP, L2TP/IPSec and SSTP
Also you can create on the Remote Access Policy the allowed Authentication Policies for
MSChap-v2 and EAP as well as a PresharedKey

If EAP is enabled, then you have to make additional settings to the NAP configuration.
There you find Policies - Network Polices and Connection Request Policies.

In the Network Policy, you have a Forefront TMG Default Policy.  
On the Contraints tab, you find the setting MS-CAHP-v2 as well as EAP settings where you can define up to three different EAP modes
- Smartcard

The first two have a certificate, ist has to be valid, but cannot see it on the client.

Now the methods are defined, which are allowed !!! Thats all
But If you think, that this forces the client to use cert based authentication, then you make the deal without MS logic....

To get certbased authentication to run, you need a certificate, which is assigned nowhere, it just has to reside in the computers cert store.

I created my own template for this on my CA to make sure it will renew automatically.
The cert is a copy of the Webserver template, I added beside the existing Server Authentication the "IP security, IKE intermediate" setting for the key usage and also changed the template to ask for the names during the request. Then I requested that cert on the TMG and put the external name as subject, and the internal + external name as Alternative Subject Name.

On the client, I created several VPN setting in all combinations. For the computer PSK and cert based, for the user MS-Chap-v2, EAP-MS-Chap-v2, PEAP and cert based, just to see, what happens on the client.

With the certificate, all connections went fine, PSK used PSK and cert base used the cert. On the client I could see, that the computer cert is my special IPSEC cert, and the user cert is my user cert.

When I deleted my IPSec cert on TMG, all cert based connection where denied, but after rebooting the TMG, I could logon again, but in this case, all connection felt back to PSK, no certs are used at all. So readded the Cert with no change, restarted the firewall services and the certs are working again on the client.

Means in fact, to get cert based authentication work, you need the described certificate on the TMG and better to make a reboot.

Last try was to delete the computer cert from the client. Nevertheless it is not seen anywhere, it brakes the computer authentication.

The information about the certificate are logges on both system in the securtity log with the task category IPSec Main Mode.
salt-eitAuthor Commented:
Hi Bembi,

Thanks for all you comments.

Note that I am not using EAP certificate authentication. It is username/password auth with MS-CHAPv2. But IPSec still uses a machine certificate on both the server and the client because I am not using PSK. As I have already mentioned, the server setup works because my computers that are members of the domain connect just fine.

The problem is on the client computer when the client is NOT a domain member.

Normally in the certificate common name of the client you would specify the FQDN of the client such as pc1.domain.local

But the computers with the problem are non domain members. They belong to a workgroup so they have no DNS name.
So subject I use is simply the computer name, although I have tried different options for the subject name. Please check my original post.

The below security event on the TMG server confirms that the problem is the client and that its certificate is not valid.

So do you know what should be different on the certificate when a computer does not belong to the domain?

Log Name:      Security
Source:        Microsoft-Windows-Security-Auditing
Date:          8/5/2012 4:08:47 PM
Event ID:      4652
Task Category: IPsec Main Mode
Level:         Information
Keywords:      Audit Failure
User:          N/A
Computer:      NFSIFWL000002.natfsi.local
An IPsec Main Mode negotiation failed.

Local Endpoint:
      Principal Name:            -
      Network Address:      
      Keying Module Port:      500

Local Certificate:
      SHA Thumbprint:      -
      Issuing CA:            -
      Root CA:            -

Remote Endpoint:
      Principal Name:            -
      Network Address:      
      Keying Module Port:      60434

Remote Certificate:
      SHA thumbprint:            -
      Issuing CA:            -
      Root CA:            -

Additional Information:
      Keying Module Name:      IKE
      Authentication Method:      Certificate
      Role:                  Responder
      Impersonation State:      Not enabled
      Main Mode Filter ID:      77447

Failure Information:
      Failure Point:            Remote computer
      Failure Reason:            IKE failed to find valid machine certificate. Contact your Network Security Administrator about installing a valid certificate in the appropriate Certificate Store.

      State:                  Sent second (KE) payload
      Initiator Cookie:            ba4ca9b34ea468b2
      Responder Cookie:      aad530cac5e26acb
OK, just something to read.... :-)

Didn't really found the solution, but something what you can try:
If you Goto RRAS properties - > Security, you can click on authentication methods, where you find EAP MS-CHAP-v2 and also "Allow machine certificate authentication for IKEv2.

IKEv2 works on the client, if any valid cert is installed and can be resolved. But IPSec works only, if the client cert and the server cert are from the same CA.

The computer certificate has to include the name, which the client presents to the server as well as the server certificate has to bé the name which is presented to the client.

There should be only one machine certificate on the client. If you have more than one, the first fitting is used. Also, if the client (or server) cert is a SAN cert, the order in the subject alternative name may influence, what is checked.

As stated in the article, the client can have a lot of different certificate usage types, while the server needs "IP security IKE intermediate" and Server Authentication in the same cert.

As Domain members connect, but domain foreign computers not, it can only be a client issue in my mind. You may try to set the workgroup name of the client to the same NETBIOS name like the domain. If the client belongs to a different Domain, IPSec will not work whenever a forein cert is cert for the other domain.

What you see in your log is, that the client fails on the IKE level, the machine handshake, and therefore no user authentication is sent. The client presents his cert, and it is denied at once, so no more certs are send back from the server.  

Don't rely to all the comment in the article, because it is tested with Win7 beta, nevertheless a lot of things to verify and a in deep test.

Hope this may give the right hint.
salt-eitAuthor Commented:
Hi Bembi,

Thanks for all your effort.

I got it solved now, although I cannot say exactly what the issue was.

The link you posted did not really solve this issue, but I am sure it will help a whole lot of other people.

What I did was to remove all the CA certificates of my CA in all stores and also all the client computer certificates.

I then imported the client certificate again in the MMC console by right-clicking on the Personal Certificate store of the Computer store and telling it to store in that store where I had right-clicked.
The CA certificate I imported in the same way, but I chose the  Trusted Root Certification Authority of the computer store.

This time it worked.

I don't know the exact problem, because previously I had the exact same certificates in the exact same places.
Although previously when I imported certificates, I had imported the CA and client certificate in my User stores and then I just copy/paste from there into the computer store - don't know if that caused an issue?
I did it this way because when you double-click a PFX file to import and you use the defaults then it imports the certificate to the User store and not the computer store.

So for the record, the details on my client cert:

Subject Common Name:   burgvpc-PC
Public Key: RSA 2048 Bits
Signature Algorithm: sha1RSA
Signature hash algorithm: sha1
Enhanced Key Usage (EKU): Server Authentication, Client Authentication
Key Usage: Digital Signature, Key Encipherment
Thumbprint algorithm: sha1

Note that I did not need to add WORKGROUP or anything like that to the subject - just the computer name was fine.
Also, my CA's CRL URL is not accessible from the client and for L2TP does not need to be - only for SSTP it does.

So it seems to have been an issue with the CA certificate imports.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
salt-eitAuthor Commented:
Resolved it myself, Bembi's comments helped in the troubleshooting to verify all is in place but ultimately it was not related to the issue.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft Forefront ISA Server

From novice to tech pro — start learning today.