Issuing Machine Certificates to Mac OSX 10.5 for ISA 2006 L2TP VPN

ozofriendly used Ask the Experts™

I am unable to issue a certificate to my Mac client for l2TP VPN access to my ISA Server 2006 VPN endpoint.

Vista and XP clients all work fine, but so far every time I generate an IPSEC offline cert for the Mac using our MS Cert Svcs server, it is deemed a user certificate, so I can't select it as a machine certificate when I configure the L2TP client.

Can anyone tell me how I issue a machine certificate using Microsoft Certificate Services?

Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Distinguished Expert 2017


See if the below helps you along:

An alternative could be to generate a CSR using openssl on the Mac and have it signed by the MS CA.

An example they present is that you would load up the certificate and then export it and the private key to be imported on the Mac.


Hi arnold,

The first link is how we issue machine certs for our XP clients, but does not work from a Mac.

The second presupposes the use of an OpenSSL CA, which we don't have (we use MS CAs). In any case, the instructions are for OSX 10.4. it's all somewhat different with 10.5.

The third link looks like a possible way forward, so I'll do some more digging and see if anyone has had any success creating a CSR this way.
Distinguished Expert 2017

You do not use an OpenSSL ca.  You can use the openssl client on the Mac to generate a CSR request.  you then use the CSR request and have it signed by the MS CA.  You get the signed certificate from the MS CA and add it to the Mac.

Note the common name is the name of your system.

Alternatively, you can import the certificate into an XP.  Then export the certificate with the private key as a pfx file.  There is a way using openssl on the mac to convert the PFX file into a DER format which I think what you need on the Mac.  Note that if you set a password when exporting the PFX, you would either need to strip the password latter on or you will be prompted for the certificate password prior to it being used for the L2TP.

Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!


OK, so following similar guidelines as per this link: 

I was able to generate the CSR using the cert assistant.
Then I set subject alternate name and client authentication EKU attributes for the pending certificate before issuing it.
Once issued and installed in the system keychain along with the keys and CA certs, I was able to select the machine cert for the Mac and initiate the IKE.

Unfortunately, now I have hit another wall. Phase 1 seems to complete, albeit with a couple of suspect entries in the oakley log on the ISA server, but phase 2 fails.

On the Mac I get the message: "IPSec connection failed <IKE Error 18 (0x12) Invalid id information>" in the ppp.log
In the oakley log I get messages such as:

10-27: 14:23:48:801:4b8 Sending: SA = 0x01A8E008 to 2.56265
10-27: 14:23:48:801:4b8 ISAKMP Header: (V1.0), len = 1716
10-27: 14:23:48:801:4b8   I-COOKIE fe1da70dd37e5e90
10-27: 14:23:48:801:4b8   R-COOKIE f1bd65d0ffbdc54d
10-27: 14:23:48:801:4b8   exchange: Oakley Main Mode
10-27: 14:23:48:801:4b8   flags: 1 ( encrypted )
10-27: 14:23:48:801:4b8   next payload: ID
10-27: 14:23:48:801:4b8   message ID: 00000000
10-27: 14:23:48:801:4b8 Ports S:9411 D:c9db
10-27: 14:23:48:801:4b8 Adding Exempt filter: src 145900cb dst c85900cb srcport 4500 dst port 56265
10-27: 14:23:48:832:4b8
10-27: 14:23:48:832:4b8 Receive: (get) SA = 0x01a8e008 from
10-27: 14:23:48:832:4b8 ISAKMP Header: (V1.0), len = 68
10-27: 14:23:48:832:4b8   I-COOKIE fe1da70dd37e5e90
10-27: 14:23:48:832:4b8   R-COOKIE f1bd65d0ffbdc54d
10-27: 14:23:48:832:4b8   exchange: ISAKMP Informational Exchange
10-27: 14:23:48:832:4b8   flags: 1 ( encrypted )
10-27: 14:23:48:832:4b8   next payload: HASH
10-27: 14:23:48:832:4b8   message ID: a1738007
10-27: 14:23:48:832:4b8 invalid payload received
10-27: 14:23:48:832:4b8 GetPacket failed 3613
10-27: 14:23:49:598:250 retransmit: sa = 01A8E008 centry 00000000 , count = 1
 and eventually
10-27: 14:24:19:599:250 Ports S:9411 D:c9db
10-27: 14:24:19:599:24c SocketError 10054
10-27: 14:24:51:600:250 retransmit exhausted: sa = 01A8E008 centry 00000000, count = 6
10-27: 14:24:51:600:250 SA Dead. sa:01A8E008 status:35ed
10-27: 14:24:51:600:250 isadb_set_status sa:01A8E008 centry:00000000 status 35ed

Can anyone shed any light on this "IKE Error 18"? I'm at a bit of a loss.
Distinguished Expert 2017

Usually, you need to open a separate question for a subject that differs.
I.e. this question deals with the Certificate issue.

You should open a separate question for this issue after verifying that the issue on the VPN server is not related to the certificate.
Could you look at the logs on the VPN server side to see what it is showing?  The certificate might be getting rejected. Or there is a mismatch in the information exchange (invalid payload)
It fails because it kept retransmitting the request without a response.

My success in issuing a machine certificate means that this question can be closed.

Using the MS Cert Server command line certainly enabled OSX to recognise the issued certificate as a 'machine' certificate, but I suspect that's only half the story. Something is missing in this process which results in the failure of the IKE negotiation.

I'll open another question when I can spend some more time looking at this problem in more depth.
Distinguished Expert 2017

The issuance of the certificate is the premise of your question.
When configuring the L2TP connection are you able to select the Machine Certificate?  What is the result on the remote side when the connection is attempted?
When you imported the machine certificate did you export it as a DER that includes the certificate and the private key?

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial