Has anyone successfully setup IPsec between Solaris 8 and Windows 2000 Server

Posted on 2003-11-12
Last Modified: 2013-12-05
Has anyone successfully setup IPsec connections between Solaris 8 and Windows 2000 server, out there? I know automatic IKE was not added to Solaris until 9 with the addition of the in.iked daemon. IKE is required by Windows 2000 for IPsec. So has anyone managed in getting such a connection working or equally has anyone tried and found that it just won't work withour Sol9?

I'm up against a wall on this one and the clocks ticking so I need a definitive answer either way asap. An before anyone asks I can't 'just try' it as there's not enough time and the unix and windows teams are seperate.

I just need to either here from someone who has achieved this or someone that knows that it fundamentaly won't work before I commit.

Question by:doofry
  • 5
  • 3
LVL 38

Accepted Solution

yuzh earned 500 total points
ID: 9736843
Solaris 8 IPSec supports AH (authentication) and ESP (encryption) headers, and "shared secrets" (manual keying), but not automatic (ISAKMP or IKE) keying. Solaris 9 supports IKE.

Information about how to setup IPsec for Solaris 8 can be found:

Full doc:

Also Have a look at the following docs:
LVL 38

Expert Comment

ID: 9737451
Document ID   Synopsis   Date
ID47820   SunScreen[TM] IKE Interoperability with Microsoft (R) Windows 2000 and XP   21 Oct 2002



Description Top

Document Body Top

Disclaimer: This document is not intended to be a substitute for the SunScreen 3.2 documentation or Microsoft documentation, but rather as an abbreviated overview on setting up IKE between these two products. There are many variations not shown here that should also work. The procedures below will give the reader a basic background for deployment. Discussion of the IKE protocol and the meaning of the basic IKE parameters will not be covered. The examples will assume Windows 2000 system, but the Windows XP interface should be nearly identical.

Basic Interoperability

Using mmc to set up the Windows 2000 system for IKE support

Simple example: Connection from Windows system to SunScreen, using pre-shared keys in Transport mode

Obtaining a CA-signed certificate with Windows 2000

Example 1: Windows Certification Authority

Example 2: Using a Windows CA server to generate a CA request to a non-Windows CA

Example 3: Generating a CSR using OpenSSL

Obtaining a CA-signed certificate with SunScreen

Example: Creating a IKE VPN punch-in for Windows systems with arbitrary IP addresses

Example: Using Windows as a Remote Administration Station

Note on "SUBJECT=" vs. "ISSUER="

Command Line considerations

Reference to troubleshooting documentation

Basic Interoperability:
SunScreen supports IPsec/IKE with

IPsec with Manual Keying

Pre-shared Keys

Self-signed certificates (RSA-MD5, RSA-SHA1, DSA-SHA1)

CA Signed X.509 Certificates (RSA-MD5, RSA-SHA1, DSA-SHA1) For remote administration traffic, SunScreen only allows IKE with Self-signed or CA signed certificates. IKE with pre-shared keys and IPsec with Manual Keying are disallowed for security reasons.

Windows 2000 and XP support IKE with

Pre-shared Keys

Microsoft Kerberos v5 protocol

CA Signed x.509 certificates (RSA-MD5, RSA-SHA1)

Windows and SunScreen both support IKE in either Transport or Tunnel mode.

Based on the above information, Windows IKE can interact with SunScreen IKE using Pre-shared keys or CA-signed certificates (RSA-MD5 or RSA-SHA1) for normal connections, and CA-signed certificates only for remote administration connections. In the case of CA-signed certificates, the certificate on each system must be signed by a common root CA.

Using mmc to set up the Windows 2000 system for IKE support
- Click Start->Run..., type mmc

A window will appear, labeled Console1, with a subwindow labeled Console Root.

- Click Console->Add/Remove Snap-in

- Click "Add..." and add the following snap-ins:

Certificates: Choose Computer Account, Local Computer

IP Security Policy Management: Choose Local Computer

- Click OK, then Console->Save to save these settings.

Connection from Windows system to SunScreen, using pre-shared keys in Transport mode
The following example is the simplest IKE connection scenario. This setup is useful for getting familiar with the administrative interfaces of both systems.

Windows 2000 System IP address =

SunScreen IP address =

Windows System Setup:

- Open "IP Security Policies on Local Machine".

These policies can most easily be accessed through "mmc" but are also available through:

Control Panel-> Administrative Tools ->Local Security Policy, click on "IP Security Policies on Local Machine"

With "mmc", there will be more options for troubleshooting later, so this example will use that method. Open "Console1", as created above.

- Create a new security policy

There will be three default policies listed: Client, Secure Server, and Server. We will create a new custom policy.

Click Action-> Create IP Security Policy. The IP Security Policy wizard will appear.

Click Next, name the policy, and click Next again.

Uncheck "Activate the default response rule" and click Next.

Click Next. Leave "Edit properties" checked and click "Finish".

Click "Add..." and then Next. Since we are using Transport mode for this example, click the radio button entitled "This rule does not specify a tunnel".

Click Next, Select "All network connections", and click Next again.

Click the radio button labeled "Use this string to protect the key exchange (preshared key)" and enter a string.

Click Next, then choose Add to add a new filter.

Name the filter and click "Add..." again.

A wizard will appear. Click Next. Choose "My IP Address" as the source address. Click Next and choose "A specific IP address" as the destination address. Enter the IP address.

Click Next, then Select protocol type "Any". Click Next again. Click Finish. There will be a be a new IP filter, listed as "Mirrored".

Click "Close" Check the radio button for your new policy, then click Next.

Click Next, then click "Require Security". (Do not click Next yet).

Click "Edit..." and uncheck "Accept unsecured communication, but always respond using IPSec".

Click OK, then Next, then Finish.

There will be a new filter listed and checked. Click Close.

Right-click on the new policy and choose "Assign".

The Windows system should be ready to go at this point.

SunScreen system setup:

- Create a new policy or edit the existing policy.

- Create an address object for both the Screen and the Windows system. In our example, the objects will be called "togatoga" and "win2k" respectively.

- Create a new IPsec Key. This will be the pre-shared key. For SunScreen, we must use the ASCII equivalent of the key we entered in the Windows configuration, therefore the key "thisisabadkey" becomes 0x746869736973616261646B6579.

- Create a rule allowing the services that will be encrypted. The source will be win2k, destination togatoga and action ENCRYPT.

- Choose encryption type "IPSEC IKE". Set some suitable parameters. For this example, we will use:


IKE = 3DES, SHA1, 2, PRE-SHARED, Preshared Key = preshared_key.

Note: For these parameters, Oakley group 2 must be used if the defaults are to be used on Windows. To view the Windows defaults, go to the IPsec Policy, General Tab->Advanced->Methods page. The defaults are:

Table 1.

Encryption Algorithm
 Hash Algorithm
 Oakley Group

Oakley group 5 is not supported by Windows. Also, Oakley group is referred to as Diffie-Hellman group in the Windows interface.

- Click on the options tab and click "Transport Mode". Choose the Screen for the Destination Screen parameter.

- Click OK and save and activate the policy.

- Test the encrypted connection, running snoop to verify that IKE is working.

Obtaining a CA-signed certificate with Windows 2000: Windows Certification Authority
The simplest way to obtain a CA-signed certificate with Windows is by using another Windows system.

The first example will be to get a certificate signed by a departmental Windows Certificate Authority. The software comes as part of Windows 2000 Server and Advanced Server. Please consult the documentation on how to set this server up.

- Connect to the server with Internet Explorer. The URL will be http://servername/certsrv/ 

- Click "Request a certificate", then click Next.

- Click "Advanced Request", then click Next.

- Click "Submit a certificate request to this CA using a form", then click Next.

- Create a Certificate Request, creating a new key set and using the local machine store. This will store the private key in the machine's key store. An example follows.

- Click "Submit".

If the server is set up to automatically issue certificates, you will be prompted to install the certificate. If not, go back to the web site when the certificate is issued.

- Chose "Install this certificate" and the certificate will be automatically added to the local machine's keystore.

You must also add the root certificate of the signer to the list of Trusted Root CAs.

Go back to http://servername/certsrv/ and click "Retrieve the CA certificate or certificate revocation list"

Click on "Install this CA certification path" to make the CA trusted. You will be prompted to add the certificate to the Root Store.

To verify that everything has gone smoothly, open mmc and the console you have been working on (Console1 in this example). Go to Certificates. The root CA certificate should be located under Trusted Root Certification Authorities or Intermediate Certification Authorities. Your signed certificate should be listed under Personal->Certificates. Double-clicking on it should indicate that there is a private key that corresponds to the certificate and the certification path should show OK. An example follows:

Using a Windows CA server to generate a CA request to a non-Windows CA
In this example, we will use a Netscape CMS server as the Certificate Authority. This procedure would work just as well to a commercial CA. In that case, it might not be necessary to import the root CA's certificate, as there are several root certs preinstalled in the Windows Certificate Store.

As far as the author is aware, there is no straightforward way to generate a private key and a CSR (certificate signing request) without using the Microsoft Certificate Services or installing a web server. Any information to the contrary would be greatly appreciated. It is possible to import a key and certificate pair in .p12 format if that method is desired. This example will show how to generate a request on behalf of the client.

- Connect to the Windows CA server at http://servername/certsrv/ as in the previous example.

- In the form, choose "Use local machine store" and "Save request to a PKCS#10 file". The effect of these actions will be to generate a private key and put it in the local keystore and the generation of a CSR in standard format that can be sent to any CA. No issuance request will be made to the local CA.

The CSR is now ready to send to a certificate authority. An example with Netscape CMS follows. Procedures will be similar for any commercial CA.

- Click on SSL Server. Cut and Paste the CSR request from the file just generated and click Submit. Take note of the request number.

- Wait for email confirmation that the certificate has been issued and then go to the Retrieval tab. Enter the request number to retrieve the certificate. Copy the PKCS7 encoded certificate chain to a file.

- Go to mmc->Certificates->Personal->Certificates. Right click and choose All Tasks->Import...

- Find the certificate file and Place it in the Personal Certificate store.

You will see two new certificates installed in the certificate store. One is the system's signed certificate and one is the Root Certificate of the CA.

- Move the Root CA certificate to the Trusted Root Certificates->Certificates folder.

- Double-click on the system's issued certificate and verify its integrity, as in the previous example.

Generating a CSR using OpenSSL
The author considers this method to be the most secure since the Windows administrator has complete control of the private key at all times.

Download the OpenSSL bundle for Windows. There are several precompiled versions available on the Internet. The OpenSSL zip file from the Apache modssl distribution is the easiest to install on Windows, as it extracts all precompiled binaries and libraries into one directory. The example will assume that you have downloaded this distribution and extracted all of the files into one directory. For simplicity, this example will also create all configuration, input, and output files in this same directory.

- Go into the openssl directory.

- Create a skeleton configuration file called "openssl.cnf", for example:

 distinguished_name     = req_distinguished_name
 C              = US
 ST             = MA
 L              = Burlington
 O              = Sun Microsystems
 OU             = QA
 CN             = Win2K Admin
 emailAddress           =              

- Create a private key and CSR:

 C:\openssldir> openssl req -config openssl.cnf -newkey rsa:1024 -keyout key.pem -out req.pem              

Explicitly type in all of the values for the Distinguished Name. Do not just hit return for the defaults, even if they are the same values!

You will have a private key on your system in this directory (key.pem) and a CSR (req.pem). Treat the key.pem as sensitive information. This is your private key!

- Send the "req.pem" file to a CA to be signed. Follow example 2 above for a Netscape CMS server.

- Retrieve your signed certificate and place it in the openssl directory. This example will assume the file is called "signedcert.crt".

- Merge the private key and signed certificate into a pkcs12 file by running the "openssl pkcs12" command, e.g.:

 C:\openssldir> openssl pkcs12 -export -inkey key.pem -in signedcert.crt -out combined.p12              

- Import the certificate into the Personal keystore with mmc -> Certificates -> Personal -> Certificates. Right-click and choose import. Browse to the pkcs12 file (combined.p12) and import it. You should also import the CA root certificate as in example 2 above. Verify the certificate by double clicking on it, making sure that it has an associated private key and that the certificate path is OK.

It is probably wise to delete all of the key and certificate files from the the openssl directory once the key and certificate have been imported.

Obtaining a CA-signed certificate with SunScreen
- Bring up a policy in the GUI.

- Click Certificate->Generate IKE Certificate

- Click "Generate CA Request"

- Enter a Distinguished Name and choose type rsa-md5 or rsa-sha1. For example:

- Click Generate. A private key will be generated and stored in the local database. A CSR request will appear. Cut and paste it into a file.

- Bring this CSR request to the web page of a CA.

Following the example above, we have for the Microsoft CA:

- Request a certificate->Advanced Request->Submit a certificate request using a base64 encoded PKCS#10 file...

- Cut and paste the CSR into the form and click Submit.

- When the certificate is issued, Click Download CA certificate (in Base 64 encoded format), and save this file somewhere.

- Go back to the SunScreen GUI and go to Certificate->Import IKE Certificate.

- Give the certificate a name (this is just a handle for use in SunScreen) and paste in the the text from the file.

- Click Install Certificate. A dialog indicating success should appear.

- Go back to the Windows CA server and click "Retrieve the CA certificate or certificate revocation list" and download the CA certificate, Base 64 encoded.

- Go to the SunScreen GUI and select Certificate Import IKE Certificate. Install this certificate as above.

- Go to Certificate->Search to list the certificates in SunScreen. There will be a special group called "IKE root CA certificates". Select this group from the pull-down and click "Edit". Add the Root CA Certificate to the Include list. This group is similar to the Trusted Root Certificate Store in Windows.

Following the example above for the Netscape CMS CA, we have:

- Click on "SSL Server" and cut and paste the CSR into the PKCS#10 Request form. Fill out the contact information and click Submit. Take note of the request number.

- When you receive notification that the certificate has been issued, go to Retrieval and enter your request number to retrieve the certificate. Copy and paste the Base 64 encoded certificate into a file.

- Go to the SunScreen GUI and and go to Certificate->Import IKE Certificate.

- Give the certificate a name (this is just a handle for use in SunScreen) and paste in the the text from the file.

- Click Install Certificate. A dialog indicating success should appear.

- Go back to the Netscape CMS server and Click Retrieval->Import CA Certificate Chain->Display Certificates in the CA certificate chain for importing individually into a server. Click Submit. Cut and paste the Base 64 encoded certificate into a file.

- Go to the SunScreen GUI and select Certificate Import IKE Certificate. Install this certificate as above.

- Go to Certificate->Search to list the certificates in SunScreen. There will be a special group called "IKE root CA certificates". Select this group from the pull-down and click "Edit". Add the Root CA Certificate to the Include list.

- Save the changes to the policy and test the connection.

Creating a IKE VPN punch-in for Windows systems with arbitrary IP addresses
This example will assume a Windows system with an arbitrary IP address, a SunScreen with a static "Internet" facing address of and a private "internal" network of

Windows setup:

Follow the basic steps of the pre-shared key example above, with the following changes:

- When asked whether this rule specifies a tunnel, click "The tunnel endpoint is specified by this IP address". Fill in the "Internet" facing IP address of the SunScreen.

- Continue through as in the pre-shared key example until reaching the "Authentication Method" page.

- Select "Use a certificate from this Certificate Authority (CA):" and click browse.

- Select the certificate of the Root CA that signed your certificate. Note that there will not be a list of your system's certificates, only those of the CAs.

The Windows system will automatically select its own certificate that was signed by this CA when the IKE negotation happens.

- Click Next and continue on to the "IP Filter List" page. Create a new IP filter as in the pre-shared key example, with the following change.

- At the "IP Traffic Destination" page, select "A specific IP Subnet" as the Destination address and fill in the "internal network".

-When you are done, edit the Filter and uncheck the "Mirrored" box.

- Keep filling out the forms as before, select the new IP filter's radio button and click Next, continuing on as in the preshared-key example.

- Now create a another IP filter list, as above. This filter will be for the opposite tunnel direction.

The tunnel endpoint this time will be the IP address of the Windows system. The Microsoft documentation explains that IPsec tunneling is meant for static IP addresses. If the IP address of the system changes periodically, this value will also have to be changed, unfortunately. Transport mode would not require this configuration change, but would have to assume Internet routable addresses internally. The author is not aware of any way around this limitation.

- Choose the same values as before for the Certificate, etc.

- Create a new filter with the opposite source and destination addresses, and uncheck the mirror box. Select this new filter and continue on as before. When you are finished, you should have two rules checked, each the mirror image of each other, both in filtering rules and tunnel addresses.

- Be sure to assign this new security policy as active.

SunScreen setup:

- Create the following address objects, if not already defined:

"internal-network" - an object with the addresses in the internal network ( in this example)

"Internet" - everything except the local addresses, i.e. include "*", exclude "internal-network" and any other SunScreen IP addresses.

- Create an object for the Windows IKE certificate.

- Click Certificate->Associate IKE Certificate

- Give the object a name. This is just the "handle" for use in SunScreen.

- Enter the Distinguished Name of the certificate. This can be found in Windows by double-clicking on the certificate and viewing "Subject "in the Details tab. For example, in Windows:

The Distinguished Name should be entered in the format: SUBJECT=CN=name<space>,...,C=country. Note that the output on this screen may not be entirely accurate. The "S" may be "ST" and the "E5 may be "MAILTO". The ordering is also backwards. So the Distinguished Name above would be entered in SunScreen as "SUBJECT=C=US, ST=ma, L=burlington, O=sun, OU=qa,, CN=Test User 2". Do not put quotes around the name in the GUI. For example...

It is extremely important that the Distinguished Name be entered EXACTLY correctly, or the IKE negotiation will fail!

Please reference the troubleshooting document for information regarding verification of Distinguished Names.

- For scalability, create a new certificate group and add this certificate to that group. In this example, we will call the group "Internet-certs".

- Create a rule allowing these services into the local network in a tunnel.

The source address would be Internet and the destination address would be the internal network.

- In the IKE window, choose "Internet-certs" as the source certificate and the SunScreen's certificate (signed by the same CA) as the destination certificate. The Authentication Method should be RSA-SIGNATURES, as that is the Windows compatible choice. Algorithm choices should follow the same principles as in the pre-shared key example. A sample screenshot follows...

- Click on the "Options" tab and choose "Tunnel Mode". The Source Tunnel field should be blank, as the Windows systems will be using their actual IP addresses. The Destination tunnel should be the address object for the "Internet" facing address of the SunScreen. The Destination Screen should be set to the SunScreen. For example:

- Save the rule and save, activate the policy, and test the connection to the internal network.

Using Windows as a Remote Administration Station
This example will use a Windows system as a remote admin station in Transport mode. It will assume a dynamic IP address.

Windows setup:

Create a new IP Security policy in "mmc". Follow the example instructions for pre-shared keys above, with the following exceptions:

- If there is a separate administrative interface, use that as the IP address (we will use the same address,, for this example).

- Instead of choosing pre-shared keys, choose "Use a certificate from this Certificate Authority (CA):", as in the previous certificate examples. Remember, remote administration with IKE is only allowed with certificates.

SunScreen setup:

- Edit the policy you wish to remotely administer.

- These instructions assume the existence of some SunScreen objects that were created in the IKE punch-in example. Refer to those instructions for more details.

- Edit the existing screen object by pulling down to "Screen", clicking "Search", pulling down to the object, and clicking Edit. Click on the "Primary/Secondary" tab. If you wish to limit what IP address can be used for administration, pull down on "Administrative IP Address" and select an address. Pull down on "IKE Administrative Certificate" and select the screen's CA signed certificate". For example:

- Click OK tochange the screen object.

- Click on "Administrative Access". Add a new access rule for Remote Administration.

The Screen object should be the screen. The address object should be "Internet" (this can be any address you choose if you wish to limit by IP address as well). Choose appropriate IKE parameters, using the Windows CA signed certificate as the Source Certificate. Note that the Destination Certificate field is grayed out. The Destination Certificate is the IKE Administrative Certificate defined in the screen object above.

- Click on the Options tab and select "Transport Mode". Click OK, save the changes and activate.

- To test, bring up a browser on the Windows system and connect to http://screen_ip_address:3852/. Be sure to have proxying turned off in the web browser.

Note on "SUBJECT=" vs. "ISSUER="
When associating IKE certificates, the previous examples used "SUBJECT=" when defining the Distinguished Name. Naming a certificate in this manner allows access by that one particular certificate. It is possible to define an IKE certificate association with "ISSUER=CA_Distinguished_Name", where CA_Distinguished_Name is the Name of the CA that signed a certificate. This kind of scenario is sometimes used for scalability, for example, trust all certificates signed by the departmental CA server. By doing this, an entry for each individual certificate is not needed. While this setup requires less administration, the security implications make it less than desirable and we recommend against it.

Command Line considerations
All of the operations shown in the SunScreen GUI can also be done via command line. The command line method has a few more steps and is more error-prone, due to possible typographical errors, however it is a viable alternative to the GUI. This article will briefly describe some common operations.

Creating a CSR for a CA-signed certificate:

Use "ssadm certlocal" to create a private key and CSR, for example:

 # ssadm certlocal -Ikc -m 1024 -t rsa-sha1 -D "CN=SunScreen Admin, OU=QA, O=Sun Microsystems, C=US"
 Generating, please wait...
 Certificate Request generated.

The above operation will create a private key in the SunScreen database. The output is suitable for cutting and pasting for submission to a CA.

Importing a certificate:

Importing a certificate on the command line takes two steps.

Step 1: Import the certificate into the SunScreen database

You shouldhave an ASCII certificate file, i.e. something like this:

 # more /tmp/cert
 -----END X509 CERTIFICATE-----    

Use the "ssadm certdb" command to import the file, for example:

 # ssadm certdb -Ia < /tmp/cert

Step 2: Create a SunScreen object associated with this certificate.

The Distinguished Name string must be entered exactly, so confirm the same first with "ssadm certdb"...

 # ssadm certdb -Il
 Certificate Slot Name: 4   Type: if-modn
             Subject Name: <CN=tk-host2, O=Sun Microsystems, C=Argentina>
             Key Size: 2048
             Public key hash: 459BC59799FCF7C01B37B9C1A2457154

Then add the certificate with your choice of object name and the exact Distinguished Name, e.g.

 edit> add certificate "tk-host2-cert" SINGLE IKE "SUBJECT=CN=tk-host2, O=Sun Microsystems, C=Argentina"    

If it is your own certificate (i.e. you have the private key), add "LOCAL screenname" at the end, for example:

 edit> add certificate "togatoga-CA-signed" SINGLE IKE CN=SunScreen Admin, OU=QA, O=Sun Microsystems, C=US"
        LOCAL "togatoga"

Adding a certificate to the proper group:

Any CA root certificate must be added to the "IKE root CA certificates" group in order to be trusted. Assuming you imported a root certificate and called it "myCA-root", an example would be:

 edit> add_member certificate "IKE root CA certificates" "myCA-root"    

Exporting a certificate:

The ssadm certdb command can also be used to export a certificate. For example:

 # ssadm certdb -Ie "CN=tk-host2, O=Sun Microsystems, C=Argentina" > /tmp/cert

Sample rules:

IKE rules can get very long and cumbersome on the command line. Here are some example rules to help with the syntax. Please refer to the man pages for more information.

IKE with pre-shared keys in Transport mode:

 # ssadm edit ike-preshared-transport -c "list rule 1"
 1 "ftp" "win2k" "togatoga" IPSEC ESP("3DES", "SHA1") IKE("3DES", "SHA1", 2, PRE-SHARED,
   "preshared_key") TRANSPORT DESTINATION_SCREEN "togatoga" ALLOW

IKE with certificates in tunnel mode:

 # ssadm edit ike-cert-tunnel -c "list rule 1"
 1 "ftp" "Internet" "internal-net" IPSEC ESP("3DES", "SHA1") IKE("3DES", "SHA1", 2,
   RSA-SIGNATURES, "Internet-certs", "togatoga-Netscape-issued") DESTINATION_TUNNEL
   "togatoga" DESTINATION_SCREEN "togatoga" ALLOW

IKE remote admin rule

 # ssadm edit ike-remote-admin -c "list screen"
 "togatoga" ADMIN_IP "togatoga" IKE("togatoga-Netscape-issued")  CDP DNS NIS
 # ssadm edit ike-remote-admin -c "list accessremote"
 1 SCREEN "togatoga" USER "admin" "Internet" IPSEC ESP("3DES", "SHA1")

Due to space constraints, troubleshooting IKE connections is covered in another article, Infodoc 47821.

All references to 'Windows', 'Windows 2000', or 'Windows XP' refer to the
Microsoft (R) Windows operating system, Microsoft (R) Windows 2000, or
Microsoft (R) Windows XP respectively. Screen shot(s) reprinted by
permission from Microsoft Corporation      

LVL 38

Expert Comment

ID: 9737461

ID47821   Troubleshooting SunScreen[TM] IPsec/IKE   16 Oct 2002



Description Top

Document Body Top

Basic Troubleshooting of SunScreen IKE

Note: This document does not discuss basic IKE setup. Please refer to the documentation or other articles for that information.

These troubleshooting tips are based on what the author has seen in his own testing and observations of mistakes by various users of the product. This is by no means an exhaustive list of troubleshooting techniques. Keep in mind that many of these methods will involve killing the IKE daemon or removing SA's from the kernel, so care should be taken in a production environment as they could interrupt ALL active IKE connections.

"It doesn't work" - How to start

You activate a policy, attempt a connection, and it just appears to hang. The usual complaint of "it doesn't work" usually indicates that we do not have enough information to point to what the problem is or perhaps we do not have a methodology to figure out what the problem is.

Start with the basics.

Many times it is good to do a sanity check before getting too far into troubleshooting.

Do you have the correct IP addresses?

Do the keys, algorithms, certificates, and other parameters match?

Use snoop

What do you see between the two systems when you run snoop? Are IKE packets being exchanged on UDP port 500? Do you see IKE traffic in only one direction? Do IKE negotiations happen over and over? Do you see encrypted traffic (ESP or AH packets, IP protocol 50 and 51)? If encrypted traffic appears to actually pass as expected between the SunScreen(s), is your routing in working order?

Investigate based on what you found with snoop. Some possibilities are described below.

Symptom: Traffic for IPsec with Manual keying is not getting decrypted on one or both sides

Likely Causes:

rule mismatch

key mismatch

improper number of bits in key definitions


If your rules and keys are matching on both sides, but the keys you defined do not have the right number of bits for the algorithm, encryption and decrypting will fail. Check the man page for the algorithm to determine the number of bits. Generating a key in the GUI will automatically produce the right number of bits for the algorithm if you need help.

Symptom: IKE traffic is exchanged on port 500, but you never see encrypted IPsec traffic

Likely Causes:

Peer certificate missing from "IKE manually verified certificates" group

CA cert missing from "IKE root CA certificates" group

Certificate definition not defined correctly or typo in Distinguished Name

Intermediate network device blocking traffic

Stale SA information

Problem/Bug with IKE daemon on one or both sides

Problem/Bug with screen_ipsec kernel module


Check the certificate groups.

If you are using self-signed certificates, the self-signed certificate of the peer must be in the "IKE manually verified certificates" group or the IKE authentication will fail.

If you are using signed certificates, the root certificate of the CA must be in the "IKE root CA certificates" group.

Make sure that no other device is blocking traffic

Enable packet logging and check the packet logs for IKE messages

View with:

 # ssadm log get logapp iked | ssadm logdump -i- -ta -V                  

Bounce the daemon, especially on the sender

Note: this is very obtrusive and should only be done in testing. On SunScreen,

the easiest way to do this is:

 # pkill -f ss_iked; ssadm activate policy                  

Expire all SA's

 # ssadm lib/statetables -fs                  

The initiator will try to renegotiate in this case. Again, this is very obtrusive.

Start the ike daemon in debug mode

  # pkill -f ss_iked
  # LD_LIBRARY_PATH=/usr/lib/sunscreen/lib /usr/lib/sunscreen/lib/ss_iked -Sd                  

(daemon will start in debug mode in the foreground)

For ongoing or unpredictable problems, you can have the daemon automatically go into debug mode and log to a file. Edit the




    $IKED -S  


   $IKED -Sd >/some/file 2>&1 & 

where /some/file is the path to some file on a partition large enough to hold ongoing debug output. Note the ampersand at the end of this line. Without this ampersand, the activation of policies will appear to hang.

The debug output is sometimes useful, sometimes not. Many times you will get an English message about some certificate or algorithm problem. Sometimes numerical codes come up. Error 24 means authentication failed during Phase 1. Although that is not very specific, it at least gives a direction to look into. Error 24 messages are sometimes accompanied by certificate Distinguished Names. Make sure that the peer Distinguished Name matches the configuration exactly. Error 8197 means "Negotiation timed out, the other end didn't respond". Remember that sometimes the other end did not respond because the IKE packets did not get there. Error 14 means no proposal chosen, which would indicate that the two systems cannot agree on a common authentication method, authentication algorithm, encryption algorithm, and Oakley group. Error 16384 means that Phase 1 completed successfully, i.e. it is not really an error. In this case, check the packet filter logs to make sure that the SunScreen[SM] is not blocking outgoing traffic due to its rulebase.

Below is a table of some error codes that might be useful:

Table 1.

Error Code
 Invalid flags
 Invalid SPI
 Attributes not supported
 No proposal chosen
 Payload Malformed
 Invalid Key Information
 Invalid ID Information
 Invalid Certificate Encoding
 Invalid Certificate
 Certificate Type Unsupported
 Invalid Certificate Authority
 Invalid Hash Information
 Authentication Failed
 Invalid Signature
 Certificate Unavailable
 Negotiation timed out, no response from peer
 Received UDP Host Unreachable from peer
 Received UDP Port Unreachable from peer
 IKE phase 1 negotiation successful

Symptom: IKE traffic is only seen in one direction

Likely Causes:

ss_iked died on one of the systems

routing misconfiguration

ss_iked in unresponsive state

intermediate network device blocking traffic

Did iked die on one of the systems?

Possibly ICMP error messages about port 500 may come back, depending on config. Look for a core file, or if it is reproducible, get debug output to find out why it core dumped or exited.

Is iked "hung" or in some other undesirable state?

Bouncing it may help. Try to find out what the bug is in this case and reproduce it.

Are the rules using the right tunnel addresses?

Is something blocking the IKE traffic or is the traffic passing through the firewall because of some problem with the rules?

Symptom: IKE negotiated successfully, but there is only IPsec traffic in one direction

Likely Causes:

SunScreen is blocking decrypted traffic due to rules

Routing on SunScreen or "internal" device is misconfigured

Check the traffic logs for both issues with filtering and with IKE. Make sure that detailed logging is enabled on the interface in question.

Check the routing again and run snoop on the inside. Is traffic getting through? If not, routing or IP forwarding on the firewall may be to blame, or possible anti-spoofing rules on the interfaces are incorrectly defined. If packets are getting through, routing on the internal hosts or routers may be to blame. Routing is particularly prone to errors in a tunnelling configuration.

Compilation Error Messages

I am getting this error message when trying to compile:

  Command failed (status 245)
  Error message: security_parameters direction error: "telnet" "host1" "host2" IPSEC
  ESP(0x90125, "DES", "key") AH(0x2112, "MD5", "key_ah") TRANSPORT ALLOW compiler error

A security parameters direction error indicates that the compiler cannot figure out who is supposed the be the encryptor or decryptor. This will happen when creating a manual IPSEC or IKE with pre-shared keys rule without defining a source or destination screen. This can also happen when there is some problem locating your private key in the IKE (or SKIP) database.

You can look at what is in the IKE certificate database with:

 # ssadm certdb -Il                  

and see if you have the associated IKE private keys with:

 # ssadm certlocal -Il                  

One hint: If you run the certdb -Il command and you have the private key, the output will look like

 Certificate Slot Name: 3   Type: if-modn
              (Private key in certlocal slot 7)
              Subject Name: <C=US, O=Sun, OU=Service, CN=test123>
              Key Size: 512
              Public key hash: 11111111111111111111111111111111                  

The private key message will be missing if you only have the public key.

Conversely, if you have the private key for both the source and destination certificates, you will also get this error. This second scenario of having both private keys should never happen. It means that either you are using two of your own certificates or that you have the private key of the peer, therefore making it compromised. This second scenario might happen in testing but hopefully not in a real installation.

Another reason that you may get this error is that the DN used in your SunScreen certificate object does not match the DN in the IKE database. This condition should not happen if you create and import certificates using the GUI, but if you manually import the certificates and define the objects with the command line, it may be a common occurrence. Spaces and case matter!

When using signed certificates, if one certificate is a root CA and the other certificate is signed by that CA, you may also get this message. The remedy in this case is to define source or destination screen in the configuration.

Error: a rule in the policy uses the IPsec encryption algorithm AES,
but this algorithm is not installed on the system.  
Please install this algorithm from the package SUNWcryx.
The policy cannot be activated until you install this package,
or remove the rule(s) that use this algorithm.
compiler error
compiler error

This error will occur in two situations. The first is obvious and what the error message states: you are using an algorithm that is not installed on the system and you need to install the packages that contain this cryptor. On Solaris[TM] 9, DES and 3DES are included without installing any additional software, but other algorithms such as AES need to be downloaded.

The second situation where this can occur is when the kernel module for the algorithm in question is installed on the system, but did not get loaded. On a new installation where you are upgrading, a simple reboot can fix this and it won't happen again. Troubleshooting techniques are to try to manually load the kernel module and see if it loads or figure out why it won't load.

Further SunScreen IKE troubleshooting

The above troubleshooting tips are by no means inclusive of everything that you can do. More advanced methods include using "ssadm debug_level" and setting the IPSEC and CDP bits high. This will produce a lot of debug output on console and slow down the machine considerably. It is a good method for debugging problems that are reproduced in the lab, when a kernel type of issue is suspected.

Windows 2000 IKE troubleshooting help

When using Windows as an IKE peer, there are a few considerations.


- When using certificates, make sure that there is only one CA listed in the configuration. Windows allows you to add multiple CAs and authentication methods and this has been known to sometimes cause interoperability problems.

- When using certificates, be sure that the Distinguished Name in the SunScreen configuration matches what is in the certificate. When viewing a certificate in Windows, some abbreviations are used for components. For example, "S" should be "ST" and "E" should be "MAILTO". The order of elements in the Distinguished Name is also reversed. When in doubt, run ss_iked in debug mode on the screen and look at what certificate the Windows system presents.

- When using pre-shared keys, Windows expects the key as a typed string and SunScreen expects the key as the ASCII equivalent. So, for example, the key "abc" on Windows is "0x616263" on SunScreen.

- Be sure that the SunScreen IKE parameters that were chosen are compatible with Windows. The only valid Windows certificate authentication method is RSA-SIGNATURES. For other IKE values, follow the following table for compatible Windows defaults:

Table 2.

 Oakley (Diffie-Hellman) Group

On an export controlled system, 3DES may not be available. Windows automatically downgrades to DES. A message shows up in the audit log, but this action may not be apparent from the configuration. Thus the table on an export controlled Windows 2000 system will look like this:

Table 3.

 Oakley (Diffie-Hellman) Group

Note: Upgrading to Service Pack 3 will install 3DES and avoid this problem altogether.

Debugging actions:

-Force the Windows client to renegotiate.

Method 1: Unassign and Assign the activate policy

Right-click on the active IP Security Policy. Click Un-assign. Right-click again and click Assign. Retry the connection.

Method 2: Stop and start the IKE service

On the command line, type

 C:\> net stop policyagent
 The IPSEC Policy Agent service is stopping.
 The IPSEC Policy Agent service was stopped successfully.
 C:\> net start policyagent
 The IPSEC Policy Agent service is starting.
 The IPSEC Policy Agent service was started successfully.                  

- Get a debug log

Use regedit to create a registry key:


Add a key called EnableLogging with REG_DWORD value of 1.

Stop and start the IKE service, as above. There will be a very verbose log file called


Some of this debugging information can be useful in tracking down a problem.

Comprehensive Backup Solutions for Microsoft

Acronis protects the complete Microsoft technology stack: Windows Server, Windows PC, laptop and Surface data; Microsoft business applications; Microsoft Hyper-V; Azure VMs; Microsoft Windows Server 2016; Microsoft Exchange 2016 and SQL Server 2016.

LVL 38

Expert Comment

ID: 9737511
Here's an IPsec How-To, ir covers Solaris, Windows etc:

Author Comment

ID: 9739888
Okay, Yuzh it would appear that you are confirming pretty much what I'd found discussed on the net, that IPsec using IKE between Solaris 8 and W2K is a no go. Our own Sun guys have finally said pretty much the same thing also, so I think its time to look down another route as we can't move to Solaris 9 (WebSEAL is not certified for 9).

LVL 38

Expert Comment

ID: 9744171
If you stay with Solaris 8, you can use secure shell connetion between the SunBox and the Windows
PC. You can use X tunneling to run the UNIX app and display on the Windows box. You need to run
sshd (secure shell sever deamon in the Sun Box).  Secure shell comes with secure FTP (sftp), you can
get it from (free):


Author Comment

ID: 9746635
I should maybe better explain the setup. Solaris is running WebSeal, part of our Identity Management infrastructure. It is reverse proxying connections to our backend Domino Servers (via ICMs) which run on W2K. So the connection is coming via the Solaris boxes to the W2K boxes. These are all servers and not W2K clients connecting to Solaris to run Unix apps.

Anyway I think I'll now abandon the IPsec solution for now and look at another for this release.

Expert Comment

ID: 10101348
I have set this up before. I used a preshared key for the authentication. The tricky part is to know how Windows IPSec works. If you are running it as a server (ie. RRAS is not running essentially turning it into a router), then Windoze will running the IPSec in TRANSPORT mode rather than tunnel mode. If I remember correctly, solaris will try to running in TUNNEL mode by default. The only way to modify the mode setting in Windows is to install RRAS which is bad bad bad; so, best to reconfigure Solaris to run in transport mode.

Author Comment

ID: 10103357
              thanks for your comment. This info is very interesting and hopefully will help anyone else that may be trying to configure such a set-up. We have decided to go down another road , but would not have been allowed to use pre-shared keys anyway. It is very useful to know that someone has used this setup and got it working though.#

Always better to hear how someone HAS done it than to just read the theory of how it SHOULD work.


Featured Post

Complete VMware vSphere® ESX(i) & Hyper-V Backup

Capture your entire system, including the host, with patented disk imaging integrated with VMware VADP / Microsoft VSS and RCT. RTOs is as low as 15 seconds with Acronis Active Restore™. You can enjoy unlimited P2V/V2V migrations from any source (even from a different hypervisor)

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

FreeBSD on EC2 FreeBSD ( is a robust Unix-like operating system that has been around for many years. FreeBSD is available on Amazon EC2 through Amazon Machine Images (AMIs) provided by FreeBSD developer and security office…
OfficeMate Freezes on login or does not load after login credentials are input.
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.

832 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question