HI,
For 32 bit Vista: the basic Windows VPN will not work. You need the Cisco VPN Release 5!
For 32 bit Vista: the cisco VPN clint is not supported at the moment! Please Install Anyconnect!
Best Regards,
Istvan
Main Topics
Browse All TopicsI have done extensive research on the internet regarding this subject and I am at my wit's end. I have spent the last few weeks working on configuring a new Cisco ASA 5510 VPN connection utilizing L2TP/IPsec. The entire configuration is in a lab environment and works perfectly. I can login to the VPN and move traffic using my iPhone. The same is true for the clean install Windows XP SP3 laptop that I have for my test environment.
The problem comes from the 2 windows vista laptops that myself and the IT Director are using to connect to the VPN. I have an x64 Vista Business SP2 Dell Precision M4400, and my boss has one of the new Latitudes with x86 Vista Business SP1. Neither one of them can connect and immediately receive the following: Error 789 - The L2TP connection attempt failed because the security layer encountered a processing error during initial negotiations with the remote computer.
The failure is immediate, so I assumed that it had something to do with IKE. After installing the Microsoft IPSec Diagnostic Tool, and running it for the remote connection to the ASA, it returned the following: IPSec Service Diagnosis - Failed: IKEEXT and PolicyAgent services are not running. Ipsec is inactive. Putting the ASA in debug mode for isakmp and ipsec is to no avail as the failure is before it even reaches the device.
I ran services.msc and looked for the IKE and AuthIP (something like that for the name, I can't remember exactly what) and the IPsec PolicyAgent service, and to my surprise, neither one of them exists in my services window at all. After a scan through the registry, I saw that both of these services have a key named ServiceDllUnloadOnStop set to a DWORD value of 1, but many other services have that as well, and many of them are listed in the services.msc window.
I have a feeling that this is a Vista issue (maybe even a Dell OEM Version issue), but the folks at Dell didn't know what I was talking about. My next step is to go in on Monday and do a fresh Vista Business install on that laptop and see if those services exist.
What I am wondering is if anyone has seen this before and could maybe point me in the right direction as to how to get those services enabled so that they will run. I have a feeling that it is the only thing that is keeping me from being able to finish this project. I have included my ASA configuration below, but I do not believe that this has anything to do with my vpn settings as both the iPhone and Windows XP clients can connect with their respective native clients.
Thanks you all for any guidance you could give me in regards to this problem
This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.
Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.
If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.
Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.
Access the answers to your technology questions today.
30-day free trial. Register in 60 seconds.
Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.

Try it out and discover for yourself.
30-day free trial. Register in 60 seconds.
Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.
Maybe I should have been a little more succinct with my question, but I wanted to show everyone the things that I have already scratched off the list of possible reasons why this is not working.
I have it narrowed down to being an issue regarding the loading of IKEEXT and IPSec PolicyAgent services. Something seems to be blocking those services from loading and I can't figure out what all it is, and how I can force them to start. Any suggestions allong these lines would be greatly appreciated, or even a direction I can start to look in. I am also wondering if it could be Active Directory related, so I went through the local policy, and couldn't find anything (maybe I am looking in the wrong place).
-Istvan: We are not wanting to use the Clisco VPN client because it cannot be controlled through group policy. Also, it has issues telling Vista that it is connected to the corporate network. The Windows client itself can change the state of the OS to connected automatically and the sync will work as intended. As for AnyConnect, we are not licensed for it, and according to the documentation at Cisco.com and the technician I spoke with at Cisco, using an L2TP/IPsec connection is fully supported.
-Mutahir: Well, on the cisco side of things, I cannot get a log as the failure is prior to the connection. Vista is failing to execute the IKEEXT service (IKE and AuthIP Service Module). If I try and go to a command prompt and enter NET START IKEEXT I get an invalid service command, yet all of the registry entries for that service and the IPSec PolicyAgent service are listed in the registry, but they don't seem to be registered with Svchost so Windows is not loading the service.
UPDATE:
I managed to get the IPsec services started on my machine. After a little bit of research, I found that if you are missing services in the services.msc screen, you can manually configure their startup mode through the registry. For these particular services, the required key is as follows:
HKLM\SYSTEM\CurrentControl
HKLM\SYSTEM\CurrentControl
Under each of those keys, create a new DWORD value. Name is Start and for Automatic Start, set the value to 2. Once I changed both of those keys to contain that new DWORD, the services started as usual and IPsec was enabled.
At this point, the Microsoft IPsec debugger is useless, and below is the output from the isakmp debugging on the ASA. Now, at least they are talking to each other, but phase 1 IKE still isn't authenticating. I may have to direct this over to the Cisco support folks though, but any advice you all can provide would be greatly appreciated.
http://support.microsoft.c
check the above links out ; also do you get any error code on vista at this step ?
http://www.stevens.edu/itw
From the Above Link :
789: The L2TP connection attempt failed because the security layer encountered a processing error during initial negotiations with the remote computer
Cause: Your operating system is not correctly configured to connect to an L2TP server. The connection attempt is failing before a connection to the server is ever attempted. This error is caused entirely by a misconfiguration of the user's operating system. Location, or connect type has no bearing on receiving this error message.
Solution: Windows Vista
Open Control Panel >> Administrative Tools >> Services
Make sure the IKE and AuthIP IPsec Keying Modules service is started and configured to start automatically at startup.
If the service is already started, restart it.
Make sure the IPsec Policy Agent service is started and configured to start automatically at startup. If the service is already started, restart it.
Attempt to connect to the VPN server again.
I think you should be able to connect once you have checked the settings again ;
Also, check this link as well :
http://forums.isaserver.or
UPDATE:
Everything works now in terms of the routing issues using L2TP\IPsec. I still have a DNS issue, but for that, I believe that I need to contact Microsoft. According to Cisco, applying the intercept-dhcp enable to the group-policy and properly configuring your VPN client to not accept the default route from the ASA corrects the split-tunneling issues that I was having.
The Code Snippet shows a final, working ASA configuration to support connections from the Windows Vista VPN Client (x86 or x64), the Windows XP Client, the iPhone VPN Client, and the Cisco VPN Client.
Below are the steps required to configure the Windows VPN Client:
1. On the General tab, put the public IP address of the outside interface of the ASA
2. On the Options tab, click PPP Settings... and make sure that only the first checkbox is checked
3. On the Security tab, the defaults can be left alone
4. On the Networking tab, according to documentation found on the net, disabling connections types that you do not need is best practice. I only left TCP/IPv4 enabled.
5. No matter the choice in step 4, enter the properties for IPv4. On the properties page, click the Advanced button. On the IP Settings tab, ensure that you uncheck the "Use default gateway on remote network" box.
That should be all that you need to get L2TP\IPsec connectivity between Vista and the ASA up and running. The last piece is to get the DNS information to pass down (which doesn't seem to work currently). I hope that this information helps someone out there with getting a similar configuration up and running.
Business Accounts
Answer for Membership
by: mutahirPosted on 2009-08-01 at 02:28:28ID: 24994656
Your feeling is right !
I have seen and faced this error not with ASA but with draytek routers and it just won't connect ; turned out to be a vista issue as other os can connect l2tp/ipsec.
I would suggest is to get a log of the vpn transaction on either side and contact both vendors (cisco and microsoft) with the respective logs.