VPN L2TP connection to Zywall USG100, Windows xp error 789


Windows XP Pro VPN L2TP connection to Zywall USG100.

error 789 : The l2tp connection attempt failed because the security layer encountered a processing error during initial negotiations with the remote computer

It happened to windows xp pro dial up connection only, so when laptop is behind another zywall usg100 ADSL connection VPN works fine, but when laptop is dialup connection (mobile) error is there. Where is problem?
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.


does the zywall show any logs on this?
Is the windows firewall off? (or any other firewall- macafee, etc) udp 4500, 500 ESP (protocol#50) traffic might be getting blocked
Also, are you using a preshared-key for the xp client and the zywall? (if so, try re-setting)

on the xp machine, make sure the following services are started, and if they are. try restarting, then try connecting:

IPSEC Services
Raikka_Author Commented:
No logs at all, windows firewall is off, yes preshared key and ipsec services is restarted. Same problem.

Behind ADSL and another zyxel USG100 workstation connects well.
any detailed logging besides error message in windows event logs?
Get Blueprints for Increased Customer Retention

The IT Service Excellence Tool Kit has best practices to keep your clients happy and business booming. Inside, you’ll find everything you need to increase client satisfaction and retention, become more competitive, and increase your overall success.

Raikka_Author Commented:
Yes, Event ID 1002:

The IP address lease for the Network Card with network address BAD0BEEFFACE has denied by the DHCP server (The DHCP server sent a DHCPNACK message)
Raikka_Author Commented:
Computer IP is:


VPN server (USG100) where trying to connect IP is:
Raikka_Author Commented:
Client VPN error:

Error 792


Error 789
the workstation that connects successfully-  is it getting a dhcp address in the 93.106.8.X  subnet ?
Is the xp machine using a pre-shared key?  This would be in the properties of the vpn connection > security tab>IPSec settings

you stated earlier zywall is using a preshared-key.  Did you reset this key on the client having the issues?
Raikka_Author Commented:
Client is using windows xp built-in VPN connection L2TP, when client is behind another USG100 using builtin VPN it works well, but when using telephone GPRS connection, it fails.

Yes, workstation GPRS connection is 93.106.199.x subnet... There is also 3more laptop, they all use different GPRS connection and all fails.

Only client that what is behind ADSL works well.
Raikka_Author Commented:
Preshared key is correct, bacause it works behind ADSL
did you check all dialup settings on the zywall? Is the zywall configured to accept traffic on the 93.106.8.X
can the dialup machines ping the vpn interface of the zywall before attempting vpn connection?
Raikka_Author Commented:
No, client cannot ping Zywall WAN ip even when VPN connection is establish behind ADSL connection:
ping 83.145.227.xx

reguest timed out.
Raikka_Author Commented:
Now there is open ping from:

But it doesn´t help...
you are positive that the zywall does not have a specific dial-up profile for remote access-users?
try this
on the properties of the vpn connection, go to security tab, then advanced
on the drop-down menu, select "optional encrytion"  to see if it connects without encryption
make sure mschap v2 is checked
we can work out encryption issue once we can connect
then on the networking tab under "type of vpn" click settings
make sure
enable lcp extensions and enable software compression is checked
on type of vpn, try "automatic"
Raikka_Author Commented:
Tested, Don´t help....

Zywall USG manual, there is exsample L2TP, that is where settings are.... Here is similar settings....

But won´t work!:!:!:!::!
I will pick this up later this evening. I will need to look at the zywall manual for this. It might help to use a network sniffer such as Wireshark (free) to get packet captures to see what is happening inside the wire.  You can use built-in netmon in administrative tools also

and again, are these remote users using strictly dialup, ethernet cable or wireless
sorry forgot to ask earlier
p.435-439 gives the settings for the windows client, it does not differentiate if it is strictly dialup. You can make two seperate connection, one strictly for dialup, the other for ethernet, with suggested security settings and try
Raikka_Author Commented:
>and again, are these remote users using strictly dialup, ethernet cable or wireless
>sorry forgot to ask earlier

Dialup connetion (mobile phone) This is strange, because VPN connection works well behind ADSL (cable connection) Is XP VPN client "general" setting "first connect" "dial another connection first" setting okay, it is empty? And it works behind ADSL, not working telephone connection?
Raikka_Author Commented:
USG100 manual page 447, there is L2TP VPN Example, that is using client computers, just like manual settings...
I don't follow you on last comment--
Raikka_Author Commented:
Zywall USG 100 manual:

Version 2.12 page 447 --->

There is L2TP VPN example, that example is used to connect client computer to USG100 VPN router, and it works through ADSL connection but not "mobile" VPN connection...
I am downloading manual, I have an older copy and the page reference is wrong. will take a look at this

are we looking at same thing? Am I missing something, as the example just shows a remote user. Will this firewall allow  PPTP vpn using the windows client instead of l2tp?
Raikka_Author Commented:
USG100 allow 3 different VPN settings:

L2TP VPN (inside IPSEC)
my question is, most of these remote clients are using wireless cards at various locations, and some using PPP dialup connections?
another question I have is that the ip addresses that are assigned to local clients should be private ip's, in a different subnet that your inside lan hosts.

 A few posts ago you stated >>The IP address lease for the Network Card with network address BAD0BEEFFACE has denied by the DHCP server (The DHCP server sent a DHCPNACK message) Are you referring to a dynamic ip that a remote client is getting at their remote location, or an ip the zywall is handing out?

please clarify

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.