Solved

VPN... HASH, NOTIFY:PAYLOAD_MALFORMED!!!

Posted on 2003-10-27
13
6,034 Views
Last Modified: 2013-11-16
Hello Experts, here is my situtation:
We are using a SuperStack3 3com firewall, and we are using its as a primary point for VPN clients to access our network.

I have successfully set up VPN software clients to access this, and the 3Com VPN/Firewall can access it fine as well. When attempting to use a Linksys BESFX41 as a VPN client, I am getting the following error in our 3Com FW logs..

I have tried using different shared secrets, and tried different combos for the encryption and authentication, but it always seems to fail at HASH, NOTIFY:PAYLOAD_MALFORMED.

Any help would be greatly appriciated.

The IP addresses have been removed to protect the innocent :)

10/27/2003 16:10:03.416 SENDING>>>> ISAKMP OAK MM (MsgID: 0x0) (SA)    
10/27/2003 16:10:03.944 RECEIVED<<< ISAKMP OAK MM (MsgID: 0x0) (KE, NON)
10/27/2003 16:10:04.096 NAT Discovery : Peer IPSec Security Gateway doesn't support VPN NAT Traversal (I dont think this is related, but not positive?)    
10/27/2003 16:10:04.096 SENDING>>>> ISAKMP OAK MM (MsgID: 0x0) (KE, NON, VID, VID, VID)  
10/27/2003 16:10:04.880 RECEIVED<<< ISAKMP OAK MM (MsgID: 0x0) *(ID)
10/27/2003 16:10:04.880 SENDING>>>> ISAKMP OAK INFO (MsgID: 0x1426AB29) *(HASH, NOTIFY:PAYLOAD_MALFORMED)
10/27/2003 16:10:12.880 IKE Responder: No response - remote party timeout or SA mis-match

Thanks,

Mike
0
Comment
Question by:UnifiedIT
  • 7
  • 6
13 Comments
 
LVL 2

Expert Comment

by:sh00t3r
Comment Utility
YO Mike,

A couple things...

You'll probably have to open up the port AND port triggering in the linksys routers....

Use the ISAKMP port, 500. Then make sure the Linksys router supports IPSEC. That should do it.

Sh00t3r

0
 
LVL 2

Author Comment

by:UnifiedIT
Comment Utility
Hey sh00t3r..

I just want to make sure we are on the same page here.

The linksys router is the VPN client. on its main page it has a place to test the connection.

Now opening up ports and using port triggering would be for cpu's on my LAN, but I am just trying to create the tunnel between the Linksys and the 3Com right now..

Thanks again..
0
 
LVL 2

Expert Comment

by:sh00t3r
Comment Utility
So is the linksys inside your network?? Or is it a site to site VPN?
0
 
LVL 2

Author Comment

by:UnifiedIT
Comment Utility
The 3 Com is our corporate FW and main access point to our network.

The Linksys is a VPN router located at my home.
I was using a Linksys router and passing IPSec through so I could use the software client on my PC, but I decided to go with a Hardware VPN/FW for the added FW security.

I cannot get the linksys to open the VPN tunnel.

The logs from above were on the corporate FW...
0
 
LVL 2

Expert Comment

by:sh00t3r
Comment Utility
You should try opening up the ports and port triggering as stated above. Then use the software client on your PC. As long as you don't open up anymore ports there won't be too much additional protection that the VPN/FW linksys will do as opposed to just running the linksys as a router. I believe it does stateful inspection of packets but it will continue to do this if you use the suggested setup.

0
 
LVL 2

Author Comment

by:UnifiedIT
Comment Utility
Thanks for your help sh00t3r, but this defeats the purpose for my question.

I can make the software client work.

What I want is to get this working and I should beable to get the hardware VPN connected. I just have run into this snag and I see no reason why we cant get around it and make the harware client work.

Any other ideas would be greatly appriciated.

Thank you

Mike
0
Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

 
LVL 2

Expert Comment

by:sh00t3r
Comment Utility
Do you have PPTP enabled on the Linksys router?
0
 
LVL 2

Accepted Solution

by:
sh00t3r earned 250 total points
Comment Utility
Also upgrade the firmware. Here are some links below that my also provide additional help.

http://www.homenethelp.com/vpn/router-linksys.asp


You may still need to open up those ports I specified even if using the linksys as the VPN client...
http://kb.linksys.com/cgi-bin/om_isapi.dll?clientID=884172&QuestionText=vpn&SelectName1=&advquery=%5bs%5d%5bRank%2c%2050%3a%5bSum%3a%20vpn%5d%5bMerge%3a%20%5bThesaurus%3a%20vpn%5d%5d%5d&infobase=linksysrev.nfo&record={3F0}&softpage=IKW_ENU_JDocView


0
 
LVL 2

Author Comment

by:UnifiedIT
Comment Utility
Ok thanks,

Ill look into it further... I did upgrade the Firmware first thing, and I believe that PPTP is turned off.

I will check out the links and get back to you.

Mike
0
 
LVL 2

Author Comment

by:UnifiedIT
Comment Utility
I cant get this thing to work, it appears that the IPSec that the Linksys uses is different then the IPSec that everyone else uses.. I have 3Com working on this with me.

As far as opening up ports on the router, that is a good idea, but it defeats the purpose of me purchasing this Hardware. I was looking for an extra layer of support, and the L2TP XP client hats the NATed FW, and the 3Com client doesnt provide XAUTH or L2TP authentication..

Basically, I am still looking for that extra level of authentication, and it is not working with my combo of products!

Thank you for your help thus far.
0
 
LVL 2

Expert Comment

by:sh00t3r
Comment Utility
Well no resolution thus far. I don't see how the Linksys would use a different version of IPSEC. IPSEC is IPSEC. Good luck though and let us know how far you get so we can help you with any other issues.

Sh00t3r
0
 
LVL 2

Author Comment

by:UnifiedIT
Comment Utility
Thanks

I have 3com working on it now... Ill let you know the outcome..
From what I understand, the techs at 3com told me that they believe that Linksys uses a proprietary IPSEC protocol. Im not sure if this is true, but I will keep you updated..

Thanks again,

Mike
0
 
LVL 2

Author Comment

by:UnifiedIT
Comment Utility
This was never resolved and I will probably be switching to a Cisco PIX.. Thanks to those that helped..
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Suggested Solutions

Do you have a windows based Checkpoint SmartCenter for centralized Checkpoint management?  Have you ever backed up the firewall policy residing on the SmartCenter?  If you have then you know the hassles of connecting to the server, doing an upgrade_…
The DROP (Spamhaus Don't Route Or Peer List) is a small list of IP address ranges that have been stolen or hijacked from their rightful owners. The DROP list is not a DNS based list.  It is designed to be downloaded as a file, with primary intention…
Internet Business Fax to Email Made Easy - With eFax Corporate (http://www.enterprise.efax.com), you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, fr…
This video demonstrates how to create an example email signature rule for a department in a company using CodeTwo Exchange Rules. The signature will be inserted beneath users' latest emails in conversations and will be displayed in users' Sent Items…

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now