?
Solved

VPN... HASH, NOTIFY:PAYLOAD_MALFORMED!!!

Posted on 2003-10-27
13
Medium Priority
?
6,148 Views
Last Modified: 2013-11-16
Hello Experts, here is my situtation:
We are using a SuperStack3 3com firewall, and we are using its as a primary point for VPN clients to access our network.

I have successfully set up VPN software clients to access this, and the 3Com VPN/Firewall can access it fine as well. When attempting to use a Linksys BESFX41 as a VPN client, I am getting the following error in our 3Com FW logs..

I have tried using different shared secrets, and tried different combos for the encryption and authentication, but it always seems to fail at HASH, NOTIFY:PAYLOAD_MALFORMED.

Any help would be greatly appriciated.

The IP addresses have been removed to protect the innocent :)

10/27/2003 16:10:03.416 SENDING>>>> ISAKMP OAK MM (MsgID: 0x0) (SA)    
10/27/2003 16:10:03.944 RECEIVED<<< ISAKMP OAK MM (MsgID: 0x0) (KE, NON)
10/27/2003 16:10:04.096 NAT Discovery : Peer IPSec Security Gateway doesn't support VPN NAT Traversal (I dont think this is related, but not positive?)    
10/27/2003 16:10:04.096 SENDING>>>> ISAKMP OAK MM (MsgID: 0x0) (KE, NON, VID, VID, VID)  
10/27/2003 16:10:04.880 RECEIVED<<< ISAKMP OAK MM (MsgID: 0x0) *(ID)
10/27/2003 16:10:04.880 SENDING>>>> ISAKMP OAK INFO (MsgID: 0x1426AB29) *(HASH, NOTIFY:PAYLOAD_MALFORMED)
10/27/2003 16:10:12.880 IKE Responder: No response - remote party timeout or SA mis-match

Thanks,

Mike
0
Comment
Question by:UnifiedIT
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 7
  • 6
13 Comments
 
LVL 2

Expert Comment

by:sh00t3r
ID: 9630701
YO Mike,

A couple things...

You'll probably have to open up the port AND port triggering in the linksys routers....

Use the ISAKMP port, 500. Then make sure the Linksys router supports IPSEC. That should do it.

Sh00t3r

0
 
LVL 2

Author Comment

by:UnifiedIT
ID: 9633342
Hey sh00t3r..

I just want to make sure we are on the same page here.

The linksys router is the VPN client. on its main page it has a place to test the connection.

Now opening up ports and using port triggering would be for cpu's on my LAN, but I am just trying to create the tunnel between the Linksys and the 3Com right now..

Thanks again..
0
 
LVL 2

Expert Comment

by:sh00t3r
ID: 9634117
So is the linksys inside your network?? Or is it a site to site VPN?
0
Get real performance insights from real users

Key features:
- Total Pages Views and Load times
- Top Pages Viewed and Load Times
- Real Time Site Page Build Performance
- Users’ Browser and Platform Performance
- Geographic User Breakdown
- And more

 
LVL 2

Author Comment

by:UnifiedIT
ID: 9634310
The 3 Com is our corporate FW and main access point to our network.

The Linksys is a VPN router located at my home.
I was using a Linksys router and passing IPSec through so I could use the software client on my PC, but I decided to go with a Hardware VPN/FW for the added FW security.

I cannot get the linksys to open the VPN tunnel.

The logs from above were on the corporate FW...
0
 
LVL 2

Expert Comment

by:sh00t3r
ID: 9634398
You should try opening up the ports and port triggering as stated above. Then use the software client on your PC. As long as you don't open up anymore ports there won't be too much additional protection that the VPN/FW linksys will do as opposed to just running the linksys as a router. I believe it does stateful inspection of packets but it will continue to do this if you use the suggested setup.

0
 
LVL 2

Author Comment

by:UnifiedIT
ID: 9634458
Thanks for your help sh00t3r, but this defeats the purpose for my question.

I can make the software client work.

What I want is to get this working and I should beable to get the hardware VPN connected. I just have run into this snag and I see no reason why we cant get around it and make the harware client work.

Any other ideas would be greatly appriciated.

Thank you

Mike
0
 
LVL 2

Expert Comment

by:sh00t3r
ID: 9634523
Do you have PPTP enabled on the Linksys router?
0
 
LVL 2

Accepted Solution

by:
sh00t3r earned 750 total points
ID: 9634558
Also upgrade the firmware. Here are some links below that my also provide additional help.

http://www.homenethelp.com/vpn/router-linksys.asp


You may still need to open up those ports I specified even if using the linksys as the VPN client...
http://kb.linksys.com/cgi-bin/om_isapi.dll?clientID=884172&QuestionText=vpn&SelectName1=&advquery=%5bs%5d%5bRank%2c%2050%3a%5bSum%3a%20vpn%5d%5bMerge%3a%20%5bThesaurus%3a%20vpn%5d%5d%5d&infobase=linksysrev.nfo&record={3F0}&softpage=IKW_ENU_JDocView


0
 
LVL 2

Author Comment

by:UnifiedIT
ID: 9634601
Ok thanks,

Ill look into it further... I did upgrade the Firmware first thing, and I believe that PPTP is turned off.

I will check out the links and get back to you.

Mike
0
 
LVL 2

Author Comment

by:UnifiedIT
ID: 9675421
I cant get this thing to work, it appears that the IPSec that the Linksys uses is different then the IPSec that everyone else uses.. I have 3Com working on this with me.

As far as opening up ports on the router, that is a good idea, but it defeats the purpose of me purchasing this Hardware. I was looking for an extra layer of support, and the L2TP XP client hats the NATed FW, and the 3Com client doesnt provide XAUTH or L2TP authentication..

Basically, I am still looking for that extra level of authentication, and it is not working with my combo of products!

Thank you for your help thus far.
0
 
LVL 2

Expert Comment

by:sh00t3r
ID: 9675459
Well no resolution thus far. I don't see how the Linksys would use a different version of IPSEC. IPSEC is IPSEC. Good luck though and let us know how far you get so we can help you with any other issues.

Sh00t3r
0
 
LVL 2

Author Comment

by:UnifiedIT
ID: 9781240
Thanks

I have 3com working on it now... Ill let you know the outcome..
From what I understand, the techs at 3com told me that they believe that Linksys uses a proprietary IPSEC protocol. Im not sure if this is true, but I will keep you updated..

Thanks again,

Mike
0
 
LVL 2

Author Comment

by:UnifiedIT
ID: 10342658
This was never resolved and I will probably be switching to a Cisco PIX.. Thanks to those that helped..
0

Featured Post

Optimize your web performance

What's in the eBook?
- Full list of reasons for poor performance
- Ultimate measures to speed things up
- Primary web monitoring types
- KPIs you should be monitoring in order to increase your ROI

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

To setup a SonicWALL for policy based routing to be used with the Websense Content Gateway there are several steps that need to be completed. Below is a rough guide for accomplishing this. One thing of note is this guide is intended to assist in the…
This article offers some helpful and general tips for safe browsing and online shopping. It offers simple and manageable procedures that help to ensure the safety of one's personal information and the security of any devices.
Have you created a query with information for a calendar? ... and then, abra-cadabra, the calendar is done?! I am going to show you how to make that happen. Visualize your data!  ... really see it To use the code to create a calendar from a q…
In this video, Percona Solution Engineer Rick Golba discuss how (and why) you implement high availability in a database environment. To discuss how Percona Consulting can help with your design and architecture needs for your database and infrastr…
Suggested Courses
Course of the Month10 days, 16 hours left to enroll

770 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question