• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 6205
  • Last Modified:

VPN... HASH, NOTIFY:PAYLOAD_MALFORMED!!!

Hello Experts, here is my situtation:
We are using a SuperStack3 3com firewall, and we are using its as a primary point for VPN clients to access our network.

I have successfully set up VPN software clients to access this, and the 3Com VPN/Firewall can access it fine as well. When attempting to use a Linksys BESFX41 as a VPN client, I am getting the following error in our 3Com FW logs..

I have tried using different shared secrets, and tried different combos for the encryption and authentication, but it always seems to fail at HASH, NOTIFY:PAYLOAD_MALFORMED.

Any help would be greatly appriciated.

The IP addresses have been removed to protect the innocent :)

10/27/2003 16:10:03.416 SENDING>>>> ISAKMP OAK MM (MsgID: 0x0) (SA)    
10/27/2003 16:10:03.944 RECEIVED<<< ISAKMP OAK MM (MsgID: 0x0) (KE, NON)
10/27/2003 16:10:04.096 NAT Discovery : Peer IPSec Security Gateway doesn't support VPN NAT Traversal (I dont think this is related, but not positive?)    
10/27/2003 16:10:04.096 SENDING>>>> ISAKMP OAK MM (MsgID: 0x0) (KE, NON, VID, VID, VID)  
10/27/2003 16:10:04.880 RECEIVED<<< ISAKMP OAK MM (MsgID: 0x0) *(ID)
10/27/2003 16:10:04.880 SENDING>>>> ISAKMP OAK INFO (MsgID: 0x1426AB29) *(HASH, NOTIFY:PAYLOAD_MALFORMED)
10/27/2003 16:10:12.880 IKE Responder: No response - remote party timeout or SA mis-match

Thanks,

Mike
0
UnifiedIT
Asked:
UnifiedIT
  • 7
  • 6
1 Solution
 
sh00t3rCommented:
YO Mike,

A couple things...

You'll probably have to open up the port AND port triggering in the linksys routers....

Use the ISAKMP port, 500. Then make sure the Linksys router supports IPSEC. That should do it.

Sh00t3r

0
 
UnifiedITAuthor Commented:
Hey sh00t3r..

I just want to make sure we are on the same page here.

The linksys router is the VPN client. on its main page it has a place to test the connection.

Now opening up ports and using port triggering would be for cpu's on my LAN, but I am just trying to create the tunnel between the Linksys and the 3Com right now..

Thanks again..
0
 
sh00t3rCommented:
So is the linksys inside your network?? Or is it a site to site VPN?
0
KuppingerCole Reviews AlgoSec in Executive Report

Leading analyst firm, KuppingerCole reviews AlgoSec's Security Policy Management Solution, and the security challenges faced by companies today in their Executive View report.

 
UnifiedITAuthor Commented:
The 3 Com is our corporate FW and main access point to our network.

The Linksys is a VPN router located at my home.
I was using a Linksys router and passing IPSec through so I could use the software client on my PC, but I decided to go with a Hardware VPN/FW for the added FW security.

I cannot get the linksys to open the VPN tunnel.

The logs from above were on the corporate FW...
0
 
sh00t3rCommented:
You should try opening up the ports and port triggering as stated above. Then use the software client on your PC. As long as you don't open up anymore ports there won't be too much additional protection that the VPN/FW linksys will do as opposed to just running the linksys as a router. I believe it does stateful inspection of packets but it will continue to do this if you use the suggested setup.

0
 
UnifiedITAuthor Commented:
Thanks for your help sh00t3r, but this defeats the purpose for my question.

I can make the software client work.

What I want is to get this working and I should beable to get the hardware VPN connected. I just have run into this snag and I see no reason why we cant get around it and make the harware client work.

Any other ideas would be greatly appriciated.

Thank you

Mike
0
 
sh00t3rCommented:
Do you have PPTP enabled on the Linksys router?
0
 
sh00t3rCommented:
Also upgrade the firmware. Here are some links below that my also provide additional help.

http://www.homenethelp.com/vpn/router-linksys.asp


You may still need to open up those ports I specified even if using the linksys as the VPN client...
http://kb.linksys.com/cgi-bin/om_isapi.dll?clientID=884172&QuestionText=vpn&SelectName1=&advquery=%5bs%5d%5bRank%2c%2050%3a%5bSum%3a%20vpn%5d%5bMerge%3a%20%5bThesaurus%3a%20vpn%5d%5d%5d&infobase=linksysrev.nfo&record={3F0}&softpage=IKW_ENU_JDocView


0
 
UnifiedITAuthor Commented:
Ok thanks,

Ill look into it further... I did upgrade the Firmware first thing, and I believe that PPTP is turned off.

I will check out the links and get back to you.

Mike
0
 
UnifiedITAuthor Commented:
I cant get this thing to work, it appears that the IPSec that the Linksys uses is different then the IPSec that everyone else uses.. I have 3Com working on this with me.

As far as opening up ports on the router, that is a good idea, but it defeats the purpose of me purchasing this Hardware. I was looking for an extra layer of support, and the L2TP XP client hats the NATed FW, and the 3Com client doesnt provide XAUTH or L2TP authentication..

Basically, I am still looking for that extra level of authentication, and it is not working with my combo of products!

Thank you for your help thus far.
0
 
sh00t3rCommented:
Well no resolution thus far. I don't see how the Linksys would use a different version of IPSEC. IPSEC is IPSEC. Good luck though and let us know how far you get so we can help you with any other issues.

Sh00t3r
0
 
UnifiedITAuthor Commented:
Thanks

I have 3com working on it now... Ill let you know the outcome..
From what I understand, the techs at 3com told me that they believe that Linksys uses a proprietary IPSEC protocol. Im not sure if this is true, but I will keep you updated..

Thanks again,

Mike
0
 
UnifiedITAuthor Commented:
This was never resolved and I will probably be switching to a Cisco PIX.. Thanks to those that helped..
0

Featured Post

Evaluating UTMs? Here's what you need to know!

Evaluating a UTM appliance and vendor can prove to be an overwhelming exercise.  How can you make sure that you're getting the security that your organization needs without breaking the bank? Check out our UTM Buyer's Guide for more information on what you should be looking for!

  • 7
  • 6
Tackle projects and never again get stuck behind a technical roadblock.
Join Now