I am getting desperate. I am unable to complete Phase I on a site-to-site IPSec VPN between an ISA 2004 and a 3Com CR870 router (the 3Com connects well with another 3Com using IPSec). When I initiate the connection from the 3Com, the ISA logging files shows an "IKE Client" and "Initiated Connection" with the Rule being "Allow VPN Client traffic to ISA Server". If I look at the 3Com log, it show a Phase I attempt and is waiting for a response and finally aborts after approximately 30 seconds. Ethereal network software show a single packet arriving at the ISA Server destined for port 500. ISA Server immediately responds with an ICMP message to the 3Com "Destination unreachable (port unreachable)". The ISA 2004 shows a VPN session with the 3Com under Sessions. Phase I never completes and I am at a loss to understand the ICMP message. Port software shows UDP Port 500 open with "UDP port 500 (isakmp service): LISTENING". ISA 2004 closes the Session after 90 seconds.
It appears that ISA 2004 receives the request but never responds. I checked to ensure that the 3Com had port 500 open, which it did.
In ISA 2004 firewall rules, I have all protocols open for the "Remote Site" firewall access rule. I have and inbound access rule and and an outbound access rule (inbound from the "Remote Site" and outbound to the "Remote Site". "Remote Site" being the network object identifying the remote site.
I am curious why the rule for the initial connection would be the VPN Client rule. I just turned of VPN Client Access and the rule "Allow VPN Site-to-Site traffic to ISA Server" appeared instead when I attempted another connection. But the end result was the same.
It appears that ISA 2004 receives the Phase I connection but never sends a response to the 3Com. It instead waits for a response from the 3Com and finally times out (which the 3Com also does). The session appears under Sessions and times out after 90 seconds.
The 3Com does not have different settings for Phase I and Phase II (just one group of settings). I select 3DES, SHA1, Diffie-Hellman Group 1 (768), Renegotiate after 28800 seconds and a pre-shared key. I have turned PFS (Perfect Forward Secrecy) both on and off for testing. I assume the 3DES, SHA1 and 28800 apply for both Phase I and Phase II.
It is an Edge firewall configuration. Internal addresses are 192.168.168.0 - 192.168.168.255. Address of remote site is 192.168.164.0 - 192.168.164.255.