Stumped troubleshooting ipsec vpn
Posted on 2014-01-27
I'm stumped troubleshooting this vpn connection.
Scope of this question is to get ping replies working end to end across an ipsec vpn tunnel (site to site). currently, the tunnel connects, i can send a ping thru the tunnel from building1 to the datacenter, the ping is received at the datacenter, the datacenter host replies, but the reply never gets back to the building1 side that initiated the ping.
cisco rvs4000 vpn router
wan static ip, we'll call it a.b.c.d
upstream gateway a.b.c.z (cox cable)
local address a.b.c.d
local group subnet 10.100.1.0 / 24
remote group ip z.x.c.v (ie: public ip of datacenter)
remote group subnet: 172.16.1.0 / 24
keying mode: ike with preshared key
phase 1 encryption: 3des
group: 1024bit (ie group2)
key life 28800 sec
phase 2 encryption 3des
preshared key: password (or whatever, it matches the remote endpoint)
key lifetime 28800
paloalto pan-4050 router
wan: vpn endpoint z.x.c.v (public ip)
upstream gateway: z.x.c.z (datacenter core switch, expedient colo)
vpn setup identical to above
the vpn tunnel DOES connect, shows "up"
if i initiate a ping from 10.100.1.5 to 172.16.1.70... with wireshark running on both machines:
10.100.1.5 sends the packet to 10.100.1.254 (the cisco)
z.x.c.v (datacenter wan) receives the encapsulated packet and decrypts it, routes it to 172.16.1.70
on 172.16.1.70, wireshark sees the ping from 10.100.1.5
172.16.1.70 replies to 10.100.1.5
172.16.1.65 (paloalto inside interface) receives it and encapsulates it for a.b.c.d (building1 wan)
z.x.c.v does forward it to z.x.c.z (upstream device) as seen by port-mirroring the wan uplink
it never arrives at a.b.c.d (building1 wan, cisco rvs router)
if i initiate a ping from 172.16.1.70 destined for 10.100.1.5:
the intside interface of the paloalto receives it, packs it up and forwards it to the upstream gateway (as seen on the wire, port mirroring the wan uplink).
no traffic is received at building1
i've had paloalto support in their device for a week, they've proved beyond all doubt that the traffic is being handled properly and being passed upstream correctly
i've replaced the router at building1 (changed from a netgear vpn router, to a cisco vpn router). both the netgear, and the cisco, have the identical symptoms. tunnel connects, traffic gets from building1 to the datacenter, but not back.
when i traceroute from building1 (10.100.1.5) to google (18.104.22.168) my first hop is as expected my internal gateway (10.100.1.254, the cisco). but, the very next hop is 10.16.72.1 (14ms, assume not my cable modem). the next hop after that is NOT a.b.c.z (upstream public gateway), it is something completely different (but still on cox network)
i've tried asking cox what the heck is 10.16.72.1 and to check my cable modem routing table to make sure it's correct... but the best they could do for me is tell me to reboot my cable modem and router.
the physical wan port of the cisco at building1, is directly connected to the one and only ethernet port on the cable modem. nothing is in between.
so, i need ideas as to why the return traffic can't get back.