• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1182
  • Last Modified:

Stumped troubleshooting ipsec vpn

I'm stumped troubleshooting this vpn connection.

Scope of this question is to get ping replies working end to end across an ipsec vpn tunnel (site to site).  currently, the tunnel connects, i can send a ping thru the tunnel from building1 to the datacenter, the ping is received at the datacenter, the datacenter host replies, but the reply never gets back to the building1 side that initiated the ping.

the setup:

cisco rvs4000 vpn router
wan static ip, we'll call it a.b.c.d
upstream gateway a.b.c.z  (cox cable)

vpn setup:
ipsec tunnel
local address a.b.c.d
local group subnet / 24
remote group ip z.x.c.v  (ie: public ip of datacenter)
remote group subnet: / 24
keying mode: ike with preshared key
phase 1 encryption: 3des
auth: md5
group: 1024bit (ie group2)
key life 28800 sec

phase 2 encryption 3des
auth md5
PFS disable
preshared key: password (or whatever, it matches the remote endpoint)
group: 1024bit
key lifetime 28800

datacenter setup:
paloalto pan-4050 router
wan:  vpn endpoint z.x.c.v (public ip)
upstream gateway:  z.x.c.z (datacenter core switch, expedient colo)
vpn setup identical to above

the vpn tunnel DOES connect, shows "up"

if i initiate a ping from to with wireshark running on both machines: sends the packet to (the cisco)
z.x.c.v (datacenter wan) receives the encapsulated packet and decrypts it, routes it to
on, wireshark sees the ping from replies to (paloalto inside interface) receives it and encapsulates it for a.b.c.d (building1 wan)
z.x.c.v does forward it to z.x.c.z (upstream device) as seen by port-mirroring the wan uplink
it never arrives at a.b.c.d (building1 wan, cisco rvs router)

if i initiate a ping from destined for
the intside interface of the paloalto receives it, packs it up and forwards it to the upstream gateway (as seen on the wire, port mirroring the wan uplink).
no traffic is received at building1


i've had paloalto support in their device for a week, they've proved beyond all doubt that the traffic is being handled properly and being passed upstream correctly

i've replaced the router at building1 (changed from a netgear vpn router, to a cisco vpn router).  both the netgear, and the cisco, have the identical symptoms.  tunnel connects, traffic gets from building1 to the datacenter, but not back.

interesting point:
when i traceroute from building1 ( to google ( my first hop is as expected my internal gateway (, the cisco).  but, the very next hop is (14ms, assume not my cable modem).  the next hop after that is NOT a.b.c.z (upstream public gateway), it is something completely different (but still on cox network)

i've tried asking cox what the heck is and to check my cable modem routing table to make sure it's correct... but the best they could do for me is tell me to reboot my cable modem and router.

the physical wan port of the cisco at building1, is directly connected to the one and only ethernet port on the cable modem.  nothing is in between.

so, i need ideas as to why the return traffic can't get back.
  • 5
  • 4
1 Solution
QlemoBatchelor, Developer and EE Topic AdvisorCommented:
Compare the traceroute to each other public IPs. And how do you know a.b.c.d does not get traffic back? Did you use a WAN link mirror with WireShark here, too?

And make sure the paloalto device does not try to create another tunnel because of some parameter mismatch (local and remote subnets in VPN, for example).
FocISAuthor Commented:
Thanks for the reply!
In the paloalto, the debug logs are very explicit, i'm certain it sends it out the correct tunnel.  The port mirror on the paloalto side does show esp packets going to the building1 wan

I can't port-mirror (yet) at the building1 side though i should be able to tomorrow.  

The symptoms are, as seen from, the pings are sent and sent and sent with no replies.

Similarly, as seen from, those packets are sent and sent and sent and never received replies, BUT i see matching pings on the destination of   so i see the ping hit the destination, the destination replies, but the replies never hit back to

the traceroutes are similar but different:

from building1 to datacenter:
Tracing route to z.x.c.v over a maximum of 30 hops

  1     3 ms     1 ms     1 ms
  2    17 ms    11 ms    13 ms  <-- not sure what this is
  3    10 ms     9 ms     9 ms  ip98-173-132-214.cl.ri.cox.net [] <-- not our ip or upstream gw
  4    12 ms    10 ms    11 ms  ip98-173-132-222.cl.ri.cox.net []
  5    65 ms    38 ms    49 ms
  6   109 ms   207 ms   222 ms  te6-3.ar3.DCA3.gblx.net []
  7    65 ms   107 ms    75 ms  CONTINENTAL-BROADBAND.Te6-2.ar5.CHI1.gblx.net []
  8    71 ms    67 ms    68 ms  te1-2.4006.cr2.350ec.chcgil.e-xpedient.com []
  9    78 ms    72 ms    73 ms  te1-4.4005.cr1.strlng.clevoh.e-xpedient.com []
 10    74 ms    69 ms    76 ms  te1-2.4002.cr2.strlng.clevoh.e-xpedient.com []
 11    67 ms    74 ms    74 ms  te2-7-1.4007.151-core.expedient.com []
 12    72 ms    67 ms    70 ms  z.x.c.v
Trace complete.

from datacenter:
Tracing route to a.b.c.d [68.99.x.x]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  paloalto []
  2     1 ms    <1 ms    <1 ms  z.x.c.z  (upstream gateway)
  3    11 ms    11 ms    11 ms  te1-3.4007.cr2.strlng.clevoh.e-xpedient.com []
  4    11 ms    11 ms    11 ms  te1-2.4002.cr1.strlng.clevoh.e-xpedient.com []
  5    11 ms    11 ms    11 ms  te1-4.4005.cr2.350ec.chcgil.e-xpedient.com []
  6    11 ms    11 ms    11 ms  te1-2.4006.cr1.350ec.chcgil.e-xpedient.com []
  7    11 ms    11 ms    11 ms  te6-2.ar5.chi1.gblx.net []
  8    31 ms    31 ms    31 ms  cox-com.ethernet15-2.ar6.dal2.gblx.net []
  9    57 ms    76 ms   114 ms  clvdhdrj01-xe000.0.rd.cl.cox.net []
 10    57 ms    57 ms    57 ms  ip98-173-132-221.cl.ri.cox.net []
 11    57 ms    57 ms    56 ms  ip98-173-132-217.cl.ri.cox.net []
 12    70 ms    68 ms    73 ms  a.b.c.d [our static ip 68.99.x.x]
Trace complete.
QlemoBatchelor, Developer and EE Topic AdvisorCommented:
Ok, that tells us the packets should flow between both routers. ESP packets might get filtered on their way back, though. Really difficult to tell. Also, there could be a MTU mismatch leading to excess fragmentation - or requiring fragmentation, but not doing that. Depending on firmware bugs this might be an issue if the correct MTU is only a few bytes different (we often see issues with 1500 instead of 1492 bytes to use).
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

FocISAuthor Commented:
Good catch qlemo, i was in a hurry and missed that ip address :)

The cisco at the building1 side was already 1492 mtu, but the datacenter side was 1500, so i just changed that to 1492.

having saved the changes, the tunnel remains fully connected but the esp packets of the "ping reply" don't appear to reach back to building1.

when we had a netgear vpn router at building1 last week, it has a built in packet scanner with download to wireshark for review - we could see the ping request leaving, but never saw the ping reply come back.  

i'll hook up a port mirror device at building1 on tuesday and see what can be seen (still have one in place at the datacenter)

happy to try any other ideas at all, and to provide more info if it helps
FocISAuthor Commented:
i wanted to mention some more aspects:

when i initiate a ping from directly to, the route is there, the datacenter router passes esp packets upstream (that's as far as we can track it) but the "ping request" never makes it to (as viewed on the nic of

also, we have an identical cisco rvs4000 router on another cox cable modem in the building (different account, different static ip) with identical vpn settings (tailored to the different static cox ip address), and THAT tunnel connects, and pings pass in both directions

i've requested that the datacenter isp capture our packets on our upstream gateway but their answer was (rightfully so) "no way, that's a core switch" - all i can assume is since the packets left our rack in good health, they should be leaving the building too.

what do you think about the "odd" 2nd hop leaving building1?  is that some sort of internal vpn between cleveland and rhode island (cl.ri.cox.net gets from cleveland to RI for the cox pop).  i wonder if it is a cox tunnel, if that's double/tripple wrapping the packets and killing the crypto (yet it works on the way out from building1 to the datacenter)
QlemoBatchelor, Developer and EE Topic AdvisorCommented:
I don't think the 2nd hop is wrapping, just routing. If it did anything, the tunnel would not come up.
Regarding the Netgear packet capture - did you see both unencrpyted and encrypted traffic, or only the unencrypted one? Because I still think something with the VPN settings is not correct. Maybe the other tunnel is getting the reply traffic in error?
FocISAuthor Commented:
Good thought with the two tunnels - i've just deleted the second tunnel (which worked, but with the "wrong" network).

I'll post some sanitized screenshots of the settings here
FocISAuthor Commented:
oh wow, i think that actually did it - there was some confusion between "tunnel.2" and "tunnel.3" - when i went to delete tunnel.3 at your suggestion, i noticed the wrong private ip block in tunnel.2

pings in both directions are finally replying all the way thru the tunnel now!
QlemoBatchelor, Developer and EE Topic AdvisorCommented:
Great! The private networks exchanged in IPSec are often used to map traffic to tunnel. Though that should be not necessary for reply traffic - a stateful firewall should bypass any rules, as long as the corresponding session exists.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

  • 5
  • 4
Tackle projects and never again get stuck behind a technical roadblock.
Join Now