Stumped troubleshooting ipsec vpn

Posted on 2014-01-27
Last Modified: 2014-01-28
I'm stumped troubleshooting this vpn connection.

Scope of this question is to get ping replies working end to end across an ipsec vpn tunnel (site to site).  currently, the tunnel connects, i can send a ping thru the tunnel from building1 to the datacenter, the ping is received at the datacenter, the datacenter host replies, but the reply never gets back to the building1 side that initiated the ping.

the setup:

cisco rvs4000 vpn router
wan static ip, we'll call it a.b.c.d
upstream gateway a.b.c.z  (cox cable)

vpn setup:
ipsec tunnel
local address a.b.c.d
local group subnet / 24
remote group ip z.x.c.v  (ie: public ip of datacenter)
remote group subnet: / 24
keying mode: ike with preshared key
phase 1 encryption: 3des
auth: md5
group: 1024bit (ie group2)
key life 28800 sec

phase 2 encryption 3des
auth md5
PFS disable
preshared key: password (or whatever, it matches the remote endpoint)
group: 1024bit
key lifetime 28800

datacenter setup:
paloalto pan-4050 router
wan:  vpn endpoint z.x.c.v (public ip)
upstream gateway:  z.x.c.z (datacenter core switch, expedient colo)
vpn setup identical to above

the vpn tunnel DOES connect, shows "up"

if i initiate a ping from to with wireshark running on both machines: sends the packet to (the cisco)
z.x.c.v (datacenter wan) receives the encapsulated packet and decrypts it, routes it to
on, wireshark sees the ping from replies to (paloalto inside interface) receives it and encapsulates it for a.b.c.d (building1 wan)
z.x.c.v does forward it to z.x.c.z (upstream device) as seen by port-mirroring the wan uplink
it never arrives at a.b.c.d (building1 wan, cisco rvs router)

if i initiate a ping from destined for
the intside interface of the paloalto receives it, packs it up and forwards it to the upstream gateway (as seen on the wire, port mirroring the wan uplink).
no traffic is received at building1


i've had paloalto support in their device for a week, they've proved beyond all doubt that the traffic is being handled properly and being passed upstream correctly

i've replaced the router at building1 (changed from a netgear vpn router, to a cisco vpn router).  both the netgear, and the cisco, have the identical symptoms.  tunnel connects, traffic gets from building1 to the datacenter, but not back.

interesting point:
when i traceroute from building1 ( to google ( my first hop is as expected my internal gateway (, the cisco).  but, the very next hop is (14ms, assume not my cable modem).  the next hop after that is NOT a.b.c.z (upstream public gateway), it is something completely different (but still on cox network)

i've tried asking cox what the heck is and to check my cable modem routing table to make sure it's correct... but the best they could do for me is tell me to reboot my cable modem and router.

the physical wan port of the cisco at building1, is directly connected to the one and only ethernet port on the cable modem.  nothing is in between.

so, i need ideas as to why the return traffic can't get back.
Question by:FocIS
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 4
LVL 69

Expert Comment

ID: 39813150
Compare the traceroute to each other public IPs. And how do you know a.b.c.d does not get traffic back? Did you use a WAN link mirror with WireShark here, too?

And make sure the paloalto device does not try to create another tunnel because of some parameter mismatch (local and remote subnets in VPN, for example).

Author Comment

ID: 39813405
Thanks for the reply!
In the paloalto, the debug logs are very explicit, i'm certain it sends it out the correct tunnel.  The port mirror on the paloalto side does show esp packets going to the building1 wan

I can't port-mirror (yet) at the building1 side though i should be able to tomorrow.  

The symptoms are, as seen from, the pings are sent and sent and sent with no replies.

Similarly, as seen from, those packets are sent and sent and sent and never received replies, BUT i see matching pings on the destination of   so i see the ping hit the destination, the destination replies, but the replies never hit back to

the traceroutes are similar but different:

from building1 to datacenter:
Tracing route to z.x.c.v over a maximum of 30 hops

  1     3 ms     1 ms     1 ms
  2    17 ms    11 ms    13 ms  <-- not sure what this is
  3    10 ms     9 ms     9 ms [] <-- not our ip or upstream gw
  4    12 ms    10 ms    11 ms []
  5    65 ms    38 ms    49 ms
  6   109 ms   207 ms   222 ms []
  7    65 ms   107 ms    75 ms []
  8    71 ms    67 ms    68 ms []
  9    78 ms    72 ms    73 ms []
 10    74 ms    69 ms    76 ms []
 11    67 ms    74 ms    74 ms []
 12    72 ms    67 ms    70 ms  z.x.c.v
Trace complete.

from datacenter:
Tracing route to a.b.c.d [68.99.x.x]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  paloalto []
  2     1 ms    <1 ms    <1 ms  z.x.c.z  (upstream gateway)
  3    11 ms    11 ms    11 ms []
  4    11 ms    11 ms    11 ms []
  5    11 ms    11 ms    11 ms []
  6    11 ms    11 ms    11 ms []
  7    11 ms    11 ms    11 ms []
  8    31 ms    31 ms    31 ms []
  9    57 ms    76 ms   114 ms []
 10    57 ms    57 ms    57 ms []
 11    57 ms    57 ms    56 ms []
 12    70 ms    68 ms    73 ms  a.b.c.d [our static ip 68.99.x.x]
Trace complete.
LVL 69

Expert Comment

ID: 39813761
Ok, that tells us the packets should flow between both routers. ESP packets might get filtered on their way back, though. Really difficult to tell. Also, there could be a MTU mismatch leading to excess fragmentation - or requiring fragmentation, but not doing that. Depending on firmware bugs this might be an issue if the correct MTU is only a few bytes different (we often see issues with 1500 instead of 1492 bytes to use).
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!


Author Comment

ID: 39813974
Good catch qlemo, i was in a hurry and missed that ip address :)

The cisco at the building1 side was already 1492 mtu, but the datacenter side was 1500, so i just changed that to 1492.

having saved the changes, the tunnel remains fully connected but the esp packets of the "ping reply" don't appear to reach back to building1.

when we had a netgear vpn router at building1 last week, it has a built in packet scanner with download to wireshark for review - we could see the ping request leaving, but never saw the ping reply come back.  

i'll hook up a port mirror device at building1 on tuesday and see what can be seen (still have one in place at the datacenter)

happy to try any other ideas at all, and to provide more info if it helps

Author Comment

ID: 39813978
i wanted to mention some more aspects:

when i initiate a ping from directly to, the route is there, the datacenter router passes esp packets upstream (that's as far as we can track it) but the "ping request" never makes it to (as viewed on the nic of

also, we have an identical cisco rvs4000 router on another cox cable modem in the building (different account, different static ip) with identical vpn settings (tailored to the different static cox ip address), and THAT tunnel connects, and pings pass in both directions

i've requested that the datacenter isp capture our packets on our upstream gateway but their answer was (rightfully so) "no way, that's a core switch" - all i can assume is since the packets left our rack in good health, they should be leaving the building too.

what do you think about the "odd" 2nd hop leaving building1?  is that some sort of internal vpn between cleveland and rhode island ( gets from cleveland to RI for the cox pop).  i wonder if it is a cox tunnel, if that's double/tripple wrapping the packets and killing the crypto (yet it works on the way out from building1 to the datacenter)
LVL 69

Accepted Solution

Qlemo earned 500 total points
ID: 39814416
I don't think the 2nd hop is wrapping, just routing. If it did anything, the tunnel would not come up.
Regarding the Netgear packet capture - did you see both unencrpyted and encrypted traffic, or only the unencrypted one? Because I still think something with the VPN settings is not correct. Maybe the other tunnel is getting the reply traffic in error?

Author Comment

ID: 39815197
Good thought with the two tunnels - i've just deleted the second tunnel (which worked, but with the "wrong" network).

I'll post some sanitized screenshots of the settings here

Author Closing Comment

ID: 39815220
oh wow, i think that actually did it - there was some confusion between "tunnel.2" and "tunnel.3" - when i went to delete tunnel.3 at your suggestion, i noticed the wrong private ip block in tunnel.2

pings in both directions are finally replying all the way thru the tunnel now!
LVL 69

Expert Comment

ID: 39815364
Great! The private networks exchanged in IPSec are often used to map traffic to tunnel. Though that should be not necessary for reply traffic - a stateful firewall should bypass any rules, as long as the corresponding session exists.

Featured Post

On Demand Webinar - Networking for the Cloud Era

This webinar discusses:
-Common barriers companies experience when moving to the cloud
-How SD-WAN changes the way we look at networks
-Best practices customers should employ moving forward with cloud migration
-What happens behind the scenes of SteelConnect’s one-click button

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Problem Description:   Couple of months ago we upgraded the ADSL line at our branch office from Home to Business line. The purpose of transforming the service to have static public IP’s. We were in need for public IP’s to publish our web resour…
Tired of waiting for your show or movie to load?  Are buffering issues a constant problem with your internet connection?  Check this article out to see if these simple adjustments are the solution for you.
After creating this article (, I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
Windows 10 is mostly good. However the one thing that annoys me is how many clicks you have to do to dial a VPN connection. You have to go to settings from the start menu, (2 clicks), Network and Internet (1 click), Click VPN (another click) then fi…

726 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question