Solved

Cisco IPSec VPN Tunnel Traffic Routing

Posted on 2014-01-30
13
3,927 Views
Last Modified: 2014-01-31
configuration:

main site: Cisco ASA 5520
internal interface: 172.16.4.0/255.255.252.0
internal core switch/gateway IP: 172.16.4.1
external interface IP : 72.10.102.34

The core switch knows to send all internet-bound and VPN-bound traffic to the 'inside' interface that the ASA 5520 resides on.

remote site: Cisco 861 Router
internal interface: 172.16.54.0/255.255.255.192
external interface ip: 137.99.44.102

See attached running-config for more details.

History:

We recently needed to update our existing site-to-site VPN configuration to push ALL traffic from the remote through the tunnel, not just traffic bound for the main site.  We needed internet traffic from the remote site to flow through the web filtering appliance at our main site.

I was able to get traffic flowing through by removing all NATing on the remote site router, and configuring a default tunnel gateway on the main site's ASA.

Issue:

Traffic would get routed OUT to the internet properly, but the return traffic would not get pushed back through the tunnel.

Resolution:

I called Cisco and they had me add the following two static routes:

route outside 172.16.54.0 255.255.255.192 72.10.102.33 1
route inside 172.16.54.0 255.255.255.0 172.16.4.1 1

Open in new window


This solved the issue.

Question:

How did this solve the issue?  What is going on with packet routing that makes this work? To me, having a route that specifies to send packets destined for the remote site from the 'outside' interface back to the ISP gateway address wouldn't work...  

ASA Running config is attached.
0
Comment
Question by:rmeany
  • 7
  • 6
13 Comments
 

Author Comment

by:rmeany
Comment Utility
0
 
LVL 25

Expert Comment

by:Cyclops3590
Comment Utility
What that means is that for that subnet send the packets out the outside interface to that next hop IP.  The thing is though routing decisions are done before vpn decisions are done.  So that way the traffic goes in the right direction to then get encapsulated and go thru the vpn tunnel you have going with your remote site  otherwise it would have gone to the 4.1 device and the return traffic woudl go the wrong way and never get back to the original sender.
0
 

Author Comment

by:rmeany
Comment Utility
Ok.. I get the general idea.. but my understanding of

route inside 172.16.54.0 255.255.255.0 172.16.4.1

is as follows:

any packet arriving on the inside interface, with a destination address of 172.16.54.1-172.16.54.255, should get forwarded to a device with address 172.16.4.1 (our core switch/gateway)


and route outside 172.16.54.0 255.255.255.192 72.10.102.33

is as follows:

any packet arriving on the outside interface, with a destination address of 172.16.54.1-172.16.54.63, should get forwarded to a device with address 72.10.102.33 (isp)



Which doesn't make any sense to me... Im guessing I'm interpreting the language behind the static route wrong?
0
 
LVL 25

Expert Comment

by:Cyclops3590
Comment Utility
Ok, i see your misunderstanding.  the outside/inside isn't referring to how the packet was received.  It means out which interface should the ASA find the next hop; out which interface do the packets flow.  So for that destination, it should go out which interface to relay the packet to the next hop.  Personally I don't know why its there. I think its just a legacy thing really.
0
 

Author Comment

by:rmeany
Comment Utility
Ah, ok, that's good to know...

... But why would those routes be necessary?

As it is now, there are two rules for packets destined for 172.16.54.0  ...  Though one had been modified to define a different range so that the duplicate destination rule could be entered (ASA won't let you define two static routes for a given network).

How would ASA pick which route to use?  And why would sending a packet destined for 172.16.54.0 out the outside interface to the ISP address do anything useful, considering that 172.16.54.0 is an internal network that 72.10.102.33 knows nothing about....
0
 

Author Comment

by:rmeany
Comment Utility
I'm just going to note some observations here:

If I remove  route 172.16.54.0 255.255.255.0 172.16.4.1 , it does not cause any disruption in traffic between the main site and the remote site at all, but clients at the remote site can no longer access the internet.  If I look at the ASA logs, it reports that packets returning from internet sites cannot find a valid route back to the remote site.

If I run a packet-trace in CLI, it doesn't report any problems on a tcp packet destined for the remote site, and the route-lookup step says it is using the 54.0/27 to 72.10.102.33 route..

Is it something like the VPN requiring NATing to be done from the inside interface to the outside in order to encapsulate?


If I remove the route outside 172.16.54.0 255.255.255.192 72.10.102.33 but keep the other route, then the clients at the remote site still have internet access, but I cannot initiate connections to the remote site from the main site.  If I try to ping a device at the remote site from the main site, I get TTL expired messages.  I believe this is because the other route just keeps bouncing the packet back to our core switch, which in turn bounces back to the ASA until TTL expires....
0
Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 25

Expert Comment

by:Cyclops3590
Comment Utility
what does the core switch do that your ASA isn't your layer 3 device doing all of the routing?  can you upload a diagram of how your network looks?  I need to better understand what it is doing before I can, with certainty, explain what is going on.  

However, based on what you're describing it is doing something to the packets.  From what I can tell from what your describing though:

remove  route 172.16.54.0 255.255.255.0 172.16.4.1, you see main to remote work fine, no internet

My assumption is that because the packets flow thru the vpn tunnel because the destination packets for the remote site is forced directly out outside interface to be encrypted it works between the two sites.  However, it doesn't work to the internet because it is not getting NAT'd going out and thus dropped.  Routing only works on destination.  So if the public IP was being used it should go out just fine if there was NAT'ing and then return traffic would then get "unnatted" and should be sent via the tunnel back to the remote site.  So I'm thinking there is a discrepancy in what the ASA is doing and by forcing it to the core switch, the switch fills in the gap necessary to make the packet publicly routable.

remove the route outside 172.16.54.0 255.255.255.192 72.10.102.33, you see main to remote stop, internet works

Basically the reverse of what was mentioned above.  Removing this route makes it so packets destined for the remote site are not forwarded over the tunnel and thus never reach it.  However, there must be something going on with the core switch that makes it so the internet destined packets get translated or something so that it gets back to the remote site.


However, without knowing the config of the core config or having a diagram of how everything is hooked up, I'm really just guessing at what is going on.
0
 

Author Comment

by:rmeany
Comment Utility
The setup is pretty simple.

We have our core switch with the ASA plugged into one of the gigabit ethernet ports on it.  That port is mirrored to another port which is monitored by our filtering appliance.  The filtering appliance does some fancy work to inject block pages into the stream when it sees requests for pages that should be blocked go through the monitored port.

The core switch routes any traffic bound for networks outside through this port, so it passes through the ASA and then ASA passes traffic out to the ISP's router.
0
 

Author Comment

by:rmeany
Comment Utility
There is something going on with  route outside 172.16.54.0 255.255.255.192 72.10.102.33

That prevents the loop between the core and the ASA from occurring.  


Traffic that comes into the ASA finds

route outside 172.16.54.0 255.255.255.192 72.10.102.33 first,

*I think* because it defines a more specific range (fewer addresses) than

route inside 172.16.54.0 255.255.255.0 172.16.4.1


route outside 172.16.54.0 255.255.255.192 72.10.102.33 works to get traffic routed from our main site over to the remote site, and since it is the first rule picked, the traffic routes appropriately.  

However, something about the above route does not work for returning internet traffic.  The route doesn't come up as valid.  So when that route fails, it looks like it uses

route inside 172.16.54.0 255.255.255.0 172.16.4.1

to return the internet traffic back to the remote site.

The why of this is still confusing to me though...
0
 
LVL 25

Expert Comment

by:Cyclops3590
Comment Utility
can you post the routing table of the switch.
0
 
LVL 25

Expert Comment

by:Cyclops3590
Comment Utility
also, what did the route entries look like before you changed the remote site to tunnel everything?

Have you tried removing both of those static entries?  It has to do with having 2 default gateways.  One regular and one tunneled.  you're basically overriding.  But before I can say for sure how everything changed I need to know what it looked like before the changes.
0
 
LVL 25

Accepted Solution

by:
Cyclops3590 earned 500 total points
Comment Utility
Here's my current understanding anyway:

remote site to internet:
1) packet is sent from remote to main over vpn
2) as it is coming in via encapsulated tunnel it would use the tunneled default gateway and forward to the core switch
3) this switch then sends the traffic back to the ASA for it to be NAT'ed and sent out the outside interface
4) Return traffic comes in on the outside interface, unnats the now destination IP, looks up the route to forward and realizes it goes back out the outside interface but expects it to go out the inside interface, thus a corruption.

in this case, it would be better to perform a outside,outside nat for the remote site and use the regular default route

remote site to main site:
1) first 3 steps the same  
4) The ASA then encapsulates and sends out the tunneled interface.
The return traffic being encapsulated on arrival will always use the tunneled default gateway which relays to main site IP.  However if the specific route to go out the outside interface isn't there, you have a routing loop.

From what I can tell because you don't have an outside,outside nat setup to send the remote sites traffic directly out, it must go into the main site first (which it sounds like you want for filtering purposes) before it can be properly NAT'ed on the way out.  Return traffic then effectively has to come back into the site before sent back to the ASA to be properly tunneled to the remote site.  Having the /26 route in there just helps ensure there is no odd routing as well and allows for the main site to go direct.  However there is no nat'ing that takes place to mess things up.  That is why you need the /24 route for internet traffic.  It allows nat operations to take place as intended and packets to flow without issue.

hope that makes sense.
0
 

Author Closing Comment

by:rmeany
Comment Utility
Makes sense.  I think you nailed it for me with #4..  Since the traffic going out to the internet came FROM 172.16.4.1, the return traffic needs the route back to 4.1  .. The 102.33 route doesn't satisfy that requirement, so it fails, hence the need for the redirection to 4.1.  

The 102.33 route is necessary for internal traffic to make connections to the remote site, because the 4.1 route exists.  If the 4.1 route didn't exist, the default route of 102.33 would be used, which is correct for internal traffic going to remote site.  But since the 4.1 route is necessary for returning internet traffic, we need to tell the router to 'try' the 102.33 route FIRST for packets destined for the remote site.  Otherwise, the 4.1 route will cause a loop for internal to remote site connections.  Returning Internet traffic will then fail on that route and the router will then try, and succeed, on the 4.1 route.

And yes, an (outside,outside) nat would make sense to prevent all this.. But the reason we're not just natting straight out thru the remote site's router is because we need the traffic to come back and go through our web filter, which means we need it to go through our core switch, hence the complication of this whole setup.

Makes perfect sense now, thanks for taking the time to answer this "academic" question :)
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Suggested Solutions

Every server (virtual or physical) needs a console: and the console can be provided through hardware directly connected, software for remote connections, local connections, through a KVM, etc. This document explains the different types of consol…
Quality of Service (QoS) options are nearly endless when it comes to networks today. This article is merely one example of how it can be handled in a hub-n-spoke design using a 3-tier configuration.
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

7 Experts available now in Live!

Get 1:1 Help Now