?
Solved

Cisco ASA VPN - stateful TCP connections fail

Posted on 2009-12-21
20
Medium Priority
?
1,850 Views
Last Modified: 2012-05-08
Recently built a site-to-site VPN with a business partner through my Cisco ASA 5510 (running 8.x). The biz partner is providing services to us, so traffic is essentially one-way; it will always be initiated by us.

Using ASDM, I created the VPN using their endpoint and the previously-agreed upon parameters (protected networks, etc etc etc). The tunnel comes up after the first packet.

I created a group policy that includes an extended access list that permits traffic from our network to theirs, only. For sake of argument, let's say that I am accessing a web server at their site on port 80.

TCP packets fail. That is, I get an error in the log ("Inbound TCP connection denied from [my IP address/highport] to [their IP address/80] flags SYN on interface Inside") and I receive a RST packet on the client side almost immediately.

If I modify the access list such that it explicitly permits the return traffic (i.e., any port with a 'source' of port 80), it works flawlessly (after tearing down the VPN).

Why is this? Either Cisco isn't doing something very basic (supporting stateful TCP across VPN connections) or I am missing something.
0
Comment
Question by:grabo
  • 8
  • 7
  • 3
18 Comments
 
LVL 32

Expert Comment

by:harbor235
ID: 26104875


Did you add traffic sourced from your network to the remote location to the NONAT rule?

harbor235 ;}
0
 

Author Comment

by:grabo
ID: 26104950
There is a NAT exclusion rule for (my network) -> (their network).
0
 
LVL 32

Expert Comment

by:harbor235
ID: 26105491


So, you must have an inbound ACL on the local inside interface, rememeber order of operations, in this case the ACL is applied first then encryption. So you will need to make sure the ACL on the inside interface accomodates teh traffic flow as well as teh outsid einterfaces.

harbor235 ;}
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 

Author Comment

by:grabo
ID: 26105707
The traffic is already permitted on the Inside interface, but for kicks, I added one specifically for this traffic. Still does not work.

Running through the packet tracer -- a few oddities that I do not understand.

1. It hits the NAT exemption (phase 7). Why does it also hit the other NAT rules in phase 8 and phase 9?
2. It looks like something's happening at Phase 11. What is it? That's a big clue, i think.

---

(10.0.1.224 = my machine; 90.103.14.116 = webserver)


fw#   packet-tracer input Inside tcp 10.0.1.224 9988 90.103.14.116 80 det

Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0xd7b7d8a8, priority=1, domain=permit, deny=false
        hits=775672275, user_data=0x0, cs_id=0x0, l3_type=0x8
        src mac=0000.0000.0000, mask=0000.0000.0000
        dst mac=0000.0000.0000, mask=0000.0000.0000

Phase: 2
Type: FLOW-LOOKUP
Subtype:
Result: ALLOW
Config:
Additional Information:
Found no matching flow, creating a new flow

Phase: 3
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   0.0.0.0         0.0.0.0         Outside

Phase: 4
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   10.0.0.0        255.255.0.0     Inside

Phase: 5
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group Inside_access_in in interface Inside
access-list Inside_access_in extended permit tcp any any object-group tcp-common-out
object-group service tcp-common-out tcp
 port-object eq www
 port-object eq https
 group-object jabber
 port-object eq ssh
 group-object GoToMeeting-8200
 port-object eq aol
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0xd7b4f260, priority=12, domain=permit, deny=false
        hits=7996866, user_data=0xd6874dc0, cs_id=0x0, flags=0x0, protocol=6
        src ip=0.0.0.0, mask=0.0.0.0, port=0
        dst ip=0.0.0.0, mask=0.0.0.0, port=80, dscp=0x0

Phase: 6
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0xd7b80520, priority=0, domain=permit-ip-option, deny=true
        hits=45888104, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
        src ip=0.0.0.0, mask=0.0.0.0, port=0
        dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

Phase: 7
Type: NAT-EXEMPT
Subtype:
Result: ALLOW
Config:
nat-control
  match ip Inside 10.0.1.0 255.255.255.0 Outside 90.103.14.112 255.255.255.248
    NAT exempt
    translate_hits = 256, untranslate_hits = 0
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0xd8a7f328, priority=6, domain=nat-exempt, deny=false
        hits=256, user_data=0xd8a7f268, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
        src ip=10.0.1.0, mask=255.255.255.0, port=0
        dst ip=90.103.14.112, mask=255.255.255.248, port=0, dscp=0x0

Phase: 8
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (Inside) 1 10.0.0.0 255.255.0.0
nat-control
  match ip Inside 10.0.0.0 255.255.0.0 Outside-PaeTec any
    dynamic translation to pool 1 (*myexternalIP* [Interface PAT])
    translate_hits = 21194670, untranslate_hits = 1663383
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0xd7efdfd8, priority=1, domain=nat, deny=false
        hits=34191159, user_data=0xd7efdf18, cs_id=0x0, flags=0x0, protocol=0
        src ip=10.0.0.0, mask=255.255.0.0, port=0
        dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

Phase: 9
Type: NAT
Subtype: host-limits
Result: ALLOW
Config:
nat (Inside) 1 10.0.0.0 255.255.0.0
nat-control
  match ip Inside 10.0.0.0 255.255.0.0 Outside-Stage2 any
    dynamic translation to pool 1 (*myotherexternalIP* [Interface PAT])
    translate_hits = 7200, untranslate_hits = 0
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0xd7efda30, priority=1, domain=host, deny=false
        hits=45424444, user_data=0xd7efd618, cs_id=0x0, reverse, flags=0x0, protocol=0
        src ip=10.0.0.0, mask=255.255.0.0, port=0
        dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

Phase: 10
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 out id=0xd89abe58, priority=70, domain=encrypt, deny=false
        hits=11, user_data=0x81ad24, cs_id=0xd8b731c0, reverse, flags=0x0, protocol=0
        src ip=10.0.1.0, mask=255.255.255.0, port=0
        dst ip=90.103.14.112, mask=255.255.255.248, port=0, dscp=0x0

Phase: 11
Type: ACCESS-LIST
Subtype: ipsec-user
Result: DROP
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 out id=0xd89cd6e8, priority=70, domain=ipsec-user, deny=true
        hits=10, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0
        src ip=10.0.1.0, mask=255.255.255.0, port=0
        dst ip=90.103.14.112, mask=255.255.255.248, port=0, dscp=0x0

Result:
input-interface: Inside
input-status: up
input-line-status: up
output-interface: Inside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

0
 
LVL 15

Expert Comment

by:Voltz-dk
ID: 26105779
If what you are making if a VPN filter, then they are a tad special.  They are bi-directional, and the remote network is always specified first (no matter the direction).  It certainly reduces the flexibility of such filters :(

You can check this out:
https://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_configuration_example09186a00808c9a87.shtml#bidf
0
 
LVL 32

Expert Comment

by:harbor235
ID: 26105897


i will admit it's a little confusing following this, I thought you posted something detailing the traffic was dropped on the inside interface? I do not see that comment now?

However, your NAT exempt should be private source IP to private destination IP not private to public,

i would need to see your config, again, it is hard following

harbor235 ;}
0
 

Author Comment

by:grabo
ID: 26105923
Haven't made any edits -- the traffic is dropped on the inside interface (see original question). NAT exempt is private source to private destination. Our business partner uses 90.103.0.0/16 internally (yes, we already had THAT discussion).

Will post config, but it'll take a while to redact a bunch of things.
0
 
LVL 15

Expert Comment

by:Voltz-dk
ID: 26106009
Grabo, did you even read my post 3 places above? :)
0
 

Author Comment

by:grabo
ID: 26106036
Can you explain "the remote network is always specified first"?

Here's the ACL that I currently have configured:

access-list Bizpartner-VPN extended permit ip object-group MyCompany-Nets-to-Bizpartner object-group Bizpartner-Nets-to-MyCompany log warnings
access-list Bizpartner-VPN extended deny ip object-group Bizpartner-Nets-to-MyCompany object-group MyCompany-Nets-to-Bizpartner log warnings
access-list Bizpartner-VPN remark Permit MyCompany networks to Bizpartner

I tried swapping the first two lines in the ACL (to "specify the remote network first" -- not sure what that means), but the traffic is still dropped.
0
 
LVL 15

Expert Comment

by:Voltz-dk
ID: 26106109
It's not the order of the ACEs, but source vs. destination inside the ACE.  That is why it works when you "allow the return traffic" - cuz that is how they are really defined.

So you need like:
access-l Bizpartner-VPN permit tcp host <their webserver IP> eq 80 <your network>
(in order to browse that one server)

So what you have above line 1 is reversed and have no use.
The 2nd one says to deny everything.
---
As I wrote they aren't too flexible.  But there is an alternative...
0
 

Author Comment

by:grabo
ID: 26106163
Well yes, I already realized that if I explicitly permit the return traffic (see original question), it works. But that's not stateful, is it? Aren't I then permitting any tcp traffic from Bizpartner to me, sourced on port 80?
0
 
LVL 15

Expert Comment

by:Voltz-dk
ID: 26106234
Well, it will work by ONLY specifying to allow the return traffic - cuz these ACLs work differently :)

But yes, they are not too flexible - you can't allow everything in one direction, and deny everything in the other.  They are bi-directional, and yes it means they can access you by sourcing port 80.
--
The alternative is fully flexible, but may not suit everyone.  It depends a bit on the amount of VPN traffic you run, and how you generally like to handle that.  If you only have this one tunnel, and desire is to disable their inbound traffic.

Remove the vpnfilter and disable the sysopt permit-vpn. (no sysopt permit-vpn).
sysopt permit-vpn states (globally) that inbound VPN traffic is trusted.  By disabling it, you have to explicitly allowing it in. (on the interface ACL).  And thus you have your normal flexibility.
0
 

Author Comment

by:grabo
ID: 26106268
I do not have "sysopt permit-vpn" (actually looks like "sysopt connection permit-vpn") listed in my configuration. So it sounds like I am already not permitting inbound traffic?
0
 
LVL 15

Expert Comment

by:Voltz-dk
ID: 26106429
Yes, I missed the connection keyword.  In newer versions it's on by default.  To be sure:

sh run all sysopt
0
 

Author Comment

by:grabo
ID: 26107105
ok, if it's on by default, then that's why I missed it. Sure enough

#sh run all sysopt
...
sysopt connection permit-vpn
...

If I turn this *off*, from what I read from Cisco, I need to explicitly permit the VPN traffic as well as the IKE traffic (udp 500). The policies I have already permit the VPN traffic (don't know why that was done) -- we have a handful of other VPNs.

If I do this, presumably, do I place the outbound traffic rule on the inside interface (which will then be stateful), and not place anything else on the outside interface (since I don't want to accept any new connections through the VPN?). Similarly, if for some reason I did want to permit incoming connections, they'd be put on the outside interface?
0
 
LVL 15

Expert Comment

by:Voltz-dk
ID: 26107271
Yes.

Except I've never had to explicitly allow the IKE traffic (or any other related to the tunnel) - that should be handled by the IPSEC config.

Can you direct me to where you read that? (I suppose there may be plenty of versions where I haven't done this).
0
 

Author Comment

by:grabo
ID: 26107359
The last bit of http://www.cisco.com/en/US/products/ps6120/products_tech_note09186a00807e0aca.shtml#Solution12

"Note: If you do not wish to use the sysopt connection command, then you must explicitly permit the required traffic, which is interesting traffic from source to destination, for example, from LAN of remote device to LAN of local device and "UDP port 500" for outside interface of remote devie to outside interface of local device, in outside ACL."
0
 
LVL 15

Accepted Solution

by:
Voltz-dk earned 1000 total points
ID: 26107428
I see..  It's not consistent with the description in the command reference:

Usage Guidelines
By default, the security appliance allows VPN traffic to terminate on a security appliance interface; you do not need to allow IKE or ESP (or other types of VPN packets) in an interface access list. By default, you also do not need an interface access list for local IP addresses of decrypted VPN packets. Because the VPN tunnel was terminated successfully using VPN security mechanisms, this feature simplifies configuration and maximizes the security appliance performance without any security risks. (Group policy and per-user authorization access lists still apply to the traffic.)

You can require an interface access list to apply to the local IP addresses by entering the no sysopt connection permit-vpn command. See the the access-list and access-group commands to create an access list and apply it to an interface. The access list applies to the local IP address, and not to the original client IP address used before the VPN packet was decrypted.

Examples
The following example requires decrypted VPN traffic to comply with interface access lists:

hostname(config)# no sysopt connection permit-vpn
---
I don't think you need it, but I suppose you can add it and watch hitcount or try without it.
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article explains the fundamentals of industrial networking which ultimately is the backbone network which is providing communications for process devices like robots and other not so interesting stuff.
Considering cloud tradeoffs and determining the right mix for your organization.
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
As a trusted technology advisor to your customers you are likely getting the daily question of, ‘should I put this in the cloud?’ As customer demands for cloud services increases, companies will see a shift from traditional buying patterns to new…
Suggested Courses
Course of the Month13 days, 23 hours left to enroll

807 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question