Cisco ASA: PBR and site-to-site issue (no traffic received by remote endpoint)

In our datacenter we have a ASA pair (failover active/standby) which are connected to 2 ISP's. We are currently migrating from ISP1 to ISP2 and are using Policy Based Routing (PBR) on the ASA to make the transition smooth. We can pace the migration and move services step by step to the new ISP.

Everything seems to be working pretty well with PBR except for a problem with some site-to-site VPN-connections. We have a couple of site-to-site connections coming in on the ASA, for some we have access to both endpoints, for others the endpoint is managed by a third party. So originally the site-to-site connections are terminated by the ASA on ISP1, the remote end is connecting to the WAN IP of ISP1. To migrate we want to terminate the VPN on ISP2 on the ASA. So we reconfigure the remote endpoint to connect to the WAN IP of ISP2.

During the migration we use ISP1 as default (lowest metric in static route). With PBR we make sure that VPN traffic from and to ISP2 is routed correctly.

For Site-REMOTE1 and site-REMOTE2 this is working flawlessly, the remote endpoints are now connecting via ISP2 and are setting up a tunnel where we can see traffic TX and RX on both endpoints. Services at both ends working and tunnel is functioning.

For Site-REMOTE3 we see incoming and outgoing traffic on the ASA in the datacenter, but the remote endpoint is not receiving traffic (RX = 0). The tunnel is online and counter for RX datacenter ASA = counter TX REMOTE ASA.


If I switch to ISP2 as default (lower metric for ISP2 in static route), REMOTE3 starts functioning. But because not all site-to-site have been migrated yet to new ISP2 (the third party ones) this is not a good solution. The connections not yet migrated stop working.

Does anyone have some experience with PRB and site-to-site VPN on the ASA and perhaps a possible solution? We want to migrate third party connections one by one, but this seems to be impossible with this issue.


Extra information:
- If I put Site-REMOTE3 to connect the VPN via ISP1, all starts working again (same VPN parameters)
- I have tried with static routes to the VPN subnet too, same result
- No ACL on the VPN Group Policy defined (so all traffic allowed)
- Tried IKEv1 and IKEv2
- Packet trace shows packets not being blocked (allowed) and leaving the firewall via the correct interface
- Reverse entire config (ISP2 gets lowest metric and reverse route map (go_out_ISP1) -> this fixes Site-Remote3 but breaks connection not yet migrated


We have the following relevant configuration (sensitive information removed):
interface Port-channel1
 description INSIDE
 lacp max-bundle 8
 nameif inside
 security-level 100
 ip address 1.1.1.1 255.255.255.0
 policy-route route-map GO_OUT_ISP2
!
interface Port-channel1.100
 description ISP1
 vlan 100
 nameif ISP1
 security-level 0
 ip address 10.10.10.2 255.255.255.248
!
interface Port-channel1.101
 description ISP2
 vlan 101
 nameif ISP2
 security-level 0
 ip address 20.20.20.2 255.255.255.248
!
object network Site-LOCAL
 subnet 1.1.1.0 255.255.255.0
object network Site-REMOTE1
 subnet B.B.B.B 255.255.255.0
object network Site-REMOTE2
 subnet C.C.C.C 255.255.255.0
object network Site-REMOTE3
 subnet D.D.D.D 255.255.255.0
object-group network DM_INLINE_NETWORK_12
 network-object object Site-REMOTE1
 network-object object Site-REMOTE2
 network-object object Site-REMOTE3

!
access-list acl_1_go_out_ISP2 extended permit ip any object-group DM_INLINE_NETWORK_12
access-list isp2_cryptomap extended permit ip object Site-LOCAL object Site-REMOTE1
access-list isp2_cryptomap_4 extended permit ip object Site-LOCAL object Site-REMOTE2
access-list isp2_cryptomap_2 extended permit ip object Site-LOCAL object Site-REMOTE3
!
nat (inside,isp2) source static Site-LOCAL Site-LOCAL destination static Site-REMOTE1 Site-REMOTE1 no-proxy-arp route-lookup
nat (inside,isp2) source static Site-LOCAL Site-LOCAL destination static Site-REMOTE2 Site-REMOTE2 no-proxy-arp route-lookup
nat (inside,isp2) source static Site-LOCAL Site-LOCAL destination static Site-REMOTE3 Site-REMOTE3 no-proxy-arp route-lookup
!
route-map GO_OUT_ISP2 permit 1
 match ip address acl_1_go_out_ISP2
 set interface ISP2

!
route ISP1 0.0.0.0 0.0.0.0 10.10.10.1 1
route ISP2 0.0.0.0 0.0.0.0 20.20.20.1 5
!
crypto map isp2_map1 1 match address isp2_cryptomap
crypto map isp2_map1 1 set pfs group24
crypto map isp2_map1 1 set peer WAN_IP_SITE-REMOTE1
crypto map isp2_map1 1 set ikev2 ipsec-proposal AES256
crypto map isp2_map1 5 match address isp2_cryptomap_4
crypto map isp2_map1 5 set pfs group24
crypto map isp2_map1 5 set peer WAN_IP_SITE-REMOTE2
crypto map isp2_map1 5 set ikev2 ipsec-proposal AES256
crypto map isp2_map1 2 match address isp2_cryptomap_2
crypto map isp2_map1 2 set pfs group24
crypto map isp2_map1 2 set peer WAN_IP_SITE-REMOTE3
crypto map isp2_map1 2 set ikev2 ipsec-proposal AES256

group-policy GP_VPN_REMOTE1 internal
group-policy GP_VPN_REMOTE1 attributes
 vpn-tunnel-protocol ikev2
group-policy GP_VPN_REMOTE3 internal
group-policy GP_VPN_REMOTE3 attributes
 vpn-tunnel-protocol ikev2
group-policy GP_VPN_REMOTE3 internal
group-policy GP_VPN_REMOTE3 attributes
 vpn-tunnel-protocol ikev2

tunnel-group WAN_IP_SITE-REMOTE1 type ipsec-l2l
tunnel-group WAN_IP_SITE-REMOTE1 general-attributes
 default-group-policy GP_VPN_REMOTE1
tunnel-group WAN_IP_SITE-REMOTE1 ipsec-attributes
 ikev2 remote-authentication pre-shared-key *****
 ikev2 local-authentication pre-shared-key *****

tunnel-group WAN_IP_SITE-REMOTE1 type ipsec-l2l
tunnel-group WAN_IP_SITE-REMOTE1 general-attributes
 default-group-policy GP_VPN_REMOTE1
tunnel-group WAN_IP_SITE-REMOTE1 ipsec-attributes
 ikev2 remote-authentication pre-shared-key *****
 ikev2 local-authentication pre-shared-key *****

tunnel-group WAN_IP_SITE-REMOTE3 type ipsec-l2l
tunnel-group WAN_IP_SITE-REMOTE3 general-attributes
 default-group-policy GP_VPN_REMOTE3
tunnel-group WAN_IP_SITE-REMOTE3 ipsec-attributes
 ikev2 remote-authentication pre-shared-key *****
 ikev2 local-authentication pre-shared-key *****
ISBNETAsked:
Who is Participating?
 
lrmooreCommented:
Set static route to remote3 peer
route isp2 <peer3 ip> 255.255.255.255 20.20.10.1
1
 
ISBNETAuthor Commented:
Oh my god... it was this easy... adding the static route to the peer IP worked like a charm.

Thank you very very very much!! :)
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.