asked on Cisco ASA: PBR and site-to-site issue (no traffic received by remote endpoint)
In our datacenter we have a ASA pair (failover active/standby) which are connected to 2 ISP's. We are currently migrating from ISP1 to ISP2 and are using Policy Based Routing (PBR) on the ASA to make the transition smooth. We can pace the migration and move services step by step to the new ISP.
Everything seems to be working pretty well with PBR except for a problem with some site-to-site VPN-connections. We have a couple of site-to-site connections coming in on the ASA, for some we have access to both endpoints, for others the endpoint is managed by a third party. So originally the site-to-site connections are terminated by the ASA on ISP1, the remote end is connecting to the WAN IP of ISP1. To migrate we want to terminate the VPN on ISP2 on the ASA. So we reconfigure the remote endpoint to connect to the WAN IP of ISP2.
During the migration we use ISP1 as default (lowest metric in static route). With PBR we make sure that VPN traffic from and to ISP2 is routed correctly.
For Site-REMOTE1 and site-REMOTE2 this is working flawlessly, the remote endpoints are now connecting via ISP2 and are setting up a tunnel where we can see traffic TX and RX on both endpoints. Services at both ends working and tunnel is functioning.
For Site-REMOTE3 we see incoming and outgoing traffic on the ASA in the datacenter, but the remote endpoint is not receiving traffic (RX = 0). The tunnel is online and counter for RX datacenter ASA = counter TX REMOTE ASA.
If I switch to ISP2 as default (lower metric for ISP2 in static route), REMOTE3 starts functioning. But because not all site-to-site have been migrated yet to new ISP2 (the third party ones) this is not a good solution. The connections not yet migrated stop working.
Does anyone have some experience with PRB and site-to-site VPN on the ASA and perhaps a possible solution? We want to migrate third party connections one by one, but this seems to be impossible with this issue.
Extra information:
- If I put Site-REMOTE3 to connect the VPN via ISP1, all starts working again (same VPN parameters)
- I have tried with static routes to the VPN subnet too, same result
- No ACL on the VPN Group Policy defined (so all traffic allowed)
- Tried IKEv1 and IKEv2
- Packet trace shows packets not being blocked (allowed) and leaving the firewall via the correct interface
- Reverse entire config (ISP2 gets lowest metric and reverse route map (go_out_ISP1) -> this fixes Site-Remote3 but breaks connection not yet migrated
We have the following relevant configuration (sensitive information removed):
interface Port-channel1
description INSIDE
lacp max-bundle 8
nameif inside
security-level 100
ip address 1.1.1.1 255.255.255.0
policy-route route-map GO_OUT_ISP2
!
interface Port-channel1.100
description ISP1
vlan 100
nameif ISP1
security-level 0
ip address 10.10.10.2 255.255.255.248
!
interface Port-channel1.101
description ISP2
vlan 101
nameif ISP2
security-level 0
ip address 20.20.20.2 255.255.255.248
!
object network Site-LOCAL
subnet 1.1.1.0 255.255.255.0
object network Site-REMOTE1
subnet B.B.B.B 255.255.255.0
object network Site-REMOTE2
subnet C.C.C.C 255.255.255.0
object network Site-REMOTE3
subnet D.D.D.D 255.255.255.0
object-group network DM_INLINE_NETWORK_12
network-object object Site-REMOTE1
network-object object Site-REMOTE2
network-object object Site-REMOTE3
!
access-list acl_1_go_out_ISP2 extended permit ip any object-group DM_INLINE_NETWORK_12
access-list isp2_cryptomap extended permit ip object Site-LOCAL object Site-REMOTE1
access-list isp2_cryptomap_4 extended permit ip object Site-LOCAL object Site-REMOTE2
access-list isp2_cryptomap_2 extended permit ip object Site-LOCAL object Site-REMOTE3
!
nat (inside,isp2) source static Site-LOCAL Site-LOCAL destination static Site-REMOTE1 Site-REMOTE1 no-proxy-arp route-lookup
nat (inside,isp2) source static Site-LOCAL Site-LOCAL destination static Site-REMOTE2 Site-REMOTE2 no-proxy-arp route-lookup
nat (inside,isp2) source static Site-LOCAL Site-LOCAL destination static Site-REMOTE3 Site-REMOTE3 no-proxy-arp route-lookup
!
route-map GO_OUT_ISP2 permit 1
match ip address acl_1_go_out_ISP2
set interface ISP2
!
route ISP1 0.0.0.0 0.0.0.0 10.10.10.1 1
route ISP2 0.0.0.0 0.0.0.0 20.20.20.1 5
!
crypto map isp2_map1 1 match address isp2_cryptomap
crypto map isp2_map1 1 set pfs group24
crypto map isp2_map1 1 set peer WAN_IP_SITE-REMOTE1
crypto map isp2_map1 1 set ikev2 ipsec-proposal AES256
crypto map isp2_map1 5 match address isp2_cryptomap_4
crypto map isp2_map1 5 set pfs group24
crypto map isp2_map1 5 set peer WAN_IP_SITE-REMOTE2
crypto map isp2_map1 5 set ikev2 ipsec-proposal AES256
crypto map isp2_map1 2 match address isp2_cryptomap_2
crypto map isp2_map1 2 set pfs group24
crypto map isp2_map1 2 set peer WAN_IP_SITE-REMOTE3
crypto map isp2_map1 2 set ikev2 ipsec-proposal AES256
group-policy GP_VPN_REMOTE1 internal
group-policy GP_VPN_REMOTE1 attributes
vpn-tunnel-protocol ikev2
group-policy GP_VPN_REMOTE3 internal
group-policy GP_VPN_REMOTE3 attributes
vpn-tunnel-protocol ikev2
group-policy GP_VPN_REMOTE3 internal
group-policy GP_VPN_REMOTE3 attributes
vpn-tunnel-protocol ikev2
tunnel-group WAN_IP_SITE-REMOTE1 type ipsec-l2l
tunnel-group WAN_IP_SITE-REMOTE1 general-attributes
default-group-policy GP_VPN_REMOTE1
tunnel-group WAN_IP_SITE-REMOTE1 ipsec-attributes
ikev2 remote-authentication pre-shared-key *****
ikev2 local-authentication pre-shared-key *****
tunnel-group WAN_IP_SITE-REMOTE1 type ipsec-l2l
tunnel-group WAN_IP_SITE-REMOTE1 general-attributes
default-group-policy GP_VPN_REMOTE1
tunnel-group WAN_IP_SITE-REMOTE1 ipsec-attributes
ikev2 remote-authentication pre-shared-key *****
ikev2 local-authentication pre-shared-key *****
tunnel-group WAN_IP_SITE-REMOTE3 type ipsec-l2l
tunnel-group WAN_IP_SITE-REMOTE3 general-attributes
default-group-policy GP_VPN_REMOTE3
tunnel-group WAN_IP_SITE-REMOTE3 ipsec-attributes
ikev2 remote-authentication pre-shared-key *****
ikev2 local-authentication pre-shared-key *****
Everything seems to be working pretty well with PBR except for a problem with some site-to-site VPN-connections. We have a couple of site-to-site connections coming in on the ASA, for some we have access to both endpoints, for others the endpoint is managed by a third party. So originally the site-to-site connections are terminated by the ASA on ISP1, the remote end is connecting to the WAN IP of ISP1. To migrate we want to terminate the VPN on ISP2 on the ASA. So we reconfigure the remote endpoint to connect to the WAN IP of ISP2.
During the migration we use ISP1 as default (lowest metric in static route). With PBR we make sure that VPN traffic from and to ISP2 is routed correctly.
For Site-REMOTE1 and site-REMOTE2 this is working flawlessly, the remote endpoints are now connecting via ISP2 and are setting up a tunnel where we can see traffic TX and RX on both endpoints. Services at both ends working and tunnel is functioning.
For Site-REMOTE3 we see incoming and outgoing traffic on the ASA in the datacenter, but the remote endpoint is not receiving traffic (RX = 0). The tunnel is online and counter for RX datacenter ASA = counter TX REMOTE ASA.
If I switch to ISP2 as default (lower metric for ISP2 in static route), REMOTE3 starts functioning. But because not all site-to-site have been migrated yet to new ISP2 (the third party ones) this is not a good solution. The connections not yet migrated stop working.
Does anyone have some experience with PRB and site-to-site VPN on the ASA and perhaps a possible solution? We want to migrate third party connections one by one, but this seems to be impossible with this issue.
Extra information:
- If I put Site-REMOTE3 to connect the VPN via ISP1, all starts working again (same VPN parameters)
- I have tried with static routes to the VPN subnet too, same result
- No ACL on the VPN Group Policy defined (so all traffic allowed)
- Tried IKEv1 and IKEv2
- Packet trace shows packets not being blocked (allowed) and leaving the firewall via the correct interface
- Reverse entire config (ISP2 gets lowest metric and reverse route map (go_out_ISP1) -> this fixes Site-Remote3 but breaks connection not yet migrated
We have the following relevant configuration (sensitive information removed):
interface Port-channel1
description INSIDE
lacp max-bundle 8
nameif inside
security-level 100
ip address 1.1.1.1 255.255.255.0
policy-route route-map GO_OUT_ISP2
!
interface Port-channel1.100
description ISP1
vlan 100
nameif ISP1
security-level 0
ip address 10.10.10.2 255.255.255.248
!
interface Port-channel1.101
description ISP2
vlan 101
nameif ISP2
security-level 0
ip address 20.20.20.2 255.255.255.248
!
object network Site-LOCAL
subnet 1.1.1.0 255.255.255.0
object network Site-REMOTE1
subnet B.B.B.B 255.255.255.0
object network Site-REMOTE2
subnet C.C.C.C 255.255.255.0
object network Site-REMOTE3
subnet D.D.D.D 255.255.255.0
object-group network DM_INLINE_NETWORK_12
network-object object Site-REMOTE1
network-object object Site-REMOTE2
network-object object Site-REMOTE3
!
access-list acl_1_go_out_ISP2 extended permit ip any object-group DM_INLINE_NETWORK_12
access-list isp2_cryptomap extended permit ip object Site-LOCAL object Site-REMOTE1
access-list isp2_cryptomap_4 extended permit ip object Site-LOCAL object Site-REMOTE2
access-list isp2_cryptomap_2 extended permit ip object Site-LOCAL object Site-REMOTE3
!
nat (inside,isp2) source static Site-LOCAL Site-LOCAL destination static Site-REMOTE1 Site-REMOTE1 no-proxy-arp route-lookup
nat (inside,isp2) source static Site-LOCAL Site-LOCAL destination static Site-REMOTE2 Site-REMOTE2 no-proxy-arp route-lookup
nat (inside,isp2) source static Site-LOCAL Site-LOCAL destination static Site-REMOTE3 Site-REMOTE3 no-proxy-arp route-lookup
!
route-map GO_OUT_ISP2 permit 1
match ip address acl_1_go_out_ISP2
set interface ISP2
!
route ISP1 0.0.0.0 0.0.0.0 10.10.10.1 1
route ISP2 0.0.0.0 0.0.0.0 20.20.20.1 5
!
crypto map isp2_map1 1 match address isp2_cryptomap
crypto map isp2_map1 1 set pfs group24
crypto map isp2_map1 1 set peer WAN_IP_SITE-REMOTE1
crypto map isp2_map1 1 set ikev2 ipsec-proposal AES256
crypto map isp2_map1 5 match address isp2_cryptomap_4
crypto map isp2_map1 5 set pfs group24
crypto map isp2_map1 5 set peer WAN_IP_SITE-REMOTE2
crypto map isp2_map1 5 set ikev2 ipsec-proposal AES256
crypto map isp2_map1 2 match address isp2_cryptomap_2
crypto map isp2_map1 2 set pfs group24
crypto map isp2_map1 2 set peer WAN_IP_SITE-REMOTE3
crypto map isp2_map1 2 set ikev2 ipsec-proposal AES256
group-policy GP_VPN_REMOTE1 internal
group-policy GP_VPN_REMOTE1 attributes
vpn-tunnel-protocol ikev2
group-policy GP_VPN_REMOTE3 internal
group-policy GP_VPN_REMOTE3 attributes
vpn-tunnel-protocol ikev2
group-policy GP_VPN_REMOTE3 internal
group-policy GP_VPN_REMOTE3 attributes
vpn-tunnel-protocol ikev2
tunnel-group WAN_IP_SITE-REMOTE1 type ipsec-l2l
tunnel-group WAN_IP_SITE-REMOTE1 general-attributes
default-group-policy GP_VPN_REMOTE1
tunnel-group WAN_IP_SITE-REMOTE1 ipsec-attributes
ikev2 remote-authentication pre-shared-key *****
ikev2 local-authentication pre-shared-key *****
tunnel-group WAN_IP_SITE-REMOTE1 type ipsec-l2l
tunnel-group WAN_IP_SITE-REMOTE1 general-attributes
default-group-policy GP_VPN_REMOTE1
tunnel-group WAN_IP_SITE-REMOTE1 ipsec-attributes
ikev2 remote-authentication pre-shared-key *****
ikev2 local-authentication pre-shared-key *****
tunnel-group WAN_IP_SITE-REMOTE3 type ipsec-l2l
tunnel-group WAN_IP_SITE-REMOTE3 general-attributes
default-group-policy GP_VPN_REMOTE3
tunnel-group WAN_IP_SITE-REMOTE3 ipsec-attributes
ikev2 remote-authentication pre-shared-key *****
ikev2 local-authentication pre-shared-key *****
Hardware FirewallsCiscoNetworkingVPN
Last Comment
ISBNET
ASKER CERTIFIED SOLUTION
THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
Thank you very very very much!! :)