Site to Site VPN on Cisco ASA - Tunnel Up No Traffic After Upgrade

After upgrading ASA5520 (Main office) and ASA5505 (Remote office) from 8.2 to 8.4 (and attempting to re-learn NAT) the site to site VPN is no longer passing traffic. The IPSec tunnel is up. Fairly sure it have something to do with the changes in 8.2-8.4 but not sure what. Main office is on its own dedicated fiber DIA and remote office is on cable modem (bridged) with static IP.

Here is a parsed config showing the relevant bits from the remote office side:

ASA Version 8.4(3) 
!
interface Vlan1
 nameif outside
 security-level 0
 ip address dhcp setroute 
!
interface Vlan10
 nameif inside
 security-level 100
 ip address 192.168.0.254 255.255.255.0 
!
object network NAT-Inside
 subnet 0.0.0.0 0.0.0.0
object network Main-Office
 subnet 192.168.1.0 255.255.255.0
object network Remote-Office
 subnet 192.168.0.0 255.255.255.0
!
access-list outside_cryptomap extended permit ip object Remote-Office object Main-Office 
!
nat (inside,outside) source static Remote-Office Remote-Office destination static Main-Office Main-Office no-proxy-arp route-lookup
!
object network NAT-Inside
 nat (inside,outside) dynamic interface
!
crypto map outside_map 1 match address outside_cryptomap
crypto map outside_map 1 set peer 209.69.xxx.xxx 
crypto map outside_map 1 set ikev1 transform-set ESP-AES-128-SHA ESP-AES-128-MD5 ESP-AES-192-SHA ESP-AES-192-MD5 ESP-AES-256-SHA ESP-AES-256-MD5 ESP-3DES-SHA ESP-3DES-MD5 ESP-DES-SHA ESP-DES-MD5
crypto map outside_map 1 set ikev2 ipsec-proposal AES256 AES192 AES 3DES DES
crypto map outside_map interface outside
!
group-policy GroupPolicy_209.69.xxx.xxx internal
group-policy GroupPolicy_209.69.xxx.xxx attributes
 vpn-tunnel-protocol ikev1 ikev2 
tunnel-group 209.69.xxx.xxx type ipsec-l2l
tunnel-group 209.69.xxx.xxx general-attributes
 default-group-policy GroupPolicy_209.69.xxx.xxx
tunnel-group 209.69.xxx.xxx ipsec-attributes
 ikev1 pre-shared-key *****
 ikev2 remote-authentication pre-shared-key *****
 ikev2 local-authentication pre-shared-key *****
!
class-map inspection_default
 match default-inspection-traffic
!

Open in new window

Thank you in advance.
LVL 3
ThePhreakshowAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

SouljaSr.Net.EngCommented:
sh crypto isakmp sa  shows the tunnel up ?

What does your firewall log say when you attempt to send traffic. Is it allowing it?
Also what is you xlate table showing?
SouljaSr.Net.EngCommented:
Can you also show the main office config?
ThePhreakshowAuthor Commented:
There are no IKEv1 SAs

IKEv2 SAs:

Session-id:4, Status:UP-ACTIVE, IKE count:1, CHILD count:1

Tunnel-id                 Local                Remote     Status         Role
158852231    209.69.xxx.xxx/500     97.70.xxx.xxx/500      READY    RESPONDER
      Encr: AES-CBC, keysize: 256, Hash: SHA96, DH Grp:5, Auth sign: PSK, Auth verify: PSK
      Life/Active Time: 86400/8091 sec
Child sa: local selector  192.168.1.0/0 - 192.168.1.255/65535
          remote selector 192.168.0.0/0 - 192.168.0.255/65535
          ESP spi in/out: 0xf359eab3/0x64d6ecff
Why Diversity in Tech Matters

Kesha Williams, certified professional and software developer, explores the imbalance of diversity in the world of technology -- especially when it comes to hiring women. She showcases ways she's making a difference through the Colors of STEM program.

SouljaSr.Net.EngCommented:
What does your sh crypto ipsec sa show?
ThePhreakshowAuthor Commented:
Here is a clean config from the MAIN office site with relevant stuff:

ASA Version 8.4(3)
!
!
interface GigabitEthernet0/0
 nameif inside
 security-level 100
 ip address 192.168.1.1 255.255.255.0 
!
interface GigabitEthernet0/2
 nameif dmz
 security-level 25
 ip address 10.1.1.177 255.255.255.240 
!
interface GigabitEthernet0/3
 speed 100
 duplex full
 nameif outside
 security-level 0
 ip address 209.69.xxx.xxx 255.255.255.240 
!
object network DMZ-Net
 host 10.1.1.0
object network Main-Office
 host 192.168.1.0
object network NAT-DMZ
 subnet 0.0.0.0 0.0.0.0
object network NAT-Inside
 subnet 0.0.0.0 0.0.0.0

access-list inside extended permit ip any any 
access-list Main-Office standard permit 192.168.0.0 255.255.255.0 
access-list Main-Office standard permit 192.168.1.0 255.255.255.0 
access-list dmz-in extended permit icmp any any 
access-list outbound extended permit ip any any 
access-list to-dmz extended permit icmp any any 
access-list DefaultRAGroup_splitTunnelAcl standard permit host 192.168.1.0 
access-list outside_cryptomap_1 extended permit ip object Main-Office object Remote-Office 

icmp unreachable rate-limit 1 burst-size 1
icmp permit any inside
icmp permit any dmz
icmp permit any outside
!
nat (inside,outside) source static Main-Office Main-Office destination static Remote-Office Remote-Office no-proxy-arp route-lookup
!
object network NAT-Inside
 nat (inside,outside) dynamic interface
access-group inside in interface inside
access-group dmz-in in interface dmz
access-group to-dmz in interface outside
route outside 0.0.0.0 0.0.0.0 209.69.xxx.xxx 1
crypto dynamic-map SYSTEM_DEFAULT_CRYPTO_MAP 65535 set ikev2 ipsec-proposal AES256 AES192 AES 3DES DES
crypto map outside_map 1 match address outside_cryptomap_1
crypto map outside_map 1 set peer 97.70.xxx.xxx 
crypto map outside_map 1 set ikev1 transform-set ESP-AES-128-SHA ESP-AES-128-MD5 ESP-AES-192-SHA ESP-AES-192-MD5 ESP-AES-256-SHA ESP-AES-256-MD5 ESP-3DES-SHA ESP-3DES-MD5 ESP-DES-SHA ESP-DES-MD5
crypto map outside_map 1 set ikev2 ipsec-proposal DES 3DES AES AES192 AES256
crypto map outside_map 65535 ipsec-isakmp dynamic SYSTEM_DEFAULT_CRYPTO_MAP
crypto map outside_map interface outside

 tunnel-group-list enable
group-policy DefaultRAGroup internal
group-policy DefaultRAGroup attributes
 wins-server value 192.168.1.2 192.168.3.2
 dns-server value 192.168.1.2 192.168.3.2
 vpn-tunnel-protocol l2tp-ipsec 
 split-tunnel-policy tunnelspecified
 split-tunnel-network-list value DefaultRAGroup_splitTunnelAcl
 default-domain value demmer.com
group-policy DfltGrpPolicy attributes
 vpn-tunnel-protocol ssl-client 
group-policy GroupPolicy_97.70.xxx.xxx internal
group-policy GroupPolicy_97.70.xxx.xxx attributes
 vpn-tunnel-protocol ikev1 ikev2 
tunnel-group DefaultL2LGroup ipsec-attributes
 ikev1 pre-shared-key *****
tunnel-group DefaultRAGroup general-attributes
 address-pool VPN-Pool
 default-group-policy DefaultRAGroup
tunnel-group DefaultRAGroup ipsec-attributes
 ikev1 pre-shared-key *****
tunnel-group 97.70.xxx.xxx type ipsec-l2l
tunnel-group 97.70.xxx.xxx general-attributes
 default-group-policy GroupPolicy_97.70.xxx.xxx
tunnel-group 97.70.xxx.xxx ipsec-attributes
 ikev1 pre-shared-key *****
 ikev2 remote-authentication pre-shared-key *****
 ikev2 local-authentication pre-shared-key *****

Open in new window

ThePhreakshowAuthor Commented:
MAIN-Office# sh crypto ipsec sa
interface: outside
    Crypto map tag: SYSTEM_DEFAULT_CRYPTO_MAP, seq num: 65535, local addr: 209.69.xxx.xxx

      local ident (addr/mask/prot/port): (192.168.1.0/255.255.255.0/0/0)
      remote ident (addr/mask/prot/port): (192.168.0.0/255.255.255.0/0/0)
      current_peer: 97.70.xxx.xxx

      #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
      #pkts decaps: 5844, #pkts decrypt: 5844, #pkts verify: 5844
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 0, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #send errors: 0, #recv errors: 0

      local crypto endpt.: 209.69.xxx.xxx/500, remote crypto endpt.: 97.70.xxx.xxx/500
      path mtu 1500, ipsec overhead 74, media mtu 1500
      current outbound spi: 64D6ECFF
      current inbound spi : F359EAB3

    inbound esp sas:
      spi: 0xF359EAB3 (4082756275)
         transform: esp-aes-256 esp-sha-hmac no compression
         in use settings ={L2L, Tunnel, }
         slot: 0, conn_id: 1843200, crypto-map: SYSTEM_DEFAULT_CRYPTO_MAP
         sa timing: remaining key lifetime (kB/sec): (3962446/19250)
         IV size: 16 bytes
         replay detection support: Y
         Anti replay bitmap:
          0xFFFFFFFF 0xFFFFFFFF
    outbound esp sas:
      spi: 0x64D6ECFF (1691806975)
         transform: esp-aes-256 esp-sha-hmac no compression
         in use settings ={L2L, Tunnel, }
         slot: 0, conn_id: 1843200, crypto-map: SYSTEM_DEFAULT_CRYPTO_MAP
         sa timing: remaining key lifetime (kB/sec): (3916800/19250)
         IV size: 16 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001
Pete LongTechnical ConsultantCommented:
OK Check fire!  Phase 1 is up (using IKEv2) Phase 2 is up but you are only getting one way SPI traffic decaps but no encaps

The problem is either with NAT at MAIN-Site or Routing at Main-Site (99% or we have a bug)
Remote Site looks to be OK (or we would never see decaps at Main Site :) )

So from 192.168.1.0/24 TO 192.168.0.0/24  is the problem correct?

I don't see an object group for Remote-Office so you've omitted a bit too much?
ThePhreakshowAuthor Commented:
Sorry, I must have clipped that out... Yes, Main-Office has this:

object network Remote-Office
 subnet 192.168.0.0 255.255.255.0

I cannot ping in either direction.
Feroz AhmedSenior Network Security  / Senior System EngineerCommented:
Hi,

I have gone through the Config file ,could you try this command on remote office ASA as below and check whether traffic is passing or not .

ASA(config-t)#crypto isakmp enable outside

If the above command does not work then try this command on ASA as below :

ASA(config-t)#sysopt connection permit-vpn

and check you should be able to pass traffic between 2 sites.
ThePhreakshowAuthor Commented:
Tried both those commands separate, and even together, and even one at a time on both sides, and still not passing any traffic. Tunnel still shows its up. Lots of encaps, zero decaps.
ThePhreakshowAuthor Commented:
Here is the last part of the packet trace, trying to ping from remote network to main office server:

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: inside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule
ThePhreakshowAuthor Commented:
Gave up, wiped everything clean and started from scratch... step-by-step

Entered everything via CLI rather than using that silly VPN wizard that is in ASDM.
Works good now!

Thank all of you for your valuable input. Much of it was used to get this up and running.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
ASA

From novice to tech pro — start learning today.