Cisco ASA 5510

WellingtonIS
WellingtonIS used Ask the Experts™
on
I need to rekey a tunnel - I've tried clear ipsec sa peer IP but no luck.  Anything else?
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Distinguished Expert 2017

Commented:
Please clarify what you are trying to achieve?

Author

Commented:
I was trying to get the tunnel re-established, however it seems to have done that itself.  I’m not to sure what’s going with on and why my tunnel keeps going down.  I have other vpn tunnels and I don’t have this issue.  This particular tunnel drops by itself and re-establishes on its own
Distinguished Expert 2017

Commented:
The tunnel commonly has a timeout for idle
The tunnel setup is usually automatic when a request has to go through it, I.e. A ping to a system on the other ..
The other traffic might be consistently active as compared to this one.
An option is that the key lifetime is mismatched, one side has 28800 seconds while the other side has a higher or lower number resulting in auto refresh at the wrong time leading and resulting  to the drop
Become a CompTIA Certified Healthcare IT Tech

This course will help prep you to earn the CompTIA Healthcare IT Technician certification showing that you have the knowledge and skills needed to succeed in installing, managing, and troubleshooting IT systems in medical and clinical settings.

Author

Commented:
OK I will check that out tomorrow. Thanks
Pete LongTechnical Consultant

Commented:
If a tunnel drops, due to a timeout then as soon as it sees 'interesting traffic' that needs to. be encrypted, it will re-establish the tunnel.
You can force it like this.

</p>

Author

Commented:
OK I managed to grab logs but I'm not sure what's going on.
4|Oct 28 2019|07:48:56|713903|||||IP = 66.220.42.206, Information Exchange processing failed
5|Oct 28 2019|07:48:56|713904|||||IP = 66.220.42.206, Received an un-encrypted NO_PROPOSAL_CHOSEN notify message, dropping
7|Oct 28 2019|07:48:56|713236|||||IP = 66.220.42.206, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + NOTIFY (11) + NONE (0) total length : 40
7|Oct 28 2019|07:48:56|713236|||||IP = 66.220.42.206, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 408

Can anyone decipher for me?

After that I ran:
packet-tracer input inside tcp 10.75.x.x (my inside) 80 66.220.x.x (their inside) 80 and the tunnel appeared.  I'm still not sure what's going on here.
Distinguished Expert 2017

Commented:
Please confirm the settings match.


The processing error might be related to your side initiating an exchange to maintain the tunnel, the other side ignores the process leading to the drop.
A packet triggers tunnel setup.

Author

Commented:
They are in CA and I'm on the east coast.  The weird thing is the tunnel stays up for days and suddenly it's down.  I'm just not sure what's happening.  I'd tend to think if that was the issue, it would be dropping all the time?  Bottom line is until I get with the west coast, I can't do anything.  Thanks for the reply
 sa timing: remaining key lifetime (sec): 1265  Is there a command to change this and make it longer?
Distinguished Expert 2017

Commented:
How active is the traffic over this link.
Is the drop impacting the business?
It drops and within 10-20 seconds it is back up.
If it always drop over the weekend .......

Author

Commented:
When it's used it has a lot of traffic, x-rays, ct scans etc.  It drops for hours not minutes and then suddenly reconnects.  So far it's been weekends but it's not limited to that.  It impacts business yes because I have to call someone in and I can't seem to get sleep
Distinguished Expert 2017

Commented:
Please explain, try the following, before the weekend, setup a constant ping from your side to the remote and see if it drops.

ping -t ip_on_the_remote_side
Make sure you are getting a response, ping initiated from a resource that supposed to have access.

From your description, the drop seems to relate to idle events.

Confirming parameters, which IPs from each side, key lifetimes..
Ask whether they have a hard break, idle timeout.

Do your notification for tunnel down include notification for tunnel up?

Author

Commented:
The "other-side" is supposedly doing a constant ping.  I can't really know if that's true, however, I am usually made aware after the tunnel goes down. .  If I do a constant ping should I do this from the machine or the ASA?
Distinguished Expert 2017

Commented:
The remore pinging you. Might not maintain the tunnel you need up.
I.e. They are pinging IPA on your side from ip2 on their side, while your service/component needs access from ipb to ip1
Depending on the VPN setup both can be true ip2 to IPA is up, but IPB to ip1 periodically drops.

This issue comes up based on definition
Your side has 192.168.10.8/29
While on their side they have individual iOS defined with a /32
The tunnel appears up from your side, but the traffic is being denied on the other. A ping from the other to ip1 will maintain ..

This is why it is straight forward to confirm the VPN setup is identical on both sides before tearing out ones .... Searching for other things.

The a anomaly could be added by a recent request/modification that one side interpreted it differently

I.e. Need to add to the prior another ip set 192.168.10.2-6

Your side modified the intersting traffic from 192.168.10.8/29 to 192.168.10.0/28
While the other side simply added another segment 192.168.10.0/29
The access-list to grant rights are not significant.

The effect, is that while your side will allow traffic from your side to enter, the other side might not allow the response to flow back as according to it, the second half of the tunnel is not up.

When you check if 192.168.10.4 is allowed, they will always say that yes, it is allowed.

Author

Commented:
I will do that if and when I can reach them.  Thanks.
Pete LongTechnical Consultant

Commented:
That debug just looks like a phase 1 policy not matching, that's not unusual, the initiator keep prpossing settings until the other side accepts one:)

OK to check a tunnel is up you need to first see if phase 1 is up
show crypto isakmp

Open in new window

Then See if Phase 2 is up
show crypto ipsec sa

Open in new window


If you see packets getting encrypted and decrypted, then the tunnel is working fine.

</p>

Author

Commented:
Just to keep you all updated.  I didn't go down last night.  I'm wondering if this is a weekend thing? If that's even possible?

Author

Commented:
Here's the latest my tunnel is down:
WRMCASA# ping inside 66.220.42.201
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 66.220.42.201, timeout is 2 seconds:
?????
Success rate is 0 percent (0/5)
WRMCASA# packet-
WRMCASA# packet-tracer input inside tcp 10.75.13.90 80 66.220.42.201 80

Phase: 1
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   0.0.0.0         0.0.0.0         outside

Phase: 2
Type: WCCP-REDIRECT
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 3
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 4
Type: FOVER
Subtype: standby-update
Result: ALLOW
Config:
Additional Information:

Phase: 5
Type: NAT-EXEMPT
Subtype:
Result: ALLOW
Config:
  match ip inside host 10.75.13.90 outside host 66.220.42.201
    NAT exempt
    translate_hits = 104, untranslate_hits = 0
Additional Information:

Phase: 6
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (inside) 101 0.0.0.0 0.0.0.0
  match ip inside any outside any
    dynamic translation to pool 101 (204.28.108.219 [Interface PAT])
    translate_hits = 21790, untranslate_hits = 5443
Additional Information:

Phase: 7
Type: NAT
Subtype: host-limits
Result: ALLOW
Config:
nat (inside) 101 0.0.0.0 0.0.0.0
  match ip inside any outside any
    dynamic translation to pool 101 (204.28.108.219 [Interface PAT])
    translate_hits = 22021, untranslate_hits = 5457
Additional Information:

Phase: 8
Type: VPN
Subtype: encrypt
Result: DROP
Config:
Additional Information:

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule - I don't have a configured rule to block this...
Distinguished Expert 2017

Commented:
You may have a two fold issue, you are using a public IP which 66. is that may lead to bleed out..
Presumably on the outside interface you have an access list denying packets going to the 66 based IP access out.
Because of this, the initiation of the tunnel is possibly relied on the other side.
and potentially this is why they have a constant ping which may at one point or another terminate and might coincide with the VPN tunnel drop.

Though Are you sure, the IP on the other side is 66 and not a private block?
you may have to create a route for destination 66 to go through a tunnel/ip.

Author

Commented:
Interesting... I'm not blocking 66 public addresses, however, I believe on their side I go though 2 firewalls.  They did "rekey" the tunnel and it did come up.  However, they are playing the blame game and I'm just trying to figure out what is going on.
Distinguished Expert 2017

Commented:
You have to make sure on your side that should traffic be destined to 66. that it forces the tunnel up while the outside path to the 66 IP is blocked.

i.e. route 66.x.x.x. 255.255.255.255 IP_on the other side of the tunnel
access-list on the outside deny any 66.

Author

Commented:
So add a route 66.x.x.x (the other side) 255.255.255.255?
Distinguished Expert 2017

Commented:
you can try route 66 255.255.255.255 other side as the destiantion gateway ip

show crypto iskamp
show crypto sa

is 66. reflected as interesting traffic in the VPN tunnel?

Because of what is being sent through, and the HIPAA requirements. there are differnet ways to overlay IPs such that you usually they would have provided a private IP on their side that goes to the end point where they needed simply to avoid using a public IP as they have setup.

i.e. on their side they could have had a 172.16.254.21 as a place holder to 66. IP
on their vpn side, they will switch traffic seen from 66 entering this tunnel to appear as originating from 172.16.254.21 (source address translation)
to complete the packet header rewrite so when your side responds it will go to the correct destination.
your vpn would then use the 172.16.254.21 it would usually (provided you do not use this segment) match the tunnel rules setup the tunnel and let it flows as expected/
TimotiStDatacenter Technician
Top Expert 2012

Commented:
Do you have multiple transform-set options in your IKE settings?
Apparently that can be a problem for ASA 9.x versions.

Also,
show crypto isakmp

Open in new window

should show you who has initiated the tunnel when it's up.

Author

Commented:
I understand what you saying however, I was only give 2 66 ips from the "other Side" Maybe I need to Nat the IP to their inside, if I can get in touch with someone over there to find out how they are set up.

Author

Commented:
So far I've been monitoring the tunnel.  I turned on debugging and it's not gone down.  I'm not sure if someone did something on the other side of this tunnel or it magically stayed up.  It's been up for almost a week and I've done nothing.  I'll keep you all posted. thanks!

Author

Commented:
I've been monitoring this situation for days and finally some logs.  Can someone help me out please
tunnel.png
Distinguished Expert 2017

Commented:
It is hard to see things in an image.
or have context the possible drop deals could deal as noted before because of the duration of a VPN mismatch.

You should confirm that the VPN settings on your side match the ones on the remote

i.e.
Please confirm that the
VPN for X is
WAN IP: X
Local IP segment: x.y.z.0.24
Encryption/encapsulation AES256:.....
Key 3600 seconds
Key lifetime 28800 seconds

To your
WAN IP:
LOCAL IP

no need to include passphrase.

and have them confirm that the two match.

The divergence I suspect deals with the Key lifetime that your time frame is larger than the setting on the other side which might explain why their side terms the VPN connection while your side is not ready to initiate the renegotiation/refresh...

Author

Commented:
This is from my debugging and the 66 address is the address of the other side of the tunnel.  Problem is that side isn't exactly cooperative.  They are playing the "blame" game.  All they keep telling me is they are re-keying the tunnel and nothing more.  It's truly hard to figure this out with only 1 piece of the puzzle.

5|Nov 08 2019|13:14:53|713904|||||IP = 66.220.42.206, Received encrypted packet with no matching SA, dropping
5|Nov 08 2019|13:14:50|713904|||||IP = 66.220.42.206, Received encrypted packet with no matching SA, dropping
5|Nov 08 2019|13:14:49|713904|||||IP = 66.220.42.206, Received encrypted packet with no matching SA, dropping
5|Nov 08 2019|13:20:40|713904|||||IP = 66.220.42.206, Received encrypted packet with no matching SA, dropping
5|Nov 08 2019|13:20:40|713904|||||IP = 66.220.42.206, Received encrypted packet with no matching SA, dropping
4|Nov 08 2019|13:20:38|713903|||||IP = 66.220.42.206, Information Exchange processing failed
5|Nov 08 2019|13:20:38|713904|||||IP = 66.220.42.206, Received an un-encrypted NO_PROPOSAL_CHOSEN notify message, dropping
7|Nov 08 2019|13:20:38|713236|||||IP = 66.220.42.206, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + NOTIFY (11) + NONE (0) total length : 40
7|Nov 08 2019|13:20:38|713236|||||IP = 66.220.42.206, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + NOTIFY (11) + NONE (0) total length : 40
6|Nov 08 2019|13:20:38|713219|||||IP = 66.220.42.206, Queuing KEY-ACQUIRE messages to be processed when P1 SA is complete.
7|Nov 08 2019|13:20:38|713236|||||IP = 66.220.42.206, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 408
7|Nov 08 2019|13:20:38|715046|||||IP = 66.220.42.206, constructing Fragmentation VID + extended capabilities payload
7|Nov 08 2019|13:20:38|715046|||||IP = 66.220.42.206, constructing NAT-Traversal VID ver RFC payload
7|Nov 08 2019|13:20:38|715046|||||IP = 66.220.42.206, constructing NAT-Traversal VID ver 03 payload
7|Nov 08 2019|13:20:38|715046|||||IP = 66.220.42.206, constructing NAT-Traversal VID ver 02 payload
7|Nov 08 2019|13:20:38|715046|||||IP = 66.220.42.206, constructing ISAKMP SA payload
5|Nov 08 2019|13:20:38|713041|||||IP = 66.220.42.206, IKE Initiator: New Phase 1, Intf inside, IKE Peer 66.220.42.206  local Proxy Address 10.75.13.90, remote Proxy Address 66.220.42.201,  Crypto map (outside_map)
7|Nov 08 2019|13:21:10|713906|||||IP = 66.220.42.206, sending delete/delete with reason message
7|Nov 08 2019|13:21:10|713906|||||IP = 66.220.42.206, IKE SA MM:1574080c terminating:  flags 0x01000022, refcnt 0, tuncnt 0
7|Nov 08 2019|13:21:10|715065|||||IP = 66.220.42.206, IKE MM Initiator FSM error history (struct &0xac48a4a0)  <state>, <event>:  MM_DONE, EV_ERROR-->MM_WAIT_MSG2, EV_RETRY-->MM_WAIT_MSG2, EV_TIMEOUT-->MM_WAIT_MSG2, NullEvent-->MM_SND_MSG1, EV_SND_MSG-->MM_SND_MSG1, EV_START_TMR-->MM_SND_MSG1, EV_RESEND_MSG-->MM_WAIT_MSG2, EV_RETRY
7|Nov 08 2019|13:30:46|713236|||||IP = 66.220.42.206, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + NOTIFY (11) + NONE (0) total length : 40
7|Nov 08 2019|13:30:46|713236|||||IP = 66.220.42.206, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + NOTIFY (11) + NONE (0) total length : 40
7|Nov 08 2019|13:30:46|713236|||||IP = 66.220.42.206, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 408
5|Nov 08 2019|13:30:43|713904|||||IP = 66.220.42.206, Received encrypted packet with no matching SA, dropping
5|Nov 08 2019|13:30:43|713904|||||IP = 66.220.42.206, Received encrypted packet with no matching SA, dropping
6|Nov 08 2019|13:30:41|713219|||||IP = 66.220.42.206, Queuing KEY-ACQUIRE messages to be processed when P1 SA is complete.
6|Nov 08 2019|13:30:41|713219|||||IP = 66.220.42.206, Queuing KEY-ACQUIRE messages to be processed when P1 SA is complete.
5|Nov 08 2019|13:30:40|713904|||||IP = 66.220.42.206, Received encrypted packet with no matching SA, dropping
5|Nov 08 2019|13:30:40|713904|||||IP = 66.220.42.206, Received encrypted packet with no matching SA, dropping
4|Nov 08 2019|13:30:38|713903|||||IP = 66.220.42.206, Information Exchange processing failed
5|Nov 08 2019|13:30:38|713904|||||IP = 66.220.42.206, Received an un-encrypted NO_PROPOSAL_CHOSEN notify message, dropping
7|Nov 08 2019|13:30:38|713236|||||IP = 66.220.42.206, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + NOTIFY (11) + NONE (0) total length : 40
7|Nov 08 2019|13:30:38|713236|||||IP = 66.220.42.206, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + NOTIFY (11) + NONE (0) total length : 40
6|Nov 08 2019|13:30:38|713219|||||IP = 66.220.42.206, Queuing KEY-ACQUIRE messages to be processed when P1 SA is complete.
7|Nov 08 2019|13:30:38|713236|||||IP = 66.220.42.206, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 408
7|Nov 08 2019|13:30:38|715046|||||IP = 66.220.42.206, constructing Fragmentation VID + extended capabilities payload
7|Nov 08 2019|13:30:38|715046|||||IP = 66.220.42.206, constructing NAT-Traversal VID ver RFC payload
7|Nov 08 2019|13:30:38|715046|||||IP = 66.220.42.206, constructing NAT-Traversal VID ver 03 payload
7|Nov 08 2019|13:30:38|715046|||||IP = 66.220.42.206, constructing NAT-Traversal VID ver 02 payload
7|Nov 08 2019|13:30:38|715046|||||IP = 66.220.42.206, constructing ISAKMP SA payload
5|Nov 08 2019|13:30:38|713041|||||IP = 66.220.42.206, IKE Initiator: New Phase 1, Intf inside, IKE Peer 66.220.42.206  local Proxy Address 10.75.13.90, remote Proxy Address 66.220.42.201,  Crypto map (outside_map)
Distinguished Expert 2017

Commented:
you should ask them to cofirm the VPN settings or ask them what their VPN settings in the aggregate.

Another thing to check is whether they have two paths/public IP.

I ran into a situation along a similar path. The VPN was setup. The VPN is up and operational. The VPN was setup looking out into the future such that additional IPS were included as part of the VPN.
The future arrived and the system with the IP that was included was brought up and from it the VPN appeared to be intermittently unavailable.

As we all know, if the question is not posed in the right way the answer we get is what we knew.

I kept saying that the IP that was part of the VPN setup is having issues periodically from being able to access through the VPN. They as your opposites kept saying that the IP is part of the VPN setup.
In my case, I had another VPN that needed to be modified with another vendor who had a different style VPN setup where they list individual IPs within the VPN setup /32 blocks.
So taking that approach I rephrased the prior question and simply provided that the VPN configuration from my side is
Your WAN IP:
Your LAN block/IPs

My WAN:
MY LAN IPs in CDR
IPsegment1/29
ipsegment2/30

phase 1 3600s
phase 2 key lifetime
please confirm you have the same.

Well, since I provided that, their reply was that they had instead of ipsegment2/30
they had the individual IPS in a /32
which meant as long as I ping their side from that system to maintain the VPN connection, their side would not trigger to bring up the VPN when one of these IPs were the destination.

This is why I think and suggest that you provide your VPN outline and ask them to confirm that it is what they have as well.

You have a mismatch, the finding it is .....

Good luck.

Author

Commented:
Thanks.  That makes sense.  I can't seem to get in touch with this people so I'm stuck right now.  The good news is I'm installing a new ASA and hopefully they will have not choice but to confirm the settings.  I'll keep you posted.
Distinguished Expert 2017

Commented:
IF you have not sent them that you are setting up a new ASA, try sending that you are setting up a different device, sonicwall, checkpoint, juniper, etc. anything other than the ASA.
To see if you get a quicker response.
The other issue it does not seem that these people are responsive to the relationship you/your firm has with them...

Have you tried going up the chain on the other side?
Who is paying whom for service?
IT sounds since you are the one monioring the status of the VPN, that it is important for you.

Author

Commented:
Lets just say it's 3rd party and unfortunately, I have no control.  But thanks.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial