We help IT Professionals succeed at work.

Changing Subnets Causes VPN tunnels to fail

RPPreacher
RPPreacher asked
on
We have 16 offices with a full mesh VPN (every office connects to every other office).  Normally when we add a subnet to an office, we make the appropriate changes to the no nat ACL, the cryptomap ACL, and stop and start the cryptomap function.  This has always worked.

Recently when we do this, the tunnel works for a while but after several hours stops passing 1 or more subnets 1 direction.  We may see flapping tunnels on all subnets between two offices.  In a recent incident, Office A/Subnet 1 could ping Office B/All Subnets, Office B/Subnet 1 could ping Office A/All subnets, Office B/subnet 2 could ping Office A/subnet 1 but Office B/Subnet 2 could not ping Office A/subnet 1.

A reload does NOT solve the flapping/subnet black hole but a hard power cycle does.

Is the PIX have problems processing the ACL because of the length?  Do we need to move to turbo ACLs or is there some other issue?
Comment
Watch Question

Wow, that oughta be a big acl entry. How about connection stats? Are they exceeding whatever is supported on any of the PIX's ?

You could turn on the turboacl feature if you have enough memory and it sure would improve performance.

Cheers,
Rajesh

Author

Commented:
A show crypto ipsec sa shows the tunnels for all subnets up; however, packets are never encrypted (for the failing subnet).
Again, did you check the connection limit?

Cheers,
Rajesh

Author

Commented:
The 506E has a max of 25 peers.  We have 17.

Author

Commented:
They are all 506E or better.
You might be having less number of peers but connection/performance limit is something that you can hit with those itself;

Performance Summary
Cleartext throughput: Up to 100 Mbps
Concurrent connections: 25,000
56-bit DES IPSec VPN throughput: Up to 20 Mbps
168-bit 3DES IPSec VPN throughput: Up to 16 Mbps
128-bit AES IPSec VPN throughput: Up to 30 Mbps
256-bit AES IPSec VPN throughput: Up to 25 Mbps
Simultaneous VPN peers: 25*
* Maximum number of simultaneous site-to-site or remote access IKE Security Associations (SAs) supported

The above is what is supported on a 506E.

So if you are having 168-bit 3DES, the maximum VPN throughput can only be 16 Mbps + your normal internet traffic. It could be that the box is so small to handle so many vpns.

Cheers,
Rajesh

Author

Commented:
unlikely.

T1 at each office.  No more than 1.5 Mbps in or out.
There is one way to find it out easily if you have the CCO login (Unfortunately I don't have one).

If you go to cisco.com and tools-> there is an excellent tool (output intrepreter).

Do a 'show tech' on one of the box where you see this problem and paste the whole output onto output intrepreter to see how the CPU/Memory/Bandwidth is behaving and it would eliminate the problem if it is the horsepower of the device.

Cheers,
Rajesh

Author

Commented:
And all of your suggestions do not address the problem:

Everything was working.  No new sites.

We added a subnet to each site.  No new traffic, just a new subnet.

Furthermore, why would hard powering cycling the PIX magically fix a connection limitation?
Cisco had us shorten the ACLs by changing to network-objects.  This resolved the issue.

Explore More ContentExplore courses, solutions, and other research materials related to this topic.