Changing Subnets Causes VPN tunnels to fail

We have 16 offices with a full mesh VPN (every office connects to every other office).  Normally when we add a subnet to an office, we make the appropriate changes to the no nat ACL, the cryptomap ACL, and stop and start the cryptomap function.  This has always worked.

Recently when we do this, the tunnel works for a while but after several hours stops passing 1 or more subnets 1 direction.  We may see flapping tunnels on all subnets between two offices.  In a recent incident, Office A/Subnet 1 could ping Office B/All Subnets, Office B/Subnet 1 could ping Office A/All subnets, Office B/subnet 2 could ping Office A/subnet 1 but Office B/Subnet 2 could not ping Office A/subnet 1.

A reload does NOT solve the flapping/subnet black hole but a hard power cycle does.

Is the PIX have problems processing the ACL because of the length?  Do we need to move to turbo ACLs or is there some other issue?
LVL 20
RPPreacherAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

rsivanandanCommented:
Wow, that oughta be a big acl entry. How about connection stats? Are they exceeding whatever is supported on any of the PIX's ?

You could turn on the turboacl feature if you have enough memory and it sure would improve performance.

Cheers,
Rajesh
0
RPPreacherAuthor Commented:
A show crypto ipsec sa shows the tunnels for all subnets up; however, packets are never encrypted (for the failing subnet).
0
rsivanandanCommented:
Again, did you check the connection limit?

Cheers,
Rajesh
0
The Ultimate Tool Kit for Technolgy Solution Provi

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy for valuable how-to assets including sample agreements, checklists, flowcharts, and more!

RPPreacherAuthor Commented:
The 506E has a max of 25 peers.  We have 17.
0
RPPreacherAuthor Commented:
They are all 506E or better.
0
rsivanandanCommented:
You might be having less number of peers but connection/performance limit is something that you can hit with those itself;

Performance Summary
Cleartext throughput: Up to 100 Mbps
Concurrent connections: 25,000
56-bit DES IPSec VPN throughput: Up to 20 Mbps
168-bit 3DES IPSec VPN throughput: Up to 16 Mbps
128-bit AES IPSec VPN throughput: Up to 30 Mbps
256-bit AES IPSec VPN throughput: Up to 25 Mbps
Simultaneous VPN peers: 25*
* Maximum number of simultaneous site-to-site or remote access IKE Security Associations (SAs) supported

The above is what is supported on a 506E.

So if you are having 168-bit 3DES, the maximum VPN throughput can only be 16 Mbps + your normal internet traffic. It could be that the box is so small to handle so many vpns.

Cheers,
Rajesh
0
RPPreacherAuthor Commented:
unlikely.

T1 at each office.  No more than 1.5 Mbps in or out.
0
rsivanandanCommented:
There is one way to find it out easily if you have the CCO login (Unfortunately I don't have one).

If you go to cisco.com and tools-> there is an excellent tool (output intrepreter).

Do a 'show tech' on one of the box where you see this problem and paste the whole output onto output intrepreter to see how the CPU/Memory/Bandwidth is behaving and it would eliminate the problem if it is the horsepower of the device.

Cheers,
Rajesh
0
RPPreacherAuthor Commented:
And all of your suggestions do not address the problem:

Everything was working.  No new sites.

We added a subnet to each site.  No new traffic, just a new subnet.

Furthermore, why would hard powering cycling the PIX magically fix a connection limitation?
0
RPPreacherAuthor Commented:
Cisco had us shorten the ACLs by changing to network-objects.  This resolved the issue.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Cisco

From novice to tech pro — start learning today.