Changing Subnets Causes VPN tunnels to fail
We have 16 offices with a full mesh VPN (every office connects to every other office). Normally when we add a subnet to an office, we make the appropriate changes to the no nat ACL, the cryptomap ACL, and stop and start the cryptomap function. This has always worked.
Recently when we do this, the tunnel works for a while but after several hours stops passing 1 or more subnets 1 direction. We may see flapping tunnels on all subnets between two offices. In a recent incident, Office A/Subnet 1 could ping Office B/All Subnets, Office B/Subnet 1 could ping Office A/All subnets, Office B/subnet 2 could ping Office A/subnet 1 but Office B/Subnet 2 could not ping Office A/subnet 1.
A reload does NOT solve the flapping/subnet black hole but a hard power cycle does.
Is the PIX have problems processing the ACL because of the length? Do we need to move to turbo ACLs or is there some other issue?
Recently when we do this, the tunnel works for a while but after several hours stops passing 1 or more subnets 1 direction. We may see flapping tunnels on all subnets between two offices. In a recent incident, Office A/Subnet 1 could ping Office B/All Subnets, Office B/Subnet 1 could ping Office A/All subnets, Office B/subnet 2 could ping Office A/subnet 1 but Office B/Subnet 2 could not ping Office A/subnet 1.
A reload does NOT solve the flapping/subnet black hole but a hard power cycle does.
Is the PIX have problems processing the ACL because of the length? Do we need to move to turbo ACLs or is there some other issue?
ASKER
A show crypto ipsec sa shows the tunnels for all subnets up; however, packets are never encrypted (for the failing subnet).
Again, did you check the connection limit?
Cheers,
Rajesh
Cheers,
Rajesh
ASKER
The 506E has a max of 25 peers. We have 17.
ASKER
They are all 506E or better.
You might be having less number of peers but connection/performance limit is something that you can hit with those itself;
Performance Summary
Cleartext throughput: Up to 100 Mbps
Concurrent connections: 25,000
56-bit DES IPSec VPN throughput: Up to 20 Mbps
168-bit 3DES IPSec VPN throughput: Up to 16 Mbps
128-bit AES IPSec VPN throughput: Up to 30 Mbps
256-bit AES IPSec VPN throughput: Up to 25 Mbps
Simultaneous VPN peers: 25*
* Maximum number of simultaneous site-to-site or remote access IKE Security Associations (SAs) supported
The above is what is supported on a 506E.
So if you are having 168-bit 3DES, the maximum VPN throughput can only be 16 Mbps + your normal internet traffic. It could be that the box is so small to handle so many vpns.
Cheers,
Rajesh
Performance Summary
Cleartext throughput: Up to 100 Mbps
Concurrent connections: 25,000
56-bit DES IPSec VPN throughput: Up to 20 Mbps
168-bit 3DES IPSec VPN throughput: Up to 16 Mbps
128-bit AES IPSec VPN throughput: Up to 30 Mbps
256-bit AES IPSec VPN throughput: Up to 25 Mbps
Simultaneous VPN peers: 25*
* Maximum number of simultaneous site-to-site or remote access IKE Security Associations (SAs) supported
The above is what is supported on a 506E.
So if you are having 168-bit 3DES, the maximum VPN throughput can only be 16 Mbps + your normal internet traffic. It could be that the box is so small to handle so many vpns.
Cheers,
Rajesh
ASKER
unlikely.
T1 at each office. No more than 1.5 Mbps in or out.
T1 at each office. No more than 1.5 Mbps in or out.
There is one way to find it out easily if you have the CCO login (Unfortunately I don't have one).
If you go to cisco.com and tools-> there is an excellent tool (output intrepreter).
Do a 'show tech' on one of the box where you see this problem and paste the whole output onto output intrepreter to see how the CPU/Memory/Bandwidth is behaving and it would eliminate the problem if it is the horsepower of the device.
Cheers,
Rajesh
If you go to cisco.com and tools-> there is an excellent tool (output intrepreter).
Do a 'show tech' on one of the box where you see this problem and paste the whole output onto output intrepreter to see how the CPU/Memory/Bandwidth is behaving and it would eliminate the problem if it is the horsepower of the device.
Cheers,
Rajesh
ASKER
And all of your suggestions do not address the problem:
Everything was working. No new sites.
We added a subnet to each site. No new traffic, just a new subnet.
Furthermore, why would hard powering cycling the PIX magically fix a connection limitation?
Everything was working. No new sites.
We added a subnet to each site. No new traffic, just a new subnet.
Furthermore, why would hard powering cycling the PIX magically fix a connection limitation?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
You could turn on the turboacl feature if you have enough memory and it sure would improve performance.
Cheers,
Rajesh