Posted on 2009-05-18
We have 3 Cisco ASA 5520's connected to an HP Procurve 2824, which is serving as our Internet switch. One of the ASA's is serving as the primary firewall, one is dedicated to hosting lan-2-lan tunnels, and the 3rd was installed simply for troubleshooting this particular issue.
Our colo is providing active/standby Internet in the form of VRRP -- we have 2 cat5 plugging into the our internet switch. The VRRP and outside IPs on our ASAs are all within the same subnet.
The VPN and test ASAs are intermittently dropping packets -- They will be fine for 5-10 minutes, then will drop sometimes as much as 30-50% of packets for a period of 30 seconds to several minutes. Often, but not always, the two ASAs will drop packets simultaneously. The other ASA (primary firewall, hosting 95% of the traffic) never drops *ANY* packets!!
The drops are occuring not only for traffic both passing through the ASAs, but also for pings initiated directly from the ASA, so I think it safe to assume that it is not a configuration issue -- I have gone so far as to back out all of the config except for what is necessary to test connectivity. And yes, *ALL* devices on both sides are hard-set to 100/Full.
Other Troubleshooting I've done:
1) Replaced all the cables
2) Tried different ports on the internet switch
3) Confirmed no errors of any sort on either the switch or the firewalls
4) Changed the default gateway on the VPN ASA to the standby internet router instead of the VRRP IP....this seems to cure the problem! The "test" ASA, still pointing at the VRRP is still dropping packets! However, the primary firewall ASA is also pointed at the VRRP and is *NOT* dropping packets?
I'm at a loss....datacenter thinks its our ProCurve...they've claimed multiple customers with random packetloss when using ProCurves as Internet switches...The datacenter is terminating the Internet feeds they hand us on Foundry.