Here is the basic idea – 3 sites – NY, Fl, NJ - NY is the hub and NJ and Fl are spokes. Normal operation is as follows – NJ and FL are connected to NY via point to point T1’s with Cisco 2600’s at the sites to a Cisco 3600 at the NY site – the 3600 is then connected to a Cisco Gbic Aggregator which does of the vlans for the NY site – the Aggregator then goes to a Watchguard Firebox firewall/VPN server and out to the internet over a 4.5 mbps pipe. So remote sites are connected to the hub with point to point T1’s and use the hub’s internet connectivity. This all works fine and is running well. If none of the point to point T1’s ever go down, the next paragraph will not be an issue.
This is the issue – goal is to have redundancy at both remote locations for both internet access and site to site connectivity to the NY office. So, both remote sites have additional internet T1’s and an identical Watchguard Firbox/VPN server. Both of the 2600’s at the remote sites have a primary gateway of last resort as the 3600 back in the NY office. This works fine. They also have another gateway of last resort set with a metric of 100 – so it will be “secondary.” This is the trusted interface on the Watchguard so it will be used as a backup in the event that the point to points go down. In theory, if I shutdown the serial interface on the point to point connectivity on the 2600 at either of the remote sites the 2600 should then route all packets to the route with a metric of 100 as the main route has become unavailable. Is this correct? I am finding the rollover test process to be very rough to say the least. It is proving to be very sloppy when the point to point T1 drops and it “rolls” to the backup internet connectivity / VPN to the NY office. I am finding reboots of Windows machines, reassignment of gateways are necessary among other anomalies.
Without getting too detailed and going on too long – is the basic engineering premise behind this backup solution sound? Does it seem like it should work? If you were the engineer how would you accomplish this goal of internet and remote site to NY site redundancy ?