failover with mpls router, cable modem, and asa 5510

We have an MPLS WAN in several cities, mostly T1s, and corporate has Metro Ethernet.  All have service provider routers (default), ASA 5510 firewalls, and then various Cisco layer 2 switches.

Each site has the typical domain controller, file/print server, and wireless network, and corporate  has both intranet and internet servers.  One other site ties into corporate with a PTP VPN.

Internet access originally was through the corporate router, ultimately.  But, we offloaded Internet traffic to cable and DSL at each site.

Now, Internet traffic routing is done at firewall level, when (as I understand it) the router dutifully kicks back packets not destined for the MPLS network, and the firewall routes them through another interface to the modems (cable/DSL).

1.  Slow router changes, have to be coordinated with service provider
2.  It works, but it's ugly to document, standardize, and maintain
3.  No failover capability

MY OPTIONS? (one or more)
1.  Adding CPE routers at each site
2.  Adding Layer 3 switches at each site
3.  Replacing 5510s with 5505s at each site

I haven't decided if I need any firewalling between sites, but I don't think it would hurt, in case of an outbreak.  Might be a pain to have it, but...

Any suggestions, recommendations?

Who is Participating?

Improve company productivity with a Business Account.Sign Up

gavvingConnect With a Mentor Commented:
If you want to implement failover, you're probably going to have to replace the service provider routers with your own.  There's just so many limitations that occur when you don't have access to the routers, and there are so many options that open up when you do.  

Lets say that you own the routers, and want to have failover for the MPLS through the internet, and maybe even failover for the internet through the MPLS.  (Layer-3 switches aren't required, but are nice to have)  One possible design is to terminate the MPLS connections with the routers, have the routers be the default gateway.  The routers use a routing protocol through the MPLS cloud (BGP, EIGRP, etc) to auto learn routes and default gateway routes.  You then configure GRE tunnels from the remote site routers to the core router that are explicitly routed through IPsec tunnels terminated on the ASA firewalls.  This gives the routers dual path connectivity, and auto failover in the event that the MPLS is down.  I've configured this a couple of times, and it generally works great.  

If you don't replace the routers and have to continue to use the service provider routers, then you'll likely have to use layer-3 switches or additional routers at each location to terminate GRE tunnels through the MPLS and the Internet so that you can get routing updates.  

You can accomplish alot of this with SLA configurations, but it's more complex and not as elegant as a routing protocol solution in my opinion.  

This is just one possible way to design it.  There are others as well.  Oh and I don't see why you'd replace the 5510's with 5505's... That doesn't gain you anything...  5510's might be overkill for remote sites without any VPN connections and Internet connections of less than 10mbit though.  As for firewalling between the sites, unless you have special needs, I'd probably just depend on router ACLs and not implement additional internal firewalls.

Matt VCommented:
No chance you can consolidate your servers to the main location and user Citrix XenApp or Windows 2008 Terminal Services?

This would really simplify your network setup and all Internet could go through the main office.
StreetGlidingAwayAuthor Commented:
Already doing that with a few of our applications.  But I save a lot of money using cable/DSL for offloading all the Internet traffic at each site, and I still have to find a way to automatically failover that route when it fails, to the MPLS network.  And that's the part that complicated the firewall configurations.  I'd like a more business class modem/router infrrastructure also, than the consumer grade junk they slip into our uplink.
What Kind of Coding Program is Right for You?

There are many ways to learn to code these days. From coding bootcamps like Flatiron School to online courses to totally free beginner resources. The best way to learn to code depends on many factors, but the most important one is you. See what course is best for you.

StreetGlidingAwayAuthor Commented:
Should we keep their routers inline, or swap ours in place?  Recommended router for what we need to do?  Recommended modems or router modules for cable and DSL, or should we live with them going down every 6 weeks and be satisfied with failover?  

The 5510s are overkill, but the additional 5505 Ethernet interfaces was the primary consideration for downgrading.

Failover for Internet was the main goal, as creating a failover PTP VPN tunnel mesh seemed rather daunting--but might do if it's a common and straightforward thing (not familiar with GPE, not that I'd be doing this myself of course, I just want to know the options to discuss with partners).

So it sounds like to simplify (stop playing routing games in our firewall) our architecture and provide failover (at least for our offloaded Internet traffic) you suggest we put in our own router (tbd), remove theirs, and handle the routing anf failover there using native routing protocols and failover methods, and maybe some additional VPN tunnels on the local Internet link, in case we want resiliency and failover for the MPLS...?  For the site that has VPN already, we own that router, so just need to add a better external switch, and maybe add a layer 3 at the core?

So we'd have the T1 and cable/DSL lines coming into an external switch, then into the firewall, then into the core switch? And then we wouldn't need any layer 3 switches at the core? Or do you recommend we add that capability as part of the reconfiguration, either now or later?

If I already have a "normalized" architecture with the router fix above, am I really going to get much bang for my buck adding a Layer 3 capability at each site? Or should I really only pursue that if we want to keep service provider routers for the MPLS link?
Your set up is good. Use the ISP  MPLS for primary accees to corporate resources.
Use cable modem for Internet access and for VPN between the offices as backup.
Create Site-to-Site VPN between your offices using the C5510 firewall. Talk to the service provider. They will make changes to the router so that the VPN will be used for failover. This is common practice for most companies snd setup is easy. It  can be done here for you. Just post a question with details and it will be done.
StreetGlidingAwayAuthor Commented:
yawbe, you're saying we should leave everything as is?  in other words, keep using our ASA as our router, basically, and not add our own router or L3?  but, add a VPN link over the local Internet connection, and then work with our ISP and firewall consultant to add the necessary SLA settings to manage the failover?  I don't really care at this point about the VPN and failing over the MPLS traffic--that is pretty robust.  I really just want to keep the Internet up, via the MPLS route, when the POS cable modem goes off to lala land.  But I need it to be dynamic--I can't wait for a site to lose their cable, then call us, so we can call our consultant and the ISP to change all the routing.
StreetGlidingAwayAuthor Commented:
We use static routing on the MPLS side, I believe, so there's no dynamic or learning involved.  This is basically what I guess everyone is recommending, although I was really only looking to failover the Cable/DSL links to MPLS.  (and move/simplify the routing, be using the router instead of the firewall)

MPLS sites
192.168.x.* L2 LAN (local traffic)
192.168.*.* primary MPLS WAN, SLA/failover GRE/PPTP tunnel over Cable/DSL
*.*.*.*     primary Cable/DSL (no VPN), failover to MPLS WAN (no VPN)

Non-MPLS sites
192.168.x.* L2 LAN (local traffic)
192.168.*.* primary GRE/PPTP tunnel over Cable, SLA/failover to GRP/PPTP tunnel over DSL
*.*.*.*     primary Cable, failover to DSL (no VPN)
StreetGlidingAwayAuthor Commented:
I found the links below, which have solutions.  I guess it really comes down to the fact that I can do it multiple ways, through L3 switching, ASA/Pix firewalls, or routers.  

My question then really goes to WHICH way I should go based on what I have in place, and without wasting money, overkill, etc.  I want to swat the fly with the right size swatter, not a shotgun, much less a top-of-the-line Beretta shotgun.  

I THINK (as gavvin alludes) that I should implement my own routers and do it there, and use the ASA just for firewalling.  It simplifies my architecture and gives me more control.  And I can add L3 just for additional benefits not related to ISP failover routing.
gavvingConnect With a Mentor Commented:
Well with the information we have here it's difficult to craft a specific solution for you.  I'll bring up some more points.

For internet failover, we're discussing having the internet at one site failover to using the other site's internet right?  Those sites are connected via the MPLS and the vendor controlled MPLS routers, correct?   Unfortunately this sounds a bit easier to do than it is in reality.  The most reasonable way that I can invision to implement this requires a router or layer-3 switch that you control within your network at both sites.  Getting an MPLS provider to configure their equipment to do what we want is a near impossibility (in my experience).  If your MPLS connections between each site are 1-3 T1s in size then an 1841 router could do whats needed.  Another thing to keep in mind with this configuration is that it would only realistically work for outbound user browsing traffic.  If you host websites or other things that require static IPs or DNS entries for access from the Internet, then those are not going to failover correctly.  When it comes to inbound email there are some possibilities with an automatic failover with secondary MX records in DNS.

One important factor in this decision will be your contract with your MPLS provider.  Are you currently on a long term contract?  They generally will charge you a signficant fee to change service levels so that you can use your own routers to terminate the connection rather than their equipment.  Also if you're using the MPLS for voice PBX connection the provider could have a custom configuration that you may not want to have to administrate.  If you're on month-to-month or can renegotiate the contract, then moving off of managed routers will give you alot more flexibility in the long run.  

If your contract is such that it will cost alot of money to change, then you're probably looking at implementing an internal router or layer-3 switch ideally to accomplish this failover functionality.  

Continue with the ASA usage for firewalls as they do the best job of that.  FYI the dual-ISP examples aren't going to apply if you're trying to configure it like I've laid out.

I need to get around to writing up a white paper on this setup myself.  Just haven't gotten around to it.

StreetGlidingAwayAuthor Commented:
You had me, then lost me, about the dual ISP not working.  

For internet failover, I'm talking about each remote site using its own T1 MPLS connection up and out through the HQ router (which was the original route before adding all the remote "cheap" Internet links), when their cable/DSL modem goes down (like it did last week).

Yes, they're vendor controlled now, but don't have any failover built in now.  If that can be done easily enough, without having to add our own router or L3 switch, that's fine.  I just don't like the current setup, using our firewall as a strange router, and the vendor (default) router kicking back our Internet traffic, for the firewall to then reroute over another interface to the cable/DSL modems.  It works, but it's ugly, and not dynamic enough to handle failover (as far as I can tell).  I hadn't thought about the contract issue, I was assuming they'd gladly let us handle that ourselves and free up their equipment, or keep it inline with no changes.

Outbound traffic is fine, VPN is the only thing we have Internet-facing at remote sites, and losing that temporarily is acceptable.  

One important factor in this decision will be your contract with your MPLS provider.  Are you currently on a long term contract?  They generally will charge you a signficant fee to change service levels so that you can use your own routers to terminate the connection rather than their equipment.  Also if you're using the MPLS for voice PBX connection the provider could have a custom configuration that you may not want to have to administrate.  If you're on month-to-month or can renegotiate the contract, then moving off of managed routers will give you alot more flexibility in the long run.  

"FYI the dual-ISP examples aren't going to apply if you're trying to configure it like I've laid out."  I thought dual ISP configurations were possible with both L3 switches, our existing firewall, or routers.  Yeh, I'm surprised there's no good documentation on such a setup, since there are LOTS of growing companies in this kind of situation, trying to offload Internet traffic to leverage and extend their much more expensive T1 connections for more critical MPLS and WAN traffic, but still needing some failover capability for Internet traffic.  Prices are dropping, but not that quickly.  I guess we could add both cable AND DSL at each remote site, and load balance or failover between those two for Internet traffic, but that's additional recurring costs, not less.  Plus, it would probably take another ASA, and not all locations have cable.

Generally once you sign a contract for a managed solution, they're quite resistant to switching it to a non-managed solution.  They have to re-provision the connection, remove their equipment, and usually the charges are less, which if you're under a standard 3-year contract, they'll want to charge you a fee.  It'll vary from vendor to vendor, I'm mainly speaking of experiences with AT&T.

The Dual-ISP solutions are based upon having 2 ISP connections at a specific location.  Not using both an ISP Internet connection and an internal WAN connection for failover.  You can do failover pretty easily with 2 ISP Internet connections (DSL/Cable) at one site, with 1 ASA, but true load-balancing isn't that workable with the ASA platform.

As for documentation on the setup, it's a rather specific customized solution that requires quite a bit of work to get setup and working correctly.  But you're right, I've seen the request multiple times for both WAN failover over Internet, or Internet failover over WAN.
StreetGlidingAwayAuthor Commented:
Yeh, that's what we're facing, a custom solution with a blend of providers.  But that's how we got to where we are today, which I don't think is optimum.  I want to avoid another approach that ends up "less than optimum."  I guess I'm surprised that despite the maturity of networking and Cisco in general, that there's not a preferred method (equipment, design, protocols) for something so basic an architecture and so common a requirement and set of constraints.  

In server terms, I'd be able to quickly spec out several options (physical/virtual, etc.) and make prioritized recommendations, with detailed tradeoffs and pros/cons of each approach.  I think I hear that routers are the better route to go than L3 switching for the failover, but I've even read arguments about that.  I don't think anyone would argue that firewall-based routing is the best way to go, but it's what certified professionals implemented (maybe as the path of least resistance/cost, etc.), and I've even had someone say "leave it as is," regardless of how difficult to easily and promptly maintain, with low risk, for anyone but the designer.  

This just seems pretty cookie cutter to me, and I'm surprised there aren't at least a few cookie-cutter solutions to what a lot of people need to do the most, have affordable consumer-grade Internet with failover to a business-class WAN.  I'm still not even sure from the last two comments whether it's possible to do it with 2 ISPs (MPLS and cable or DSL), or if every place out there has to have at least 3 (1 MPLS not-to-be-mucked-with, and cable, and DSL).

I guess my best route is to just engage multiple designers, and see if they pick the same solution or come through with something more robust and that seems more standardized, and choose accordingly.

Thanks, Gavvin!
StreetGlidingAwayAuthor Commented:
All other things being equal, this seems like a pretty standard and common requirement, which cries out for a less "customized" or variable solutions in terms of approaches, options, equipment, protocol, etc,, with at least an outline of the different risks, pros/cons, and maybe with prioritized recommendations: given the pros and cons of X, Y, and Z, I'd recommend Y (under these conditions), then Z (under a different set of conditions), then Z (as a last resort, and then only if yaddah yaddah).  It's always a given that I can go talk to my provider, or my Cisco reseller and they'll "work something out."  But it's nice to have some level of confidence that you're going in the right direction with the solutions they introduce, and changes they recommend based on these types of "new" business requirements.
StreetGlidingAwayAuthor Commented:
Isn't policy-based routing on our provider's CPE the best route to go? 
Policy based routing is useful when you're trying to split traffic between connections.  So if you have 2 ISPs, you could have one router do policy based routing to decide which ISP to send the traffic outbound on.  It's not really used for failover, if you wanted to implement failover then you have to create SLA or track rules in the router config that cause the behavior to change if one of the ISPs is not active.  
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.