failover with mpls router, cable modem, and asa 5510
We have an MPLS WAN in several cities, mostly T1s, and corporate has Metro Ethernet. All have service provider routers (default), ASA 5510 firewalls, and then various Cisco layer 2 switches.
Each site has the typical domain controller, file/print server, and wireless network, and corporate has both intranet and internet servers. One other site ties into corporate with a PTP VPN.
Internet access originally was through the corporate router, ultimately. But, we offloaded Internet traffic to cable and DSL at each site.
Now, Internet traffic routing is done at firewall level, when (as I understand it) the router dutifully kicks back packets not destined for the MPLS network, and the firewall routes them through another interface to the modems (cable/DSL).
MY PROBLEMS?
1. Slow router changes, have to be coordinated with service provider
2. It works, but it's ugly to document, standardize, and maintain
3. No failover capability
MY OPTIONS? (one or more)
1. Adding CPE routers at each site
2. Adding Layer 3 switches at each site
3. Replacing 5510s with 5505s at each site
I haven't decided if I need any firewalling between sites, but I don't think it would hurt, in case of an outbreak. Might be a pain to have it, but...
Any suggestions, recommendations?
Each site has the typical domain controller, file/print server, and wireless network, and corporate has both intranet and internet servers. One other site ties into corporate with a PTP VPN.
Internet access originally was through the corporate router, ultimately. But, we offloaded Internet traffic to cable and DSL at each site.
Now, Internet traffic routing is done at firewall level, when (as I understand it) the router dutifully kicks back packets not destined for the MPLS network, and the firewall routes them through another interface to the modems (cable/DSL).
MY PROBLEMS?
1. Slow router changes, have to be coordinated with service provider
2. It works, but it's ugly to document, standardize, and maintain
3. No failover capability
MY OPTIONS? (one or more)
1. Adding CPE routers at each site
2. Adding Layer 3 switches at each site
3. Replacing 5510s with 5505s at each site
I haven't decided if I need any firewalling between sites, but I don't think it would hurt, in case of an outbreak. Might be a pain to have it, but...
Any suggestions, recommendations?
ASKER
Already doing that with a few of our applications. But I save a lot of money using cable/DSL for offloading all the Internet traffic at each site, and I still have to find a way to automatically failover that route when it fails, to the MPLS network. And that's the part that complicated the firewall configurations. I'd like a more business class modem/router infrrastructure also, than the consumer grade junk they slip into our uplink.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Should we keep their routers inline, or swap ours in place? Recommended router for what we need to do? Recommended modems or router modules for cable and DSL, or should we live with them going down every 6 weeks and be satisfied with failover?
The 5510s are overkill, but the additional 5505 Ethernet interfaces was the primary consideration for downgrading.
Failover for Internet was the main goal, as creating a failover PTP VPN tunnel mesh seemed rather daunting--but might do if it's a common and straightforward thing (not familiar with GPE, not that I'd be doing this myself of course, I just want to know the options to discuss with partners).
So it sounds like to simplify (stop playing routing games in our firewall) our architecture and provide failover (at least for our offloaded Internet traffic) you suggest we put in our own router (tbd), remove theirs, and handle the routing anf failover there using native routing protocols and failover methods, and maybe some additional VPN tunnels on the local Internet link, in case we want resiliency and failover for the MPLS...? For the site that has VPN already, we own that router, so just need to add a better external switch, and maybe add a layer 3 at the core?
So we'd have the T1 and cable/DSL lines coming into an external switch, then into the firewall, then into the core switch? And then we wouldn't need any layer 3 switches at the core? Or do you recommend we add that capability as part of the reconfiguration, either now or later?
If I already have a "normalized" architecture with the router fix above, am I really going to get much bang for my buck adding a Layer 3 capability at each site? Or should I really only pursue that if we want to keep service provider routers for the MPLS link?
The 5510s are overkill, but the additional 5505 Ethernet interfaces was the primary consideration for downgrading.
Failover for Internet was the main goal, as creating a failover PTP VPN tunnel mesh seemed rather daunting--but might do if it's a common and straightforward thing (not familiar with GPE, not that I'd be doing this myself of course, I just want to know the options to discuss with partners).
So it sounds like to simplify (stop playing routing games in our firewall) our architecture and provide failover (at least for our offloaded Internet traffic) you suggest we put in our own router (tbd), remove theirs, and handle the routing anf failover there using native routing protocols and failover methods, and maybe some additional VPN tunnels on the local Internet link, in case we want resiliency and failover for the MPLS...? For the site that has VPN already, we own that router, so just need to add a better external switch, and maybe add a layer 3 at the core?
So we'd have the T1 and cable/DSL lines coming into an external switch, then into the firewall, then into the core switch? And then we wouldn't need any layer 3 switches at the core? Or do you recommend we add that capability as part of the reconfiguration, either now or later?
If I already have a "normalized" architecture with the router fix above, am I really going to get much bang for my buck adding a Layer 3 capability at each site? Or should I really only pursue that if we want to keep service provider routers for the MPLS link?
Your set up is good. Use the ISP MPLS for primary accees to corporate resources.
Use cable modem for Internet access and for VPN between the offices as backup.
Create Site-to-Site VPN between your offices using the C5510 firewall. Talk to the service provider. They will make changes to the router so that the VPN will be used for failover. This is common practice for most companies snd setup is easy. It can be done here for you. Just post a question with details and it will be done.
Use cable modem for Internet access and for VPN between the offices as backup.
Create Site-to-Site VPN between your offices using the C5510 firewall. Talk to the service provider. They will make changes to the router so that the VPN will be used for failover. This is common practice for most companies snd setup is easy. It can be done here for you. Just post a question with details and it will be done.
ASKER
yawbe, you're saying we should leave everything as is? in other words, keep using our ASA as our router, basically, and not add our own router or L3? but, add a VPN link over the local Internet connection, and then work with our ISP and firewall consultant to add the necessary SLA settings to manage the failover? I don't really care at this point about the VPN and failing over the MPLS traffic--that is pretty robust. I really just want to keep the Internet up, via the MPLS route, when the POS cable modem goes off to lala land. But I need it to be dynamic--I can't wait for a site to lose their cable, then call us, so we can call our consultant and the ISP to change all the routing.
ASKER
We use static routing on the MPLS side, I believe, so there's no dynamic or learning involved. This is basically what I guess everyone is recommending, although I was really only looking to failover the Cable/DSL links to MPLS. (and move/simplify the routing, be using the router instead of the firewall)
MPLS sites
192.168.x.* L2 LAN (local traffic)
192.168.*.* primary MPLS WAN, SLA/failover GRE/PPTP tunnel over Cable/DSL
*.*.*.* primary Cable/DSL (no VPN), failover to MPLS WAN (no VPN)
Non-MPLS sites
192.168.x.* L2 LAN (local traffic)
192.168.*.* primary GRE/PPTP tunnel over Cable, SLA/failover to GRP/PPTP tunnel over DSL
*.*.*.* primary Cable, failover to DSL (no VPN)
MPLS sites
192.168.x.* L2 LAN (local traffic)
192.168.*.* primary MPLS WAN, SLA/failover GRE/PPTP tunnel over Cable/DSL
*.*.*.* primary Cable/DSL (no VPN), failover to MPLS WAN (no VPN)
Non-MPLS sites
192.168.x.* L2 LAN (local traffic)
192.168.*.* primary GRE/PPTP tunnel over Cable, SLA/failover to GRP/PPTP tunnel over DSL
*.*.*.* primary Cable, failover to DSL (no VPN)
ASKER
I found the links below, which have solutions. I guess it really comes down to the fact that I can do it multiple ways, through L3 switching, ASA/Pix firewalls, or routers.
My question then really goes to WHICH way I should go based on what I have in place, and without wasting money, overkill, etc. I want to swat the fly with the right size swatter, not a shotgun, much less a top-of-the-line Beretta shotgun.
I THINK (as gavvin alludes) that I should implement my own routers and do it there, and use the ASA just for firewalling. It simplifies my architecture and gives me more control. And I can add L3 just for additional benefits not related to ISP failover routing.
http://www.cisco.com/image/gif/paws/70559/pix-dual-isp.pdf
http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_configuration_example09186a00806e880b.shtml
http://www.cisco.com/en/US/prod/collateral/routers/ps5855/white_paper_c11_472858.html
My question then really goes to WHICH way I should go based on what I have in place, and without wasting money, overkill, etc. I want to swat the fly with the right size swatter, not a shotgun, much less a top-of-the-line Beretta shotgun.
I THINK (as gavvin alludes) that I should implement my own routers and do it there, and use the ASA just for firewalling. It simplifies my architecture and gives me more control. And I can add L3 just for additional benefits not related to ISP failover routing.
http://www.cisco.com/image/gif/paws/70559/pix-dual-isp.pdf
http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_configuration_example09186a00806e880b.shtml
http://www.cisco.com/en/US/prod/collateral/routers/ps5855/white_paper_c11_472858.html
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
You had me, then lost me, about the dual ISP not working.
For internet failover, I'm talking about each remote site using its own T1 MPLS connection up and out through the HQ router (which was the original route before adding all the remote "cheap" Internet links), when their cable/DSL modem goes down (like it did last week).
Yes, they're vendor controlled now, but don't have any failover built in now. If that can be done easily enough, without having to add our own router or L3 switch, that's fine. I just don't like the current setup, using our firewall as a strange router, and the vendor (default) router kicking back our Internet traffic, for the firewall to then reroute over another interface to the cable/DSL modems. It works, but it's ugly, and not dynamic enough to handle failover (as far as I can tell). I hadn't thought about the contract issue, I was assuming they'd gladly let us handle that ourselves and free up their equipment, or keep it inline with no changes.
Outbound traffic is fine, VPN is the only thing we have Internet-facing at remote sites, and losing that temporarily is acceptable.
One important factor in this decision will be your contract with your MPLS provider. Are you currently on a long term contract? They generally will charge you a signficant fee to change service levels so that you can use your own routers to terminate the connection rather than their equipment. Also if you're using the MPLS for voice PBX connection the provider could have a custom configuration that you may not want to have to administrate. If you're on month-to-month or can renegotiate the contract, then moving off of managed routers will give you alot more flexibility in the long run.
"FYI the dual-ISP examples aren't going to apply if you're trying to configure it like I've laid out." I thought dual ISP configurations were possible with both L3 switches, our existing firewall, or routers. Yeh, I'm surprised there's no good documentation on such a setup, since there are LOTS of growing companies in this kind of situation, trying to offload Internet traffic to leverage and extend their much more expensive T1 connections for more critical MPLS and WAN traffic, but still needing some failover capability for Internet traffic. Prices are dropping, but not that quickly. I guess we could add both cable AND DSL at each remote site, and load balance or failover between those two for Internet traffic, but that's additional recurring costs, not less. Plus, it would probably take another ASA, and not all locations have cable.
For internet failover, I'm talking about each remote site using its own T1 MPLS connection up and out through the HQ router (which was the original route before adding all the remote "cheap" Internet links), when their cable/DSL modem goes down (like it did last week).
Yes, they're vendor controlled now, but don't have any failover built in now. If that can be done easily enough, without having to add our own router or L3 switch, that's fine. I just don't like the current setup, using our firewall as a strange router, and the vendor (default) router kicking back our Internet traffic, for the firewall to then reroute over another interface to the cable/DSL modems. It works, but it's ugly, and not dynamic enough to handle failover (as far as I can tell). I hadn't thought about the contract issue, I was assuming they'd gladly let us handle that ourselves and free up their equipment, or keep it inline with no changes.
Outbound traffic is fine, VPN is the only thing we have Internet-facing at remote sites, and losing that temporarily is acceptable.
One important factor in this decision will be your contract with your MPLS provider. Are you currently on a long term contract? They generally will charge you a signficant fee to change service levels so that you can use your own routers to terminate the connection rather than their equipment. Also if you're using the MPLS for voice PBX connection the provider could have a custom configuration that you may not want to have to administrate. If you're on month-to-month or can renegotiate the contract, then moving off of managed routers will give you alot more flexibility in the long run.
"FYI the dual-ISP examples aren't going to apply if you're trying to configure it like I've laid out." I thought dual ISP configurations were possible with both L3 switches, our existing firewall, or routers. Yeh, I'm surprised there's no good documentation on such a setup, since there are LOTS of growing companies in this kind of situation, trying to offload Internet traffic to leverage and extend their much more expensive T1 connections for more critical MPLS and WAN traffic, but still needing some failover capability for Internet traffic. Prices are dropping, but not that quickly. I guess we could add both cable AND DSL at each remote site, and load balance or failover between those two for Internet traffic, but that's additional recurring costs, not less. Plus, it would probably take another ASA, and not all locations have cable.
Generally once you sign a contract for a managed solution, they're quite resistant to switching it to a non-managed solution. They have to re-provision the connection, remove their equipment, and usually the charges are less, which if you're under a standard 3-year contract, they'll want to charge you a fee. It'll vary from vendor to vendor, I'm mainly speaking of experiences with AT&T.
The Dual-ISP solutions are based upon having 2 ISP connections at a specific location. Not using both an ISP Internet connection and an internal WAN connection for failover. You can do failover pretty easily with 2 ISP Internet connections (DSL/Cable) at one site, with 1 ASA, but true load-balancing isn't that workable with the ASA platform.
As for documentation on the setup, it's a rather specific customized solution that requires quite a bit of work to get setup and working correctly. But you're right, I've seen the request multiple times for both WAN failover over Internet, or Internet failover over WAN.
The Dual-ISP solutions are based upon having 2 ISP connections at a specific location. Not using both an ISP Internet connection and an internal WAN connection for failover. You can do failover pretty easily with 2 ISP Internet connections (DSL/Cable) at one site, with 1 ASA, but true load-balancing isn't that workable with the ASA platform.
As for documentation on the setup, it's a rather specific customized solution that requires quite a bit of work to get setup and working correctly. But you're right, I've seen the request multiple times for both WAN failover over Internet, or Internet failover over WAN.
ASKER
Yeh, that's what we're facing, a custom solution with a blend of providers. But that's how we got to where we are today, which I don't think is optimum. I want to avoid another approach that ends up "less than optimum." I guess I'm surprised that despite the maturity of networking and Cisco in general, that there's not a preferred method (equipment, design, protocols) for something so basic an architecture and so common a requirement and set of constraints.
In server terms, I'd be able to quickly spec out several options (physical/virtual, etc.) and make prioritized recommendations, with detailed tradeoffs and pros/cons of each approach. I think I hear that routers are the better route to go than L3 switching for the failover, but I've even read arguments about that. I don't think anyone would argue that firewall-based routing is the best way to go, but it's what certified professionals implemented (maybe as the path of least resistance/cost, etc.), and I've even had someone say "leave it as is," regardless of how difficult to easily and promptly maintain, with low risk, for anyone but the designer.
This just seems pretty cookie cutter to me, and I'm surprised there aren't at least a few cookie-cutter solutions to what a lot of people need to do the most, have affordable consumer-grade Internet with failover to a business-class WAN. I'm still not even sure from the last two comments whether it's possible to do it with 2 ISPs (MPLS and cable or DSL), or if every place out there has to have at least 3 (1 MPLS not-to-be-mucked-with, and cable, and DSL).
I guess my best route is to just engage multiple designers, and see if they pick the same solution or come through with something more robust and that seems more standardized, and choose accordingly.
Thanks, Gavvin!
In server terms, I'd be able to quickly spec out several options (physical/virtual, etc.) and make prioritized recommendations, with detailed tradeoffs and pros/cons of each approach. I think I hear that routers are the better route to go than L3 switching for the failover, but I've even read arguments about that. I don't think anyone would argue that firewall-based routing is the best way to go, but it's what certified professionals implemented (maybe as the path of least resistance/cost, etc.), and I've even had someone say "leave it as is," regardless of how difficult to easily and promptly maintain, with low risk, for anyone but the designer.
This just seems pretty cookie cutter to me, and I'm surprised there aren't at least a few cookie-cutter solutions to what a lot of people need to do the most, have affordable consumer-grade Internet with failover to a business-class WAN. I'm still not even sure from the last two comments whether it's possible to do it with 2 ISPs (MPLS and cable or DSL), or if every place out there has to have at least 3 (1 MPLS not-to-be-mucked-with, and cable, and DSL).
I guess my best route is to just engage multiple designers, and see if they pick the same solution or come through with something more robust and that seems more standardized, and choose accordingly.
Thanks, Gavvin!
ASKER
All other things being equal, this seems like a pretty standard and common requirement, which cries out for a less "customized" or variable solutions in terms of approaches, options, equipment, protocol, etc,, with at least an outline of the different risks, pros/cons, and maybe with prioritized recommendations: given the pros and cons of X, Y, and Z, I'd recommend Y (under these conditions), then Z (under a different set of conditions), then Z (as a last resort, and then only if yaddah yaddah). It's always a given that I can go talk to my provider, or my Cisco reseller and they'll "work something out." But it's nice to have some level of confidence that you're going in the right direction with the solutions they introduce, and changes they recommend based on these types of "new" business requirements.
ASKER
Isn't policy-based routing on our provider's CPE the best route to go?
http://www.petri.co.il/how-to-use-cisco-ios-policy-based-routing-features.htm
http://www.petri.co.il/how-to-use-cisco-ios-policy-based-routing-features.htm
Policy based routing is useful when you're trying to split traffic between connections. So if you have 2 ISPs, you could have one router do policy based routing to decide which ISP to send the traffic outbound on. It's not really used for failover, if you wanted to implement failover then you have to create SLA or track rules in the router config that cause the behavior to change if one of the ISPs is not active.
This would really simplify your network setup and all Internet could go through the main office.