BGP with static route failover

At the start traffic from Site A properly uses the MPLS to site B. A failure it transitions to the static route (at an AD over 200, weight 0, local preference 90) to use the VPN. The problem is that when the failure is repaired it does *not* fail back, the BGP has to be cleared for it to revert back to the BGP routes instead of the static.

Desired Behavior:
Site A should prefer to use the MPLS circuit to reach site B. In the event site B has circuit problems and it's route advertisement drops, site A should use its static route to pass the traffic over a VPN to site B. Once the circuit is functioning properly again, site A should fail back to the MPLS.

Technical Details:
Site A: 1.22.0.0/24
Site B: 1.16.0.0/24

Hosts -> Cisco L3 switch (BGP AS 65000) that has a connection to a firewall and an MPLS managed router. Site A passes through several BGP ASs to reach site B.

L3 Switch: 1.22.0.1/24 (inside), 1.1.1.242/29 (shared with firewall)
MPLS managed router (BGP AS 65000): 1.22.0.2/24 (inside, neighbor with the L3 switch)
Firewall to Internet: 1.1.1.241/29

ip prefix-list siteb permit 1.16.0.0/24

route-map VPNBACKUP 10
 match ip address prefix-list siteb
 set local-preference 90
 set weight 0

ip route 1.16.0.0 255.255.255.0 1.1.1.241 205

router bgp 65001
 network 1.16.0.0 mask 255.255.255.0 route-map VPNBACKUP


The way I understand the configuration is that the 205 administrative distance on the static route makes it locally less desireable than the BGP route. The route map setting weight on the network to advertisement to a weight of 0 has it not redirect routes to it for that network. The local preference is intended to have it fail back... but it isn't.

Edit: This involves many different sites, so site A isn't the only one that needs to be impacted by this. It is also important that nothing but the 1.16.0.0/24 is impacted for routing.
DaveQuanceAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Ken BooneNetwork ConsultantCommented:
This is what you posted:
-----------
ip prefix-list siteb permit 1.16.0.0/24

route-map VPNBACKUP 10
 match ip address prefix-list siteb
 set local-preference 90
 set weight 0

ip route 1.16.0.0 255.255.255.0 1.1.1.241 205

router bgp 65001
 network 1.16.0.0 mask 255.255.255.0 route-map VPNBACKUP
----------

So this must be the config at site B.  The problem is your static route at site B needs to be
ip route 1.22.0.0 255.255.255.0 1.1.1.241 205

I am not sure I understand the need for the route map though.  Weight an local preference are only used in determining a best bgp route - not overall route in the routing table.

I don't fully understand your situation but I would do something simple like the following:

-------------



ip route 1.22.0.0 255.255.255.0 1.1.1.241 205

router bgp 65001
 network 1.16.0.0 mask 255.255.255.0
 no auto-summary
 neighbor x.x.x.x distribute-list 15 in    ***-only if need-****

access-list 15 would list the network you want to receive the advertisements for.  Only if you needed this.

So this basic setup lets you run bgp and site b will advertise the 1.16.0.0/24 as long as 1.16.0/24 is in the forwarding table.

Same thing on the other side.. So I will assume then that site B will receive a BGP route for 1.22.0.0/24.

If BGP is broken, the static route will kick in.  The static route has an admin distance of 205 so it will only kick in in the absence of a bgp route to 1.22.0.0/24.  When BGP comes back up, the BGP route will be installed in the forwarding table.

At this point I either don't fully understand what you are trying to accomplish or it just needs to be simplified.
0
DaveQuanceAuthor Commented:
There are a lot of other aspects that to it that would take a bit of text to explain (the network has 2 primary internet connections over 20+ sites, that are connected via an mpls cloud, sites for primary functions must go out one certain connection as opposed to another etc.). Without more info it is probably hard to say much on it.

I came to the conclusion I had two options ideally. For now I'm thinking on option 2 as I've already tested it but if I have time I'll probably test option 1 as it might be the better of the two.

1. Using EIGRP or OSPF advertise the 1.16.0.0/24 (Site B subnet) from the firewall that's running the VPN and directly connected to the Site A layer 3 switch, then redistribute the EIGRP or OSPF into BGP.

2. Setup tracking to the MPLS IP of the site B router to inject the static route (then it would advertise the network in BGP as the only route left during a failure) if that IP is no longer reachable. I thought of this earlier but I wasn't entirely sure how to track that way.

My lab test worked the way I wanted using the SLA (the delays are so it doesn't failover immediately due to congestion):

ip sla 10
 icmp-echo <siteb-mpls-ip>
 timeout 100
 frequency 3
 threshold 5
ip sla schedule 10 life forever start-time now

track 5 rtr 10 reachability

track 11 list boolean and
 object 5 not
 delay down 15 up 15

ip route 1.16..00 255.255.255.0 1.1.1.241 track 11
0
Ken BooneNetwork ConsultantCommented:
Ok it sounds like you have more complexity there then.  

So your setup #2 means that the vpn is now the primary route and not the MPLS cloud.  Typically you wouldn't do this as the MPLS cloud would be "considered" more reliable.  Or are you trying to use VPN over the MPLS cloud?

What you could do is weight the tracked route so that it would have an admin distance greater than that of BGP.

That way it tracks it but only puts it in the table in the absence of the BGP router just like the floating static you were trying to use earlier.
0
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

DaveQuanceAuthor Commented:
The MPLS is the preferred. I've tried just using a static route toward the VPN with a higher AD but it doesn't automatically fail back (it does failover). I have to clear the BGP or remove and re-add the static route for it to fail back. I read someone trying to do this in Expert-Exchange but his scenario was different so the solution he used doesn't seem like the right fit. He did have the exact problem with failback on a floating static route.

I've played around with enough options now that I really think option 2 is the best fit (especially since it's tested and working). I'll probably close out the question with option 2 as the solution in a few hours, after I implement it into the real environment and test.
0
Ken BooneNetwork ConsultantCommented:
Well the issue with it not falling back may have to do with the route map.  I have several customers where I have this scenario running with a floating static and bgp.  I have distribution lists on mine to ensure I am only advertising what I want but I am not using route-maps on the network statement like you are.  I am guessing that is what is holding it up.    When it comes back up does BGP show the route in the BGP table?  - Maybe you need to run no sync on bgp.  You should be able to get the bgp route inserted back into the forwarding table over a higher admin distance floating static.  Could be an IOS bug too.  I know it works and that you shouldn't have to soft-reconfigure bgp for this work.
0
DaveQuanceAuthor Commented:
An IOS bug is probably not likely odd, the same thought crossed my mind at one point t hough. The reason I think it's unlikely is during an outage all I had on was this and when the MPLS was re-introduced (before it was supposed to be and there was no plan for it to be) there were asymmetric routing issues (site A used the VPN, site B used MPLS):

ip route 1.16.0.0 255.255.255.0 1.1.1.241 205

router bgp 65001
 network 1.16.0.0 mask 255.255.255.0

This was performed on a layer 3 switch and gave the same behavior as my lab testing it. The difference in your scenario is probably that you're redistributing static with a distribute list? Maybe the using a network statement combined with a floating static is the wrong way to go about it. I'm going to try to test a redist. static with dist. list and test.

Note: my understanding is the highest any BGP routes can have for an AD is 200, so 205 should be sufficiently high.
0
DaveQuanceAuthor Commented:
As far as I can tell so far, once the MPLS router gets the route from the L3 switch through BGP it no longer pays attention to the 1.16.0.0 route coming from the cloud (the sh ip bgp only shows the L3 switch one as an option). Since it doesn't pull in that route, it never passes it to the L3 switch (hence why the L3 switch never switches back). Now the other routers in my simulated cloud show both paths as options in the show ip bgp.

FW <--> L3 Switch <- shared LAN subnet and AS number-> MPLS Router <-different AS-> MPLS Cloud

I'm going to run some debugs to figure out why the MPLS router isn't taking in the other route.
0
Ken BooneNetwork ConsultantCommented:
Well if you have a static route you don't need to redistribute them.  The network statement alone will cause bgp to advertise the route AS LONG AS the route to that prefix is in the ip forwarding table.  In the case of a static route, the static route would be in the forwarding table, therefore the network statement in BGP would advertise that network to its neighbor as originating from the router.

I think the biggest part of your problem has to do with managed routers where you can't see what is happening and have no idea what kind of things they put in place on BGP there.  That's always a bummer!
0
DaveQuanceAuthor Commented:
Well, the behavior is identical when I use a static + network statement and static + redistribute (was just trying the redistribute and a distribute list in hopes of different behavior). At this point though in my lab I can at least see where the stop-gap is (the MPLS router not holding the other route in). While the track/sla will reach  my goal without dealing with the managed router, I'll probably play with the lab and debugs to understand why afterwards.

Thank you for your input. In my own desire to know, I'll post if I figure it out.
0
Ken BooneNetwork ConsultantCommented:
sounds good.. its always difficult when there is a 3rd party device you can't see!
Good luck!
0
DaveQuanceAuthor Commented:
Attached is the lab re-creation I made (dumbed down from the real environment and very hastily modified to clear out specific info and replace it with placeholder info).

In the real environment the only access I have is A-L3Sw and A fw01. The behavior seems similar to the real environment in the problems though. My "MPLS Cloud" was just a slapped together mesh to try to emulate a cloud.

What is happening is that if the MPLS route is known to A-MPLS it passes it to A-L3SW and the floating static is ignored. After a failure that BGP route disappears and then A-L3SW advertises its path to 1.16.0.0/24. Upon failing back BGP updates occur to all cloud-mpls-0x routers *but* cloud-mpls-01 never sends those updates on to A-MPLS (none of the cloud-mpls-0x routers that connected to cloud-mpls03 send updates, they only receive).

However, all of these routers show in their sh ip bgp that they know about the site A VPN path and Site B MPLS path (more show up too due to the mesh).

That said, A-MPLS never sees the BGP route come back and neither does A-L3SW. That's why the static route never fails back to BGP.

My thought of emulating the cloud might not be proper. All I did was toss 4 routers with basic network statements for their known networks, meshed them, and created a neighbor relationship for all directly connected routers.

Is it normal for those downstream routers to *not* send updates in this situation? While the real environment behavior seems the same from what I *can* see, what I can see is very limited so it may not be the same at all. My "MPLS Cloud" might not be representative of the real cloud at all.
ModifiedDiagram.jpg
0
Ken BooneNetwork ConsultantCommented:
ok so first things first.  Initially you said:

What is happening is that if the MPLS route is known to A-MPLS it passes it to A-L3SW and the floating static is ignored.

So that is happening via BGP.

Now the reality is that the floating static should have been in the table from the beginning.  Boot up the router and there it is while we wait on BGP to establish.  So is that the case, or did we add the floating static after the fact once everything was already up?  

So lets assume BGP is up and working..and then it breaks.   Now the A-MPLS router is receiving the network route from A-L3SW so when BGP comes back up the A-MPLS router already has a better route in its table for the network and thats why it never kicks back over.  The problem is that you (and I am making assumptions here) is that since you are advertising the route to him he thinks it is better.  Since BGP to the cloud is down there is no reason to advertise the BGP route to A-MPLS.  From my thinking the only thing A-MPLS ever needs to learn from A-L3SW is the networks that live in A-L3SW.  So setup a distribution list so you don't send that update to A-MPLS and see what happens.

Am I making sense here?
0
DaveQuanceAuthor Commented:
Note: At this point my effort forward is purely academic. I do not have any tangible requirement, other than my own need to understand.

The floating static was added after the fact. When I reset all the BGP routing Site A uses the MPLS network regardless of what path it took before.

Without A-L3SW advertising 1.16.0.0/24, none of the other sites in the network will know how to get there. The diagram doesn't show them (I just tossed in a note that many other sites exist). This alternate path is required for them as well.

I think you might be right, cloud-mpls-01 might not be telling A-MPLS because it knows (from routes it knows about) that A-MPLS already knows a route. Granted that's why I put the local-preference of 90 on the route-map (so it would still think of it as a worse route). The second I remove the network statement or the static route it takes in the static route immediately after.

Edit: Likely just my not having a strong enough understanding of BGP routing update behavior. Is there a reason it doesn't send that update (to avoid loops maybe?)? Is it possible to override whatever is stopping it from not sending it?
0
DaveQuanceAuthor Commented:
With filtering the advertisement of 1.16.0.0/24 out on A-L3SW, the BGP rought *does* come back in and it *does* fail back properly. So it is the fact that cloud-mpls-01 knows that A-MPLS already has a valid path and it never informs A-MPLS of the MPLS path coming back.

I'm assuming if cloud-mpls-01 believes that it route is better for A-MPLS, it will pass the update on. My understanding of BGP route selection is in this order (on Cisco routers) http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a0080094431.shtml:

1-3 are local to the same AS or specific router, so my weight 0 local_pref of 90 is great on A-L3SW and/or A-MPLS but worthless on cloud-mpls-01.
4 cloud-mpls-01 shows A-MPLS's path to the network as 1 AS away and the BGP advertisement from B-MPLS shows two, why it's not advertising. Prepending the AS a bunch of times on A-L3SW did work though (but I had to move A-L3SW into a separate AS from A-MPLS first).

I had thought about prepending AS numbers but in this situation is that really the best way to go about it? I've read that some routers have problems with AS paths that are too long (granted they're talking 20+). As far as I've read, I'm not sure there are many other options though.
0
Ken BooneNetwork ConsultantCommented:
So when it comes to forcing the hand of another external AS BGP router it can be tough some times.  Basically you have to think of it as I am going to make a recommendation to you.. but you may or may not honor the recommendation.  Thats kinda how it is with BGP.  I would prepend the AS path as long as it is an eBGP connection.  That should do the trick.  I have had to do this many times with non uniform multi link scenarios.  I have never done more than 5 so I don't about the 20+ thing..   Just add enough until it works and you should be good - hopefully!
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
DaveQuanceAuthor Commented:
Thank you for all of your time and help with my processing through this. With your info and my lab testing the situation further I have a much clearer picture of the behavior and why it occurs.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Routers

From novice to tech pro — start learning today.