ISP Redundancy via BGP & wireless 4G?
Hello,
We presently run our own mail & web servers in our data center.
Our data center is connected to the internet with a Verizon "Metro Ethernet" line.
Our router which connects to the Verizon service has an available slot and is capable of connecting to a Verizon 4G LTE wireless card, which is supposed to "kick in" should the primary service drop.
Verizon Wireless tells us that through the miracle of BGP, they could essentially create two paths to our public IP addresses which are presently associate with the Verizon Metro Ethernet. Should the primary service go down, people could reach those public IP addresses via the wireless LTE without us needing to make DNS changes or "swing" our public addresses over to another ISP. We presently have half a class C of public addresses.
Questions:
--------------
1) Is it best practice to use wireless as our "backup" to avoid ALL "last mile" & building infrastructure issues? (e.g. a car hits the telephone pole in front of our building, or our basement floods.)
or
2) Might a heartier connection like a comcast cable modem as our backup line be better? (Though appealing because it's not "metered by the gig", it could be susceptible to a flood in the basement taking out verizon and comcast). Further, in a more widespread power outage, our data center will be on generator (as I'm guessing would verizon cell towers). Would Comcast have all equipment between us & them on battery backup?)
3) Would it be easier and less susceptible to finger-pointing to have verizon wireless backup the primary verizon internet connection? Does Comcast & Verizon play well together? If it failed to fail-over in an emergency, would we just hear a bunch of finger pointing?
4) Is it more complicated to setup this BGP magic with two competing entities vs. two companies that have the word Verizon in their names (although technically, different companies)?
Any suggestions would be very much appreciated.
Thanks,
Mike
We presently run our own mail & web servers in our data center.
Our data center is connected to the internet with a Verizon "Metro Ethernet" line.
Our router which connects to the Verizon service has an available slot and is capable of connecting to a Verizon 4G LTE wireless card, which is supposed to "kick in" should the primary service drop.
Verizon Wireless tells us that through the miracle of BGP, they could essentially create two paths to our public IP addresses which are presently associate with the Verizon Metro Ethernet. Should the primary service go down, people could reach those public IP addresses via the wireless LTE without us needing to make DNS changes or "swing" our public addresses over to another ISP. We presently have half a class C of public addresses.
Questions:
--------------
1) Is it best practice to use wireless as our "backup" to avoid ALL "last mile" & building infrastructure issues? (e.g. a car hits the telephone pole in front of our building, or our basement floods.)
or
2) Might a heartier connection like a comcast cable modem as our backup line be better? (Though appealing because it's not "metered by the gig", it could be susceptible to a flood in the basement taking out verizon and comcast). Further, in a more widespread power outage, our data center will be on generator (as I'm guessing would verizon cell towers). Would Comcast have all equipment between us & them on battery backup?)
3) Would it be easier and less susceptible to finger-pointing to have verizon wireless backup the primary verizon internet connection? Does Comcast & Verizon play well together? If it failed to fail-over in an emergency, would we just hear a bunch of finger pointing?
4) Is it more complicated to setup this BGP magic with two competing entities vs. two companies that have the word Verizon in their names (although technically, different companies)?
Any suggestions would be very much appreciated.
Thanks,
Mike
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Whether you use wireless or wired, it should work. Leave the configuration part apart. It's not that much complicated.
To me, I would use 2 different service provider if I need a backup. If verizon has some problem in their regional gateway your both links would be down.
Would recommend, if you are looking for a backup, use a second vendor apart from Verizon. That will make sense to backup. BGP is not a great deal. Your Service provider will help you in the config or you can get help from EE itself.
Please note that if you use BGP, then primary/backup concept work the best. You should not go for any "load balancing" theory in BGP.
BGP is not always required, unless Verizon has given you some private AS.
Best,
To me, I would use 2 different service provider if I need a backup. If verizon has some problem in their regional gateway your both links would be down.
Would recommend, if you are looking for a backup, use a second vendor apart from Verizon. That will make sense to backup. BGP is not a great deal. Your Service provider will help you in the config or you can get help from EE itself.
Please note that if you use BGP, then primary/backup concept work the best. You should not go for any "load balancing" theory in BGP.
BGP is not always required, unless Verizon has given you some private AS.
Best,
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Wow @pergr & @kevinhsieh!
I had an hour long conversation with another ISP , explicitly mentioning that we only had half a class C, and they didn't mention that as a problem. However, I think she was a glorified sales person, although I was really impressed about how knowledgeable she was.
In any event, it sounds like my next call is back to Verizon wireless to double-verify that they can do this with our half class C.
This really simplifies the this project because we can now stop considering other ISP's!
Thanks!
Mike
I had an hour long conversation with another ISP , explicitly mentioning that we only had half a class C, and they didn't mention that as a problem. However, I think she was a glorified sales person, although I was really impressed about how knowledgeable she was.
In any event, it sounds like my next call is back to Verizon wireless to double-verify that they can do this with our half class C.
This really simplifies the this project because we can now stop considering other ISP's!
Thanks!
Mike
For Verizon, it's a matter of whether or not they can route your /25 over to the wireless network, which I would be a bit surprised if they can do. If they can't, the IPSec/GRE tunnel option will certainly work if they are willing to do that.
It is highly likely that Verizon will do a GRE tunnel over their wireless network, and BGP in that tunnel.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
We spoke to the verizon wireless engineer today.
He continued to say no-problem doing the BGP thing with our half-a-class-c.
We told him we were concerned because everything we read here, and two different ISP's all said anything less than a full class-c would be a problem.
We told him we needed something formal, in writing, and vetted by an additional verizon engineer before we start buying equipment and spending time on it. He said he would be happy to do that.
One unsettling thing: he never heard of the less-than-full-class-c thing between different ISP's.
Per @kevinhsieh 's point, we asked about the VPN tunnel and he said they would never do that for only 1 or 2 routers: too expensive on verizon's part. We would need to have like 20 or 30 routers for that to be approved.
I'll keep everyone posted as this saga unfolds.
Thanks,
Mike
He continued to say no-problem doing the BGP thing with our half-a-class-c.
We told him we were concerned because everything we read here, and two different ISP's all said anything less than a full class-c would be a problem.
We told him we needed something formal, in writing, and vetted by an additional verizon engineer before we start buying equipment and spending time on it. He said he would be happy to do that.
One unsettling thing: he never heard of the less-than-full-class-c thing between different ISP's.
Per @kevinhsieh 's point, we asked about the VPN tunnel and he said they would never do that for only 1 or 2 routers: too expensive on verizon's part. We would need to have like 20 or 30 routers for that to be approved.
I'll keep everyone posted as this saga unfolds.
Thanks,
Mike
You are planning to do BGP only with one provider, Verizon, so the size of your network is not an issue.
They are probably advertising your /25 to the world as part of a /18, or something anyway.
The problem with smaller networks is only between different ISPs, since most filter out routes smaller than /24.
The VPN tunnel he talked about was possibly your own VPN, possibly based on your own APN, which is not useful to you. You can run BGP over wireless without any tunnel - it will just be multihop so you need a static route for that peer... But I am sure they will help with that.
You do not need much equipment. Something like a Juniper SRX100 may be enough. $500.
They are probably advertising your /25 to the world as part of a /18, or something anyway.
The problem with smaller networks is only between different ISPs, since most filter out routes smaller than /24.
The VPN tunnel he talked about was possibly your own VPN, possibly based on your own APN, which is not useful to you. You can run BGP over wireless without any tunnel - it will just be multihop so you need a static route for that peer... But I am sure they will help with that.
You do not need much equipment. Something like a Juniper SRX100 may be enough. $500.
ASKER
Interesting @pergr. I would have guessed that Verizon Wireless would be considered a different ISP from wired Verizon for our Metro Ethernet Service, but maybe not (they both do have Verizon in their name!)
Thanks!
Mike
Thanks!
Mike
ASKER
Just got off the phone with the Verizon engineer (who has been there 30+ years).
Thought he never heard of the issue of needing a full class-c, he did in fact that it would be a potential issue.
He said: not because it wouldn't work, but because of policy. Basically, if they permitted adding these bgp routes for less than a class-c, the router's routing tables would get too big, so from a policy stand-point, they don't permit it.
I countered by saying that requiring us to get a full class C when we don't need more addresses is WORSE, from a public-policy-perspective because the internet is running out of ipv4 addresses.
He said he will ask for an exception for us, and "run-it-up-the-flag-pole" for an exception.
I'll keep u posted,
Thx
Mike
Thought he never heard of the issue of needing a full class-c, he did in fact that it would be a potential issue.
He said: not because it wouldn't work, but because of policy. Basically, if they permitted adding these bgp routes for less than a class-c, the router's routing tables would get too big, so from a policy stand-point, they don't permit it.
I countered by saying that requiring us to get a full class C when we don't need more addresses is WORSE, from a public-policy-perspective because the internet is running out of ipv4 addresses.
He said he will ask for an exception for us, and "run-it-up-the-flag-pole" for an exception.
I'll keep u posted,
Thx
Mike
It is true that ISPs won't usually route to a specific /25 or smaller, but that is only outside of their own AS's.
What I mean is, they won't advertise a /25, /26, etc.
As pergr said, it will be in their /18 or whatever they're advertising to other AS's.
What I mean is, they won't advertise a /25, /26, etc.
As pergr said, it will be in their /18 or whatever they're advertising to other AS's.
ASKER
Would you consider verizon business metro ethernet to be a separate ISP from verizon wireless?
In other words, would our half-class-c be within the /18 (or whatever) they are advertising to other AS's?
Mike
In other words, would our half-class-c be within the /18 (or whatever) they are advertising to other AS's?
Mike
ASKER
BTW, we're still waiting on verizon to see if they will make an exception to their policy of not making route entries for less than a full class-c.
I'll keepu posted,
mike
I'll keepu posted,
mike
I have had PA space and a /26 from an ISP in the UK. This was routed using eBGP and MHSRP.
Not a problem.
The issue is only if you want to route your own PI space via your own AS.
Not a problem.
The issue is only if you want to route your own PI space via your own AS.
ASKER
The wheels of verizon bureaucracy grind slowly.
Still waiting on them to rule yes/no on the exception for us.
Mike
Still waiting on them to rule yes/no on the exception for us.
Mike
ASKER
They pretty much said no to BGP.
They did mention the vpn thing that @kevinhsieh mentioned, but a different person at verizon previously said they wouldn't do it for one or two routers (they would need like 20 to justify that).
However, they said the VPN solution would only work if we used IP addresses only (not by names) - which would be a deal breaker because all our outlook rpc clients reference names, and all of our external customers connect to our mainframe by name.
Still waiting on the final verdict.
Thanks again for everyone's help & input on this,
Mike
They did mention the vpn thing that @kevinhsieh mentioned, but a different person at verizon previously said they wouldn't do it for one or two routers (they would need like 20 to justify that).
However, they said the VPN solution would only work if we used IP addresses only (not by names) - which would be a deal breaker because all our outlook rpc clients reference names, and all of our external customers connect to our mainframe by name.
Still waiting on the final verdict.
Thanks again for everyone's help & input on this,
Mike
ASKER
Still waiting on Verizon. Thanks again to everyone for their input on this issue.
Mike
Mike
ASKER
Ok. Verizon said to pound sand. That we should NOT use the cellular 4G as a fail-over.
They started talking about one of their customers which had an alternate site, and they setup some kind of a hocus pokus alt route thing utilizing that.
In any event, we're going to get a comcast business cable line as an alternate means of connectivity.
We'll put one of our mail servers on comcast permanently, and in the event of a long outage, we'll make DNS changes. Not automatic, but I think it's the best/cheapest we can do.
Thanks to everyone for all the great input.
QUESTION: How am I supposed to split up the points for something like this? (ok to do an even split to all who contributed?)
Thanks,
Mike
They started talking about one of their customers which had an alternate site, and they setup some kind of a hocus pokus alt route thing utilizing that.
In any event, we're going to get a comcast business cable line as an alternate means of connectivity.
We'll put one of our mail servers on comcast permanently, and in the event of a long outage, we'll make DNS changes. Not automatic, but I think it's the best/cheapest we can do.
Thanks to everyone for all the great input.
QUESTION: How am I supposed to split up the points for something like this? (ok to do an even split to all who contributed?)
Thanks,
Mike
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
For email you can put one MX record on each ISP - so one from end server on each ISP - and then have those forward email to your internal Exchange server.
The front end servers can be either a Linux box with postfix, amavis - or just an smtp proxy on the firewall.
The front end servers can be either a Linux box with postfix, amavis - or just an smtp proxy on the firewall.
ASKER
@kevinhsieh: Wow! That's a great idea!
I will discuss with our Lan Admin if there's any issues or consequences of having a nat router on the comcast line directing traffic to our web server in the DMZ.
Presently, our DNS is managed by verizon. Any change is done via an email request because they don't have a GUI way to let the customer manage it. That's ok for planned, non-urgent changes but not great if we want a change in a hurry.
Any concerns or issues with having DNS Made Easy be responsible for DNS? (Are they a single point of failure for our entire enterprise?)
Thanks so much for an awesome suggestion: if we can do it, that would be great!
Mike
I will discuss with our Lan Admin if there's any issues or consequences of having a nat router on the comcast line directing traffic to our web server in the DMZ.
Presently, our DNS is managed by verizon. Any change is done via an email request because they don't have a GUI way to let the customer manage it. That's ok for planned, non-urgent changes but not great if we want a change in a hurry.
Any concerns or issues with having DNS Made Easy be responsible for DNS? (Are they a single point of failure for our entire enterprise?)
Thanks so much for an awesome suggestion: if we can do it, that would be great!
Mike
Any DNS provider is going to be a point of failure. With DNS Made Easy, DNS is basically all they do. I think that in their entire history they had just a few hours where they didn't meet their SLA. They weren't down, just slow Europe or something. They are in multiple datacenters, and traffic gets routed to the closest one.
Add far as connecting Comcast to your firewall, your firewall needs to be smart enough to know that your main ISP is down. My Cisco ASA can do it. Cisco routers can do it too. I use IP SLA and tracked routes.
Add far as connecting Comcast to your firewall, your firewall needs to be smart enough to know that your main ISP is down. My Cisco ASA can do it. Cisco routers can do it too. I use IP SLA and tracked routes.
ASKER
I'm on vacation , on a mobile device, and will close the call next week and split points when on a pc, but I'm very excited about the easy dns,that sounds promising!
Thanks to everyone !
Mike
Thanks to everyone !
Mike
ASKER
Thank you EVERYONE!
I am sincerely appreciative of all this great info, which makes splitting points particularly difficult.
I'm personally most excited by the DNS made Easy motif. If we can pull that off, it sounds great!
Thanks again!
Mike
I am sincerely appreciative of all this great info, which makes splitting points particularly difficult.
I'm personally most excited by the DNS made Easy motif. If we can pull that off, it sounds great!
Thanks again!
Mike
I have my own USB Internet stick and in a good area it delivers good speed.
As a failover backup, I think what Verizon has proposed is just fine. Presumably, if the failover occurs, they would fix the regular connection.
As suggested above, I would try it and see how it works.
.... Thinkpads_User