Solved

ISP Redundancy via BGP & wireless 4G?

Posted on 2013-05-30
26
1,660 Views
Last Modified: 2013-08-13
Hello,

We presently run our own mail & web servers in our data center.

Our data center is connected to the internet with a Verizon "Metro Ethernet" line.

Our router which connects to the Verizon service has an available slot and is capable of connecting to a Verizon 4G LTE wireless card, which is supposed to "kick in" should the primary service drop.

Verizon Wireless tells us that through the miracle of BGP, they could essentially create two paths to our public IP addresses which are presently associate with the Verizon Metro Ethernet.  Should the primary service go down, people could reach those public IP addresses via the wireless LTE without us needing to make DNS changes or "swing" our public addresses over to another ISP.  We presently have half a class C of public addresses.

Questions:
--------------

1) Is it best practice to use wireless as our "backup" to avoid ALL "last mile" & building infrastructure issues? (e.g. a car hits the telephone pole in front of our building, or our basement floods.)

or

2) Might a heartier connection like a comcast cable modem as our backup line be better?  (Though appealing because it's not "metered by the gig", it could be susceptible to a flood in the basement taking out verizon and comcast).  Further, in a more widespread power outage, our data center will be on generator (as I'm guessing would verizon cell towers).  Would Comcast have all equipment between us & them on battery backup?)

3) Would it be easier and less susceptible to finger-pointing to have verizon wireless backup the primary verizon internet connection?  Does Comcast & Verizon play well together?  If it failed to fail-over in an emergency, would we just hear a bunch of finger pointing?  

4) Is it more complicated to setup this BGP magic with two competing entities vs. two companies that have the word Verizon in their names (although technically, different companies)?

Any suggestions would be very much appreciated.

Thanks,
Mike
0
Comment
Question by:mike2401
  • 13
  • 4
  • 4
  • +3
26 Comments
 
LVL 17

Assisted Solution

by:pergr
pergr earned 125 total points
ID: 39210057
Since your IP address range is only /25, and they belong to Verizon, you can not use BGP to two different providers. You would need your own /24 and preferably your own AS-number too for that.

1, 2) It is probably more common to use two different fixed lines than wireless - but then real high-speed wireless has not been around for that long so why not try it.

3, 4) Not really an option due to my first comment.

The wireless option has a low cost to set up, and is fairly uncomplicated. I suggest you set it up and do some testing now and then (monthly) to to check that fail-over always works.
0
 
LVL 90

Expert Comment

by:John Hurst
ID: 39210379
I have a couple of clients with ISP wireless as their only connection (because of location) and it works fine.

I have my own USB Internet stick and in a good area it delivers good speed.

As a failover backup, I think what Verizon has proposed is just fine. Presumably, if the failover occurs, they would fix the regular connection.

As suggested above, I would try it and see how it works.

.... Thinkpads_User
0
 
LVL 17

Expert Comment

by:surbabu140977
ID: 39210504
Whether you use wireless or wired, it should work. Leave the configuration part apart. It's not that much complicated.

To me, I would use 2 different service provider if I need a backup. If verizon has some problem in their regional gateway your both links would be down.

Would recommend, if you are looking for a backup, use a second vendor apart from Verizon. That will make sense to backup. BGP is not a great deal. Your Service provider will help you in the config or you can get help from EE itself.

Please note that if you use BGP, then primary/backup concept work the best. You should not go for any "load balancing" theory in BGP.

BGP is not always required, unless Verizon has given you some private AS.

Best,
0
 
LVL 42

Assisted Solution

by:kevinhsieh
kevinhsieh earned 250 total points
ID: 39210831
Comcast won't do BGP unless you are on their fiber product. That doesn't matter anyway because you need a /24 or larger to do BGP to the internet with multiple providers. If Verizon can handle BGP between their wired and wireless divisions, that's great.

One thing that may work of Verizon is up for it is to have a redundant connection through a VPN connection. I do this right now. My primary internet connection hours through regular leased line. My border router also has a fixed IP through Comcast. I have an IPSec tunnel from my router to my ISP, and then we run GRE over that connection. My BGP peering then runs over the regular connection and the GRE tunnel. I now have redundancy for my connection, and i also have a second router with am IPSec + GRE tunnel in case of a failure in my primary router.
0
 

Author Comment

by:mike2401
ID: 39210962
Wow @pergr & @kevinhsieh!  

I had an hour long conversation with another ISP , explicitly mentioning that we only had half a class C, and they didn't mention that as a problem.  However, I think she was a glorified sales person, although I was really impressed about how knowledgeable she was.

In any event, it sounds like my next call is back to Verizon wireless to double-verify that they can do this with our half class C.

This really simplifies the this project because we can now stop considering other ISP's!

Thanks!

Mike
0
 
LVL 42

Expert Comment

by:kevinhsieh
ID: 39211244
For Verizon, it's a matter of whether or not they can route your /25 over to the wireless network, which I would be a bit surprised if they can do. If they can't, the IPSec/GRE tunnel option will certainly work if they are willing to do that.
0
 
LVL 17

Expert Comment

by:pergr
ID: 39211795
It is highly likely that Verizon will do a GRE tunnel over their wireless network, and BGP in that tunnel.
0
 
LVL 45

Assisted Solution

by:Craig Beck
Craig Beck earned 125 total points
ID: 39212742
Unless you've got your own ASN and PI address space, forget BGP unless your ISP is dual-homing your internet feed.  Even then, you won't do any BGP config.  It will all be done by the ISP on their routers.

If you want to use two ISPs they won't share BGP info between themselves.  You have to have your own ASN and PI space and your routers will share your BGP info with them.

The rep at your ISP probably didn't mention any issues with BGP and a /25 as it is possible to route a /25.  The problem is that RIRs aren't allowed to issue anything smaller than a /24 and to obtain a /24 you have to justify exactly what you need it for as the supply of IPv4 addresses is nearly gone.

You won't be able to do BGP over the wireless link.  That should just be used as an outbound backup at most.
0
 

Author Comment

by:mike2401
ID: 39220275
We spoke to the verizon wireless engineer today.
He continued to say no-problem doing the BGP thing with our half-a-class-c.
We told him we were concerned because everything we read here, and two different ISP's all said anything less than a full class-c would be a problem.

We told him we needed something formal, in writing, and vetted by an additional verizon engineer before we start buying equipment and spending time on it. He said he would be happy to do that.

One unsettling thing: he never heard of the less-than-full-class-c thing between different ISP's.

Per @kevinhsieh 's point, we asked about the VPN tunnel and he said they would never do that for only 1 or 2 routers: too expensive on verizon's part.  We would need to have like 20 or 30 routers for that to be approved.

I'll keep everyone posted as this saga unfolds.

Thanks,
Mike
0
 
LVL 17

Expert Comment

by:pergr
ID: 39220313
You are planning to do BGP only with one provider, Verizon, so the size of your network is not an issue.

They are probably advertising your /25 to the world as part of a /18, or something anyway.

The problem with smaller networks is only between different ISPs, since most filter out routes smaller than /24.

The VPN tunnel he talked about was possibly your own VPN, possibly based on your own APN, which is not useful to you. You can run BGP over wireless without any tunnel - it will just be multihop so you need a static route for that peer...  But I am sure they will help with that.

You do not need much equipment. Something like a Juniper SRX100 may be enough. $500.
0
 

Author Comment

by:mike2401
ID: 39222225
Interesting @pergr.  I would have guessed that Verizon Wireless would be considered a different ISP from wired Verizon for our Metro Ethernet Service, but maybe not (they both do have Verizon in their name!)

Thanks!

Mike
0
 

Author Comment

by:mike2401
ID: 39223724
Just got off the phone with the Verizon engineer (who has been there 30+ years).

Thought he never heard of the issue of needing a full class-c, he did in fact that it would be a potential issue.

He said: not because it wouldn't work, but because of policy.  Basically, if they permitted adding these bgp routes for less than a class-c, the router's routing tables would get too big, so from a policy stand-point, they don't permit it.

I countered by saying that requiring us to get a full class C when we don't need more addresses is WORSE, from a public-policy-perspective because the internet is running out of ipv4 addresses.

He said he will ask for an exception for us, and "run-it-up-the-flag-pole" for an exception.

I'll keep u posted,
Thx
Mike
0
 
LVL 45

Expert Comment

by:Craig Beck
ID: 39223934
It is true that ISPs won't usually route to a specific /25 or smaller, but that is only outside of their own AS's.

What I mean is, they won't advertise a /25, /26, etc.

As pergr said, it will be in their /18 or whatever they're advertising to other AS's.
0
Give your grad a cloud of their own!

With up to 8TB of storage, give your favorite graduate their own personal cloud to centralize all their photos, videos and music in one safe place. They can save, sync and share all their stuff, and automatic photo backup helps free up space on their smartphone and tablet.

 

Author Comment

by:mike2401
ID: 39234637
Would you consider verizon business metro ethernet to be a separate ISP from verizon wireless?

In other words, would our half-class-c be within the /18 (or whatever) they are advertising to other AS's?

Mike
0
 

Author Comment

by:mike2401
ID: 39245437
BTW, we're still waiting on verizon to see if they will make an exception to their policy of not making route entries for less than a full class-c.

I'll keepu posted,
mike
0
 
LVL 45

Expert Comment

by:Craig Beck
ID: 39245566
I have had PA space and a /26 from an ISP in the UK.  This was routed using eBGP and MHSRP.

Not a problem.

The issue is only if you want to route your own PI space via your own AS.
0
 

Author Comment

by:mike2401
ID: 39260423
The wheels of verizon bureaucracy grind slowly.

Still waiting on them to rule yes/no on the exception for us.

Mike
0
 

Author Comment

by:mike2401
ID: 39279192
They pretty much said no to BGP.
They did mention the vpn thing that @kevinhsieh mentioned, but a different person at verizon previously said they wouldn't do it for one or two routers (they would need like 20 to justify that).  

However, they said the VPN solution would only work if we used IP addresses only (not by names) - which would be a deal breaker because all our outlook rpc clients reference names, and all of our external customers connect to our mainframe by name.

Still waiting on the final verdict.

Thanks again for everyone's help & input on this,

Mike
0
 

Author Comment

by:mike2401
ID: 39315413
Still waiting on Verizon.  Thanks again to everyone for their input on this issue.

Mike
0
 

Author Comment

by:mike2401
ID: 39368275
Ok. Verizon said to pound sand. That we should NOT use the cellular 4G as a fail-over.

They started talking about one of their customers which had an alternate site, and they setup some kind of a hocus pokus alt route thing utilizing that.

In any event, we're going to get a comcast business cable line as an alternate means of connectivity.

We'll put one of our mail servers on comcast permanently, and in the event of a long outage, we'll make DNS changes.  Not automatic, but I think it's the best/cheapest we can do.

Thanks to everyone for all the great input.

QUESTION: How am I supposed to split up the points for something like this? (ok to do an even split to all who contributed?)

Thanks,
Mike
0
 
LVL 42

Accepted Solution

by:
kevinhsieh earned 250 total points
ID: 39368474
I recommend that you consider using using DNS Made Easy for your public DNS. They have a service where they can monitor your servers and change the public DNS records if they are not available on your Verizon connection. What I do is that I have static NAT on my Comcast router for the important servers that may need to fail over to Comcast. If my regular ISP connection is down, all traffic then goes out through Comcast, which means that the servers get new public IP addresses. DNS Made Easy will discover that the servers are not available on my regular ISP, and then switch the records over to the Comcast addresses. Once the main ISP connection is back, DNS Made Easy automatically changes the DNS records back.

 http://www.dnsmadeeasy.com/services/dns-failover-system-monitoring/

Feel free to split the points among all of the posts you find useful.
0
 
LVL 17

Expert Comment

by:pergr
ID: 39369131
For email you can put one MX record on each ISP - so one from end server on each ISP - and then have those forward email to your internal Exchange server.

The front end servers can be either a Linux box with postfix, amavis - or just an smtp proxy on the firewall.
0
 

Author Comment

by:mike2401
ID: 39370160
@kevinhsieh: Wow! That's a great idea!  

I will discuss with our Lan Admin if there's any issues or consequences of having a nat router on the comcast line directing traffic to our web server in the DMZ.

Presently, our DNS is managed by verizon.  Any change is done via an email request because they don't have a GUI way to let the customer manage it.  That's ok for planned, non-urgent changes but not great if we want a change in a hurry.

Any concerns or issues with having DNS Made Easy be responsible for DNS? (Are they a single point of failure for our entire enterprise?)

Thanks so much for an awesome suggestion: if we can do it, that would be great!

Mike
0
 
LVL 42

Expert Comment

by:kevinhsieh
ID: 39370350
Any DNS provider is going to be a point of failure. With DNS Made Easy, DNS is basically all they do. I think that in their entire history they had just a few hours where they didn't meet their SLA. They weren't down, just slow Europe or something. They are in multiple datacenters, and traffic gets routed to the closest one.

Add far as connecting Comcast to your firewall, your firewall needs to be smart enough to know that your main ISP is down. My Cisco ASA can do it. Cisco routers can do it too. I use IP SLA and tracked routes.
0
 

Author Comment

by:mike2401
ID: 39380664
I'm on vacation , on a mobile device, and will close the call next week and split points when on a pc, but I'm very excited about the easy dns,that sounds promising!

Thanks to everyone !

Mike
0
 

Author Closing Comment

by:mike2401
ID: 39405237
Thank you EVERYONE!

I am sincerely appreciative of all this great info, which makes splitting points particularly difficult.

I'm personally most excited by the DNS made Easy motif. If we can pull that off, it sounds great!

Thanks again!

Mike
0

Featured Post

New My Cloud Pro Series - organize everything!

With space to keep virtually everything, the My Cloud Pro Series offers your team the network storage to edit, save and share production files from anywhere with an internet connection. Compatible with both Mac and PC, you're able to protect your content regardless of OS.

Join & Write a Comment

Suggested Solutions

#Citrix #Citrix Netscaler #HTTP Compression #Load Balance
Getting hacked is no longer a matter or "if you get hacked" — the 2016 cyber threat landscape is now titled "when you get hacked." When it happens — will you be proactive, or reactive?
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

759 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now