Link to home
Start Free TrialLog in
Avatar of cfan73
cfan73

asked on

Dual-ISP homing for Internet fault-tolerance - several questions (BGP, DNS, etc.)

A customer needs fault-tolerance for their Internet connectivity in the least-expensive way possible.  They’re asking for a single exit point, dual-homed to different ISP connections, and automatic fail-over if the primary link goes down.  They also need to make sure that e-mail (incoming and outgoing) fails over as well.

Several questions:

1) Is this even possible with a single exit point, or would a pair (one connected to each ISP) be required?

2) If a single exit point would work, does the exit device need to run BGP at all?  How would the automatic fail-over work?  (Example config - assume this is a Cisco L3 switch (such as a 3560).

3) How will e-mail fail-over?  (Would MX records have to be modified for incoming e-mail?)

4) Does the new (secondary) ISP need to have DNS information recorded/updated for this customer, or is this necessary only if there are local web servers, etc., within the customer site?(What if the customer isn't advertising any publicly-accessible host beyond e-mail connectivity?)

That's a lot of questions - links/references are always appreciated!!


BGP.JPG
Avatar of anishpeter
anishpeter
Flag of India image

Hi.. Cfan,
  I will explain one by one.

1. It is better to have two links from two ISP's. It will make the full Internet reduntant Links
2.If you consider only outgoing traffic configure seperate Static routes with next hop Ip monitoring /IP SLA
3. If the custiomer only has email Servers published outside, cear two A Records First
mail.customer.net     xx.xx.xx.xx   -    <--- Here you give Public IP of the ISP1
mail2.customer.net     YY.YY.YY.YY  -<--- Here you give Public IP of the ISP2
Then create two MX record with equal Priority
mail.customer.net       Priorty  20
mail2.customer.net     Priority 20
This will ensure load balancing and failover also. If Both ISP is working, mails will come in both ISP links, if one links fails all mails will pass through the other ISP link
4. I assume you have a Publci DNS Server in Yahoo, Netsol etc.   If you have  some public webservers in customer site the scenario is changed. There you have to ask for  BGP configuartion from ISP or you have to buy a loadbalance r like BIG IP F5 Link controller.

Hope It's clear.

Thanks,

Peter
Avatar of cfan73
cfan73

ASKER

 
Thank you for your response – of course, I have a slew of follow-ups…  :)

1) You’re suggesting a total of four incoming links (two from EACH ISP) and two edge devices, correct - if this was done, there would be four public IP addresses.  Would you recommend using dual edge devices for each ISP, and HSRP between them so that the external ISP would only point to a single public IP?  How would this design work, in your opinion?  (Diagram?)  

Unfortunately, the customer can’t pay for the an extra two connections at this time, so we’re saddled with the diagram I’ve provided.  

2) We’re considering only outbound traffic, except for the incoming e-mail thing.  (I’ve seen that “IP SLA Responder” is a feature in the Standard images, but that “IP SLA” required the Enhanced images – I have yet to find a clear difference between the two, and would I require the Enhanced image to do what I’m needing in this scenario?)  

3) In your example, you’re doing all of this DNS work on the OUTSIDE DNS servers (wherever they are hosted), correct?  If so, is xx.xx.xx.xx is the public IP of the edge device leading to ISP1, or is xx.xx.xx.xx the “public” address of mail server #1 in the customer DMZ?   (I’m assuming xx.xx.xx.xx is the actual e-mail server, but I’m curious how the ISP would know how to route the traffic to that address.)  

- corollary to 3) above, what if the customer only has a single e-mail server – can failover still be achieved somehow?  

Finally, regarding DNS: to make the changes you’re asking for above, is this something that is typically easy to request of the outside ISP that is providing DNS for your organization?  Can you request the change through one ISP, and your other ISP’s will automatically get updated with these changes, or do you have to contact each of your two ISP’s and request the same change?  

Thank you!!  
Avatar of Jan Bacher
Does the customer have at least a /24 assignment from one of the ISPs?  Or is the customer eligible for a small assignment direct from an RIR?

If so, and the customer border device does not have the memory for [multiple] BGP peers, I would recommend having the customer request that both ISPs announce its assignment.  

That way, you have automatic fail over and no DNS changes are necessary.
I disagree about two ISP's

It is better if the two lines come from the same ISP.     Lines may goes down, entire ISPs do not go down or is at least extremely rare.

Keeping them with a single ISP allows you to keep the same public IP# no matter which line you are running on.  That's impossible with two ISPs.   Yes the MX record may work with priorities, but nothing else will.  A Dynamic DNS solution may help for Web Sites and such, but cost extra money and will not work correctly if you want a resource to be available via both lines at the same time.

You can also combine both lines together (in the correct way) so that you get more bandwidth,...then it a line goes down you get the bandwidth of one line.  Dual-ISP arrangements do not handle this properly or dependably,...no matter what the "marketing" claims they can do,...the technical physical design simply makes it impossble.

The entire redundancy project is performed by the ISP (can't do that if it is two ISPs).  Both lines come into the same Router.   The ISP uses Dynamic Routing Protocols to make the whole thing happen properly and that is performed between the ISP's Router on your end and on their end.

We ran this way for several years with two T1 lines.

Now we have a single DS3,...yep single, no redundancy,...yikes!, why?,...because when you get to that level of service and that level of quality it just simply does not go down,...and if it does you are a high priority to the carrier and they fix it fast.

I've been working where I am now for over 11 years,...I can only remember one outage in 11 years and it only lasted about 40 minutes.  This was way back when we had only single T1 before we had the dual T1s.   The panick and fear of an Internet connection going down is usually unfounded most of the time,....there are a lot bigger threats to worry about.
If your assignment is at least a /24 and it comes from one of your providers, there is a good chance that that provider will allow you to announce it from another's provider's network.  If you do not have a /24 from either provider, *ask* for one.

It's not common for an ISP to have a global problem but they do occur.  Either router updates fail, peering sessions disbanded or someone flubs a configuration.  The last impacting [major] outage was just over a month ago.

I always recommend, where possible, two providers and one set of address space to announce to the world.

pwindell is not correct unless he is referring to smaller assignments.  Having gotten LOAs (Letter of Authorization) from one provider to route its address space out another provider, I know that this is possible.
LoAs are only possible when the Provider agrees to it.  There is no law making them do it.  In my part of the world it is not likely to happen.  Also the bigger the provider the more lousey the service and the more impossible the cooperation between any two of them.  The smaller reigonal ISPs around here give excellent service and show a lot more accountability to their customers.
It is so much easier to just use one provider for the dual lines then trying to jump through the hoops you are describing and to get the ISP cooperation you describe that there is just no comparison.
I think you are selling short the abilities of an ISP to hold their own.  They are better equipped and have a higher skill level than most business internal shops.
Three ISPs in this area are run by  three different friends of mine, and I work part time for one of them (I work full time for an NBC News affiliate, and outages make "news").  They have redundancy built into their facilities out the wazzoo,..which includes multiple backbones going out of their facilities in different geographical directions and using different Carriers.  So if a whole Carrier goes down,...they use one of the other Carriers,...a major line gets cut somewhere, they just switch to another.  The redundancy that an ISP provides for themselves dwarfs anything the a customer can do for themselves.  There has not been a major outage in this area for over a decade,...to cause an outage at the ISP level would require multiple Carriers with multiple Main Lines to all go down together to exceed the ISP's redundancy capability.
Could someone at the ISP misconfigure a router?  Yes.   Would it take down the facility? Not likely because of all the internal redundancy.   Also, the people configuring them know what they are doing,...or they wouldn't be there,...and  they would know immediately that something went wrong and would immediately correct it.  Restoring a router config from a backup can be done in seconds.
I do nothing but support ISPs including their peering sessions and monitor global outages.  I'm not talking about my friends' internet connections.  These are major global players in the Internet [peering] market that have periodic problems -- some regional, some not.

It happens.  Too often to put all of your eggs in one basket.
Well, I think we have given him enough of both views that he can decide for himself.
BTW - we have 4 different connection in and out of here,...with there being 3 ISP's among them.  However there are not redundant,...they each serve a different purpose.  But the point is that I am not oblivious to both situations.
Outages are very important to us,...we have a 24pair Fiber trenched directly into our building theat comes into a traditional 19-in rack owned by AT&T because AT&T offers "Cable TV" over their phone lines and so we feed our broadcast directly into the fiber,...but I am confident that we don't need a second ISP to push that out on.  AT&T didn't feel we need to have any other redundancy other than what is built into their equipment in that rack,...and having that feed running 24/7 is very important to them.
Anyway, I think enough has been said about it.  I'm never going to think that a dual-ISP is going to be worth the hassle, and you are,...so let's just leave it at that.
The redundancy with separate carriers is not in anticipation of an "entire ISP" going down...whatever that may mean.  If a truck drives into a phone box...all copper from that box is affected.  T1, ADSL, POTS...having two circuits is not "redundant" when they both get plowed.  Same goes for a backhoe ripping up a trenched conduit.  Or a fire/water/electrical problem in a rack space that takes out all ADSL equipment in that space.

I have fiber and ADSL at the house.  I have lost the fiber due to "maintenance" for 20-30 stints more than once.  ADSL stayed up.  I've also lost ADSL due to water on the lines, heavy winds & trees knocking them loose, and a bad lightning ground.  At the same time...cable was up and running.

Having 2xADSL from the same provider will only provide redundancy when the customer needs a rip-and-rebuild on one mangled connection...and the other connection hasn't been affected.  Other than that...what are the odds that a tree will neatly slice through a 3-pair aerial wire, and only take out one pair?  Slim odds on that.
We can chase "maybes",  "what ifs", and "could happens" around for days.  The building could burn down, our old tower might fall on the building, the roof might start leaking over the server room, an employee might go beserk on election night and start shooting everybody.
Switching to another provider mens the public IP will change,...short of an LoA which may or may not be possible,...and may not be a proper solution even if you had an LoA.   The public IP changing is going to cause some things,...mainly inbound things to fail.   There are a huge amount of things at our place that you just cannot "flip" to another line of another ISP and the IP# associated with it without reconfiguration.   But if both come from the same ISP and the Public IP does not change when the line switches everything keep working and it does so transparently
I don't see the relevance of workplace violence.  Or the rest of the rant.

All the situations I described have affected my service.

The pickup truck knocking out a whole block even surprised the telco.  The steel bollards didn't do much good when the box is at street level with 50-60mph traffic.  Hi-cap customers do get better response, as we had a truck roll on-site in under 20 minutes.

There are other situations, including billing errors, unexplained ISP port blocks, re-routing in or around a metro area.

All things that can degrades service levels or cut them off entirely.  All things that actually happened to me...no magical unicorns.

I didn't think the OP was looking for anything imaginary or out-of-the box.

Facility arrangement is a valid argument, but few people have have control over the MPOE.  My drops come in wherever the carrier has already made an opening.  On a residence or privately owned structure, sometimes they will let you relocate it yourself, then move the service line.
I'm not ranting,..I'm perfectly calm.  I just speak in hyperbally a lot,...even excessively sometimes.
Workplace violence,...well we've had some reporters in the past that I'm not so sure about.  Luckily I'm upstairs where I can hear them comming up the steps before they get to me.
including billing errors, unexplained ISP port blocks
Having a  quality ISP helps there.   We had a billing issue once,...they called me on the phone and asked me what happened rather than just pulling the plug.    Port blocks,..not gonna happen,...in our agreement.   Having a good ISP is important, one that you can have a real relationship with,...I have no confidence in large "national" ISPs that have too much of the focus on "home user" situations, with businesses being an after thought,...and then take 30 minutes to finally get on the phone with their "outsourced" tech support.
Avatar of cfan73

ASKER

Hey, interesting/informational exchange, folks (seriously), but none of it answered the questions regarding the scenario I'm currently facing - two ISP's, each providing a single link to a single edge device (such as a single or HA pair of ASA').  So, to repeat post 34001674:

Thank you for your response – of course, I have a slew of follow-ups…  :)

1) You’re suggesting a total of four incoming links (two from EACH ISP) and two edge devices, correct - if this was done, there would be four public IP addresses.  Would you recommend using dual edge devices for each ISP, and HSRP between them so that the external ISP would only point to a single public IP?  How would this design work, in your opinion?  (Diagram?)  

Unfortunately, the customer can’t pay for the an extra two connections at this time, so we’re saddled with the diagram I’ve provided.  

2) We’re considering only outbound traffic, except for the incoming e-mail thing.  (I’ve seen that “IP SLA Responder” is a feature in the Standard images, but that “IP SLA” required the Enhanced images – I have yet to find a clear difference between the two, and would I require the Enhanced image to do what I’m needing in this scenario?)  

3) In your example, you’re doing all of this DNS work on the OUTSIDE DNS servers (wherever they are hosted), correct?  If so, is xx.xx.xx.xx is the public IP of the edge device leading to ISP1, or is xx.xx.xx.xx the “public” address of mail server #1 in the customer DMZ?   (I’m assuming xx.xx.xx.xx is the actual e-mail server, but I’m curious how the ISP would know how to route the traffic to that address.)  

- corollary to 3) above, what if the customer only has a single e-mail server – can failover still be achieved somehow?  

Finally, regarding DNS: to make the changes you’re asking for above, is this something that is typically easy to request of the outside ISP that is providing DNS for your organization?  Can you request the change through one ISP, and your other ISP’s will automatically get updated with these changes, or do you have to contact each of your two ISP’s and request the same change?  

Thanks again, experts!   :)

Hi.. Cfan,
  wow. How many argument. It's quite good for those using EE.
Now my turn,
I assume the customer only having incomming traffic of mail servers. not any internal web/application servers which is accessible from Internet
1. Not the four links,  only one link from each ISP. Total 2 Links.Two different ISP, better competers. Route their fiber through different paths and  terminate in one router or ( two in HSRP - if affordable). You will get two set of Publc IP.
2. The need of IP SLA is when the ISP link is up, but the ISP is not reachable. If you consider ISP link down/up only, then you can can have two simple statc routers with one above administrative distance.
But both secenarions will work perfect with IP SLA responder for this,

A)First set SLA rules

R1(config)# ip sla 1
R1(config)# icmp-echo 2.2.2.2 source-interface FastEthernet0/0  <---- my prferred ISP next hop is 2.2.2.2
R1(config)# timeout 1000
R1(config)# threshold 2
R1(config)# frequency 3
R1(config)# ip sla schedule 1 life forever start-time now

B)Define the Track object
R1(config)# track 1 ip sla 1 reachability

If you want to see the tracking
R1# show track

Track 1
IP SLA 1 reachability
Reachability is Down
1 change, last change 00:03:19
Latest operation return code: Unknown

C)Configure Static routes

R1(config)# ip route 0.0.0.0 0.0.0.0 2.2.2.2 track 1
R1(config)# ip route 0.0.0.0 0.0.0.0 3.3.3.3 10


3. XX.XX.XX.XX is the public IP address of the mail server from ISP1 segment and YY.YY.YY.YY is the public ip of the same mail server from ISP 2 segment. If you use the entries in your external DNS Server like yahoo, it will automatically propagated to everywhere in the world including your ISP. If you can afford two mail relays ( Appliance, not mail servers) use fist public IP(XX.XX.XX.XX) for fisrt relay and second address(YY.YY.YY.YY)for second relay.  You can have two public IPs from one ISP for one relay also. But the former will work fine

4. You dont need to contact ISP for DNS changes, unless you host your external DNS with ISP. I recommend put it with Netsol or Yahoo. You have to contact your ISP to create/change PTR record for mail relays. Most of the mail relays in the world starts reverse lookup to reduce spams

Hope it'clear. Please reply if you want more help.

Thanks,
Anish




Avatar of cfan73

ASKER

Hey gang - I haven't abandoned this thread yet... just got busy.  Will absorb and reply/continue in the next day or two.
Avatar of cfan73

ASKER

OK, anishpeter - more follow-up:

Yes, let's assume that the only internal server that needs to be reachable is the mail server. Everything else is hosted on the outside.  Having said that:

SLA) Something is going to have to perform NAT for the internal hosts browsing out.  Since Cisco's edge switches (such as a 3560) don't support NAT, this would have to  be done on the ASA firewall pair that will sit behind it.  The problem I'm seeing is that if the 3560 is doing SLA, how would the ASA pair know how to NAT the traffic, since it wouldn't know which outbound path the switch would take?   Seems as if the SLA would have to be on the ASA pair instead, which is fine...   Your provided SLA configuration makes sense and is helpful.

3. XX.XX.XX.XX is the public IP address of the mail server from ISP1 segment and YY.YY.YY.YY is the public ip of the same mail server from ISP 2 segment. If you use the entries in your external DNS Server like yahoo, it will automatically propagated to everywhere in the world including your ISP. If you can afford two mail relays ( Appliance, not mail servers) use fist public IP(XX.XX.XX.XX) for fisrt relay and second address(YY.YY.YY.YY)for second relay.  You can have two public IPs from one ISP for one relay also. But the former will work fine

There is only one internal web server, so you're saying here that I would have NAT configured for internal traffic over ISP1 and ISP2 translating between x.x.x.x -> internal e-mail IP on one side, and y.y.y.y -> same internal e-mail IP on the other, right?  These external MX records are pointing to public IP addresses, obviously, but are they always NAT'd to the actual private e-mail server internally (such as Exchange), or some intermediate device between?   (You mention having "two mail relays", which is why I ask.)

4. You dont need to contact ISP for DNS changes, unless you host your external DNS with ISP. I recommend put it with Netsol or Yahoo. You have to contact your ISP to create/change PTR record for mail relays. Most of the mail relays in the world starts reverse lookup to reduce spams

k, so 3) above (updating MX records w/ different public e-mail server addresses) would require contacting both of the ISP's, or would one update the other?    Lastly, on the "mail relay" concept again - this is a device that is different than the actual internal e-mail server?  If so, is this a box that typically resides in a DMZ with a public IP address, or is it internal (and NAT'd somewhere near the edge)?

Thanks again, Anish - I think we're almost there...

Avatar of cfan73

ASKER

ok, if I can just get one follow-up question answered, Anish - again, this pertains to redundancy and failover for incoming SMTP mail.  This is from your first reply:
---------------------------------------------------------------------------------------------
3. If the custiomer only has email Servers published outside, cear two A Records First
mail.customer.net     xx.xx.xx.xx   -    <--- Here you give Public IP of the ISP1
mail2.customer.net     YY.YY.YY.YY  -<--- Here you give Public IP of the ISP2
Then create two MX record with equal Priority
mail.customer.net       Priorty  20
mail2.customer.net     Priority 20
This will ensure load balancing and failover also. If Both ISP is working, mails will come in both ISP links, if one links fails all mails will pass through the other ISP link
---------------------------------------------------------------------------------------------

There is currently only one MX record, pointing to a single internal mail server.  For the above to work, e-mail coming in to xx.xx.xx.xx (from ISP1) and traffic coming in to yy.yy.yy.yy (from ISP2) would both have to be NAT'd to the same single internal IP address, correct?

The dual ISP connections are going to be handled by a high-availability ASA pair.  How can we have two static NAT mappings for the same internal IP?

Thanks again!
ASKER CERTIFIED SOLUTION
Avatar of anishpeter
anishpeter
Flag of India image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial