We help IT Professionals succeed at work.

Load Balancer for multiple ISP circuits

We are looking into getting a second data circuit as a to the primary circuit. The question is if the primary circuit goes down how would the outside know to route the traffic through the second circuit. Here is our current setup. We have a main location with about 15 remote sites that connect back to the main location via site to site tVPN tunnels. The VPN router at both ends have an associated public IP. If the main circuit went down, I would have to reconfigure everything to use the IP schema associated with the backup circuit. I know they have load balancers that balance traffic between muliple ISPs they seem to load balance out going traffic. Does this mean that I would require a unit at every location? Is there a better way of doing this. I am looking for fault tolerance more than load balancing although both would be nice.
Comment
Watch Question

Commented:
I have read a little on this but have never had to it.

From what I gather, the broad steps are you need to register an IP block from ARIN (or another registry if you are not in their area of control).
https://www.arin.net/
then you need to setup the routing and such with the ISP's that you have.  You can't do this by yourself, you will need the services of the ISP's.
I would contact your ISP's and tell them what you need and they should be able to help with this.
Top Expert 2014

Commented:
You need to use a BGP solution in co-operation with your ISP.
InSearchOfIT Director

Author

Commented:
Ok. Thanks for the info guys. Isn't BGP what ISPs use at the gateway? Any BGP solutions come to mind?

Well you could have hub site and a backup hub site with internet connections that advertises a default route into your cloud. The route advertised by the secondary hub site could have a higher default metric.
This accomplished two things, provides a primary outbound path for all sites and provides a secondary outbound path in case of failure.

Or you could as craigbeck states get a second circuit at your primary site and use BGP with your ISP(S)
to deliver an alternate path as well.

This can definitely be accomplished and you have lots of options, Do you have T-1s? There are devices out there that terminate multiple internet circuits and failover when needed.  

e.g. http://www.barracudanetworks.com/ns/products/link_overview.php?&a=google-na_LinkBalancer-General_MultipleInternetConnections&kw=%2Bmultiple%20%2Binternet%20%2Bconnections&gclid=COPg2fjc6KwCFYZ95QodM1-Olg

harbor235 ;}
Steve JenningsSr Manager Cloud Networking Ops

Commented:
I think this has been said .  .  . Get a public IP address BEHIND the ISP facing address and route to that address. You will do AS-PATH prepending to make the path thru ISP A look better than thru ISP B.  Requires coordination with ISPs BGP, etc

Radware makes a device the works well for load balancing and resilience.

Steve
gsmartinManager of IT

Commented:
BGP multi-homing is definitely an option along with GLBP or HSRP.  This option requires coordinated efforts between ISPs as well as keeps you completely dependent upon ISPs for BGP management/ administration and all other related services including DNS management.
 Now to address your original question:  Can a Load Balancer load balance inbound internet traffic for multiple ISP circuits?  The answer is yes, and with the ability to guarantee a 100% uptime unlike BGP (due to delays in rerouting traffic between ISPs).  
A load balancer/WAN Aggregators will manage link-state connectivity for each ISP internet connection.   As well as, can direct (or divide), load balance, and/or treat connections in a fault-tolerant fashion.  Furthermore, they can act as DNS SOAs – Recommended and required for ISP Failover.  
Note load balancers use primary and secondary DNS SOA servers to effectively load balance inbound internet based traffic.  By acting as DNS SOAs you can better control your DNS TTLs vs having your primary ISP manage DNS.  This enables you to significantly reduce the DNS TTL.  Load balancer appliances can be designed to work independently or together depending on your architectural needs.  
gsmartinManager of IT
Commented:
REFERENCE INFORMATION:
Bandwidth Aggregation: Combining Internet Connections to Incrementally Increase Bandwidth Capacity
http://www.ecessa.com/pages/solutions/solutions_technology_bandwidth.php

Redundant Internet access. It's more than just a good idea.
http://www.ecessa.com/pages/solutions/solutions_technology_redundancy.php

Using Multi Homing ISP Failover for Increased Internet Bandwidth and Reliability
http://www.ecessa.com/pages/solutions/solutions_technology_ispfailover.php
Multi-homing
http://www.ecessa.com/pages/solutions/solutions_technology_multihoming.php
Using Intelligent Internet Traffic Management to Ensure Application Availability
http://www.ecessa.com/pages/solutions/solutions_technology_traffic.php
 
“ISP failover can be achieved in two ways. An antiquated solution is an ISP-level technique based on the Border Gateway Protocol (BGP). This approach requires a high degree of cooperation among multiple ISPs along with the installation and maintenance of expensive and specialized routers at both ends of a link. Another drawback of BGP is the time it requires to reroute Internet traffic, which can result in costly time lost to Internet delays.

The second approach to ISP failover is a far more economical and reliable business-based solution. This approach uses specialized appliances that sit between ordinary routers on a business LAN and the WAN port of the firewall. Each appliance has two or more ports (up to 15 depending on the appliance) to connect to multiple ISPs, and requires no special configuration in the ISPs' routers. When a session is generated from the LAN, the device computes which ISP link has the most available bandwidth and routes the session accordingly. If a link becomes congested, the device automatically reduces traffic going to that link and redirects traffic to links with more available bandwidth. If a link fails, the device automatically stops traffic to the link and redirects it to functioning links.

Inbound WAN failover is achieved by designating the device as the primary and secondary authoritative DNS name server for all the domains being hosted. If an ISP link fails, the device stops advertising that link's IP address to Internet DNS caching servers, which in turn drop that address from their records and redirect traffic to active links. By setting the host name record "Time to Live" to a few seconds, the failed link is quickly removed and reinstated automatically when link connectivity is restored.

The same technique can be used to provide site failover for business continuity and disaster recovery when the appliance is installed at a backup site. In this approach, the appliance at the backup site continually tests DNS resolution to the appliance at the primary site. If the appliance at the primary site does not respond, the appliance at the backup site immediately initiates the inbound ISP failover procedure described above. Inbound user traffic is then immediately redirected to the backup site, and Internet-based business operations continue as normal.“
gsmartinManager of IT

Commented:
Some final details that may help shed some light on this topic, based on my own experiences:

A little over 3 years ago, I was researching this vary topic and looked at the different Load balancers\WAN Link Aggregators vendors.  In this category the two primary vendors F5 and Radware both have very good enterprise level appliances (F5 Link Controller and Radware LinkProof); F5 is typically recommended over Radware, if you have the budget.  These brands tend to be expensive and are typically priced upon total internet bandwidth capacity.  Therefore, their solutions traditionally have a starting cost of about $10K or more per appliance; in the range of about 5mb of throughput, and can quickly go to $20K plus as bandwidth increases.
However, in my research I came across a company called Ecessa (previously known as Astrocom) who’s been in the Telecomm industry for over 40 years.  I had found their PowerLink Wan Aggregator/ Load Balancer appliances had all of the enterprise features I required plus was more reasonably priced than Radware or F5.  Ecessa’s mid-range PowerLink 175 appliance is priced about $3,500 plus the additional 3-yr maintenance supporting up to 15 links with 175 Mb of traffic throughput, and their appliances work extremely well; next model up supports 500Mb and the next model down supports up to 60Mb.   The appliance is easy to install, configure, and manage.  Ecessa’s support team has always been easy to work with and very knowledgeable.  They work directly with you to architect, translate, and plan your environment in order to ensure a successful implementation of their appliances.
Personal note, I have a few of these appliances in production in a failover configuration across my two data centers that I’ve been using over the past three years.  My Initial implementation was load balancing two 3Mb circuits between different ISPs, now I am using 100Mb Internet connections (w/20Mb committed rates) at each of my sites; with Radware and F5 I would’ve had to upgrade and/or completely replace the appliances at a significant cost due to the bandwidth limits.  It’s more common these days, due to lower circuit costs, Ethernet hand-offs/ circuits being more prevalent you need an appliance that’s adaptable without being limited or capped off by your WAN Aggregator/LB appliance.  
Most Valuable Expert 2011
Commented:
This is being made way to complicated.

All you have to do is get two lines from the same ISP,...They both come into the same Routing device.  The ISP sets up BGP to bind the two lines.  All done! Simple!

It is possible to do it with two different ISPs as long as the two ISP will work together on it.  In the US there is an agreement that the two ISPs sign.

This cannot be done with lines that are Home-User technology (DSL, CableTV),...they must be industrial quality lines.
Top Expert 2014
Commented:
pwindell is correct, although to use two different ISPs you usually need to run your own AS and provide routing info via BGP to both ISPs.

The easiest way is to ask one ISP to provide two circuits.  Use HSRP between the two CPE devices and connect them with a L2 link.  The ISP redundancy is usually achieved by taking each line to a separate PoP.  The whole solution should be managed by the ISP, although sometimes they will ask that you provide the L2 link.
Most Valuable Expert 2011

Commented:
pwindell is correct, although to use two different ISPs you usually need to run your own AS and provide routing info via BGP to both ISPs.

In our situation the ISP handled everything,...I never had to touch any of the equipment.  This was this ISP (who is also a person I know) that said there was some kind of agreement that was required if multiple ISPs were involved.  It had an official name but I don't remember what it was called.

The agreement is centered around the fact that your Public IP range you are identified by,...is going to be associated with only one of the ISPs no matter how many are involved,...so the other ISP have to work out a "deal" with the ISP that owns that range in order for them to route to and from those addresses.
Top Expert 2014

Commented:
That's basically it.  If you have one ISP you will not have full inbound redundancy unless you run something like MHSRP on the internal interfaces of the routers (the ones where you connect your kit) whilst the ISP uses BGP to keep its routing updated based on various metrics and interface states.  That's the most common implementation, and probably what your ISP did.

Using multiple ISPs requires your own ASN and publically routeable IP range within that ASN.  Not many entities do this unless they provide IP services and is therefore probably not how your internet links were configured.
InSearchOfIT Director

Author

Commented:
Thanks for the suggestions. I would like to use different ISPs in the event there is a problem with one, I can fail over to the other.
Steve JenningsSr Manager Cloud Networking Ops
Commented:
2 ISPs, quickly get a block of public addresses, path prepend to ISP 2 .  .  . It's not that difficult. Use looking glass to make sure it is set up properly.

Steve
InSearchOfIT Director

Author

Commented:
Thanks for all the usefull tips. I really appreciate it.