Solved

Multi ISP with multiple routers.

Posted on 2008-10-02
14
1,967 Views
Last Modified: 2013-11-16
I am being asked to setup 2 cisco 2811 routers to create a redundent connection to the internet and from the internet to hosted servers inside the dmz.  I have the outboud traffic handled with HSRP no problems it works.  

For inbound traffic to hosted services I have 2 ISP circuits with statis public IPs.  Currently I am running HSRP on the WAN interfaces for both these circuits on each of 2 2811s.  So each router is connected to the 2 ISPs (as you can imagine I am having to waste some of my public ips in this config) if the primary router fails the secondary router will take over for both isp circuits.... yes it actually works right now.  and inbound connections heal pretty quickly(RDP). Yes ARP is a problem sometimes because of the timeout on the downstream ISP routers but it works so far.  

The problem is as I stated before this wastes IPs and it is a complex setup and can be a bear to troubleshoot if there is a problem. What I would like to do is hook one isp up to one router and one isp up to the other router.  
Here is where I get stuck...  I know that if a packet comes in on ISP 1 it needs to go back out the ISP 1 and the same for ISP 2 traffic.  But the watchguard I have sitting behind the 2811s allows for only one gateway per interface to be configured as best as I can tell and it currently points to the HSRP virtual ip as the gateway.
I realise to some of you reading this that this is a problem with my current config and you are rght... I fixed by using a route map and configuring multiple IPs on the watchguard, so when a packet comes in on ISP 1 it goes to .5 and .6 for ISP 2 and on the way out the packet's next hop is set by this ip as well to make sure they go back out the correct circuit.

I could use this method to solve this problem with the new setup so that the packet's next hop would be out the ISP 1 circuit or set it to the router that is connected to ISP 2 so that is can send it on...  but I would like to not have to do that because that means the Primary router will be looking at every outbound packet and either forwarding it or passing back over to the other router to forward.

How would this affect performace for ISP 2 traffic?

Is there a better way to do this?
I dont have to have specific config commands but concepts or ideas on how to accomplish this with 2 cisco 2811 dual wan routers and a Watchguard 550e firewall.
Any suggestions from cisco or watchguard folks would be great!

NOTE: Cant use BGP we are a small company and probably could get approval for an ASN or IP block.
0
Comment
Question by:justinl525
  • 7
  • 7
14 Comments
 
LVL 4

Accepted Solution

by:
urgoll earned 500 total points
Comment Utility
Did you get a block of IP from each ISP ? If so, are being accessed using block1 from ISP1 and block2 from ISP2 ? If so, this is a problem if you connect each router to a single ISP: If router A connected to ISP1 dies, then everyone trying to access block1 will fail. So for true redundancy, you need to to keep your current configuration of each router connected to both ISP.

So keep using HSRP on the inside interface to establish your primary/backup routers, and use Policy-Based Routing on the "inside" interface to determine to which ISP to send each incoming packet, based on the source IP address.

For example, assuming you have 1.2.3.0/24 assigned by ISP1, and 1.2.4.0/24 assigned by ISP2, that 2.2.2.2 is router1's gateway to ISP1 and 3.3.3.3 is router1's gateway to ISP2, you can use the snippet below.

The assumption here is that a single router is powerful enough to handle the full throughput of both of your ISP connections. You didn't specify the throughput, but for regular T1s this should not be a problem.

access-list 1 permit ip 1.2.3.0 0.0.0.255

access-list 2 permit ip 1.2.4.0 0.0.0.255
 

!

interface FastEthernet0/0

 ip policy route-map two-ISP
 

!

route-map two-ISP permit 10

 match ip address 1

 set ip default next-hop 2.2.2.2

route-map two-ISP permit 20

 match ip address 2

 set ip default next-hop 3.3.3.3

Open in new window

0
 
LVL 1

Author Comment

by:justinl525
Comment Utility
Hey urgoll,
I am with you on that... I am currently using the configuration you described... I was trying to get away from HSRP on the WAN side of the Cisco routers...
Is what you described typical for those of us who cant use BGP?
With HSRP and Static NAT there is an issue where the router responds to ARP for IPs that are configured with static nat with the BIA of the interface.  I fixed this for the LAN side of the routers so the host sees the Virtual MAC Addr in an arp reply, buton the WAN side when I look at the ISP routers arp table i still they see the BIA instead of the HSRP Virtual MAC address of my cisco routers... Can the routers configured so that the NAT "outside" interface will respond to arp requests with the virtual MAC instead of the BIA?
0
 
LVL 4

Expert Comment

by:urgoll
Comment Utility
Yes, HSRP (or VRRP which is the non-proprietary equivalent) is common when BGP is not feasable. In most cases, NAT is not the solution because it breaks the end-to-end principle.

I'm not sure we're describing the same solution, as I would not be using NAT at all on the routers. I think it would clarify the situation if you could provide more details on your current situation, especially the issue of IP addresses in use and corresponding ISP.
0
 
LVL 1

Author Comment

by:justinl525
Comment Utility
I am using NAT on the routers so that people accessing the service I have hosted behind the routers can get to them.  When they hit the public IPs on the routers I have to forward them into the DMZ.  Nat is not the solution for the Dual ISPs its is being used as normal to allow people to get in from the outside.

My current situation from ISP to LAN is like this:
From ISP 1 Router to each Cisco Router (interface ETH0 on both Cisco routers). These interfaces are configured with an individual IP and a HSRP IP assigned from ISP1.

From ISP2 Router to each Cisco router (interface ETH1 on each router). These interfaces are configured the same as the ETH0.

Then on the interface ETH2 of each router (using a SVI) I have a IP assigned individually and a HSRP IP.
These ETH2 interfaces both connect to a watchguard firewall.

Here is a rough drawing (dont laugh I failed art class)

We are talking about the same thing.... the code you have looks exactly like what my I have in my routers.....

It be that there is no other way to do this...




net.bmp
0
 
LVL 4

Expert Comment

by:urgoll
Comment Utility
Could you move the NAT to your watchguard firewall ? This way, only public IPs would be used between watchguard, routers and ISP and HSRP and policy-based routing would take care of all the issues of failover and ARP. As for wasting IPs, the best you can do is to subnet and use /29 on all subnets involved, which would consume two /29 of each ISP-provided netblock.

... Have you discussed being multi-homed with your current ISPs, or are you trying to do this on the sly ? Because with this setup, you are publicaly reachable by two distinct IP address, each address being bound to a single ISP. Probably doesn't matter much if this is for internal purposes, but not so great for services used by external entities (customers and such), as you then need to solve the issue of publishing your IP address, and DNS round-robin just doesn't cut it in case of failure of one of your ISP.
0
 
LVL 1

Author Comment

by:justinl525
Comment Utility
I dont know how I would take NAT off the routers... I would like to though. I dont think I was clear enough on the picture...  The ISP dont terminate there circuits in my routers they terminate into the CPE equipment and deliver me an ethernet interface that servers as the gateway for the block of ips they give me....
So ISP1 give me a block of say 1.1.1.1-5  thier equipment is already assigned 1.1.1.5 and my cisco routers get 1.1.1.1, 1.1.1.2, and 1.1.1.3 (HSRP)

The ISPs dont want to play nice and I have to make what I have work if you know what I mean.

Currently I am using I have multiple DNS A records that point to the circuits. Some tests show this work ok... however that was from one ISP and DNS can be funny from one ISP/CUSTOMER/MACHINE to another.

Please tell me your thoughts on the DNS stuff...
How would you take nat off the routers or can you?
0
 
LVL 4

Expert Comment

by:urgoll
Comment Utility
About the DNS issues, DNS is designed to be widely cached, so as the decrease the perceived latency, so the expectation is that DNS data does not change very often. Each DNS data has a TTL (Time To Live) value which tells DNS server how long they are allowed to keep the entry in cache before having to look it up again. In theory, you could set a small TTL for your entry (say 5 minutes) and use DNS round robin to provide both of your IP addresses. The problem is that many large consumer-oriented ISP disregard the TTL and just use a hard-coded TTL value of 1 day (or more) in order to decrease the overall amount of DNS traffic and increase cache efficiency. If one of your ISP goes down, and you take it out of the round-robin pool, those ISPs would still hand out the bad address to their customers, who would then fail to connect to you. Basically, DNS is not a good of providing network failover.

I'll get back to you about the NAT - the netblocks provided by your ISP and much much smaller than I expected, and I you are correct in stating that it is required on the routers.
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 1

Author Comment

by:justinl525
Comment Utility
Let mw know what you find out about the nat...

The DNS stuff is not good....  Right i am (have to) rely on the client device trying all addresses returned in the DNS query for an A record however I know that this is not good either for multiple reasons....I think xp default to one day
0
 
LVL 4

Expert Comment

by:urgoll
Comment Utility
Hello justinl525,
I've thought long and hard about your problem, and I can't find any better idea than do NAT on your routers the way you are already doing. The fact the the netblocks provided by your ISPs are so small, and that they are not willing to accomodate are really not playing in your favor.

Sorry for not being a great help,
Christophe
0
 
LVL 1

Author Comment

by:justinl525
Comment Utility
Thats cool, I figured that might be the case but I wanted to see...
Out of curiosity, what would larger IP blocks allow for in terms of any alternative config you were thinking of...>?
0
 
LVL 4

Expert Comment

by:urgoll
Comment Utility
With the small netblocks you have, you have to use private IP space for the subnet between your Watchguard and your two 1811 routers. If you had larger netblocks, you could subnet them and use public IP block between the routers and the Watchguard. You could therefore make it so the IP addresses used by end-user terminate on the Watchguard, which would then NAT to the internal server in the DMZ. On the way out, the watchguard could use a route map to determine which packets to route to which router.

The overall scheme would be the same, with all the drawbacks of not having a portable netblock and not using BGP, but the router configuration would be much simpler because they would not need to NAT anything. It would also allow your DMZ server to 'see' the real source IP address instead of our source NATed one.

Regards,
Christophe
0
 
LVL 1

Author Comment

by:justinl525
Comment Utility
I got it...

I am going to leave this post here a day or 2 more see of there are any watchguard folks then I will I assign points.
0
 
LVL 4

Expert Comment

by:urgoll
Comment Utility
Fine by me.


Sorry I couldn't find a perfect solution.

Regards,
Christophe
0
 
LVL 1

Author Closing Comment

by:justinl525
Comment Utility
This is what I was expecting but hoping to find something dofferent... but it works and its really the only option.
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

There are times where you would like to have access to information that is only available from a different network. This network could be down the hall, or across country. If each of the network sites have access to the internet, you can create a ne…
Quality of Service (QoS) options are nearly endless when it comes to networks today. This article is merely one example of how it can be handled in a hub-n-spoke design using a 3-tier configuration.
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

5 Experts available now in Live!

Get 1:1 Help Now