Solved

Multi ISP with multiple routers.

Posted on 2008-10-02
14
1,996 Views
Last Modified: 2013-11-16
I am being asked to setup 2 cisco 2811 routers to create a redundent connection to the internet and from the internet to hosted servers inside the dmz.  I have the outboud traffic handled with HSRP no problems it works.  

For inbound traffic to hosted services I have 2 ISP circuits with statis public IPs.  Currently I am running HSRP on the WAN interfaces for both these circuits on each of 2 2811s.  So each router is connected to the 2 ISPs (as you can imagine I am having to waste some of my public ips in this config) if the primary router fails the secondary router will take over for both isp circuits.... yes it actually works right now.  and inbound connections heal pretty quickly(RDP). Yes ARP is a problem sometimes because of the timeout on the downstream ISP routers but it works so far.  

The problem is as I stated before this wastes IPs and it is a complex setup and can be a bear to troubleshoot if there is a problem. What I would like to do is hook one isp up to one router and one isp up to the other router.  
Here is where I get stuck...  I know that if a packet comes in on ISP 1 it needs to go back out the ISP 1 and the same for ISP 2 traffic.  But the watchguard I have sitting behind the 2811s allows for only one gateway per interface to be configured as best as I can tell and it currently points to the HSRP virtual ip as the gateway.
I realise to some of you reading this that this is a problem with my current config and you are rght... I fixed by using a route map and configuring multiple IPs on the watchguard, so when a packet comes in on ISP 1 it goes to .5 and .6 for ISP 2 and on the way out the packet's next hop is set by this ip as well to make sure they go back out the correct circuit.

I could use this method to solve this problem with the new setup so that the packet's next hop would be out the ISP 1 circuit or set it to the router that is connected to ISP 2 so that is can send it on...  but I would like to not have to do that because that means the Primary router will be looking at every outbound packet and either forwarding it or passing back over to the other router to forward.

How would this affect performace for ISP 2 traffic?

Is there a better way to do this?
I dont have to have specific config commands but concepts or ideas on how to accomplish this with 2 cisco 2811 dual wan routers and a Watchguard 550e firewall.
Any suggestions from cisco or watchguard folks would be great!

NOTE: Cant use BGP we are a small company and probably could get approval for an ASN or IP block.
0
Comment
Question by:justinl525
  • 7
  • 7
14 Comments
 
LVL 4

Accepted Solution

by:
urgoll earned 500 total points
ID: 22628591
Did you get a block of IP from each ISP ? If so, are being accessed using block1 from ISP1 and block2 from ISP2 ? If so, this is a problem if you connect each router to a single ISP: If router A connected to ISP1 dies, then everyone trying to access block1 will fail. So for true redundancy, you need to to keep your current configuration of each router connected to both ISP.

So keep using HSRP on the inside interface to establish your primary/backup routers, and use Policy-Based Routing on the "inside" interface to determine to which ISP to send each incoming packet, based on the source IP address.

For example, assuming you have 1.2.3.0/24 assigned by ISP1, and 1.2.4.0/24 assigned by ISP2, that 2.2.2.2 is router1's gateway to ISP1 and 3.3.3.3 is router1's gateway to ISP2, you can use the snippet below.

The assumption here is that a single router is powerful enough to handle the full throughput of both of your ISP connections. You didn't specify the throughput, but for regular T1s this should not be a problem.

access-list 1 permit ip 1.2.3.0 0.0.0.255
access-list 2 permit ip 1.2.4.0 0.0.0.255
 
!
interface FastEthernet0/0
 ip policy route-map two-ISP
 
!
route-map two-ISP permit 10
 match ip address 1
 set ip default next-hop 2.2.2.2
route-map two-ISP permit 20
 match ip address 2
 set ip default next-hop 3.3.3.3

Open in new window

0
 
LVL 1

Author Comment

by:justinl525
ID: 22629159
Hey urgoll,
I am with you on that... I am currently using the configuration you described... I was trying to get away from HSRP on the WAN side of the Cisco routers...
Is what you described typical for those of us who cant use BGP?
With HSRP and Static NAT there is an issue where the router responds to ARP for IPs that are configured with static nat with the BIA of the interface.  I fixed this for the LAN side of the routers so the host sees the Virtual MAC Addr in an arp reply, buton the WAN side when I look at the ISP routers arp table i still they see the BIA instead of the HSRP Virtual MAC address of my cisco routers... Can the routers configured so that the NAT "outside" interface will respond to arp requests with the virtual MAC instead of the BIA?
0
 
LVL 4

Expert Comment

by:urgoll
ID: 22630437
Yes, HSRP (or VRRP which is the non-proprietary equivalent) is common when BGP is not feasable. In most cases, NAT is not the solution because it breaks the end-to-end principle.

I'm not sure we're describing the same solution, as I would not be using NAT at all on the routers. I think it would clarify the situation if you could provide more details on your current situation, especially the issue of IP addresses in use and corresponding ISP.
0
PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

 
LVL 1

Author Comment

by:justinl525
ID: 22634460
I am using NAT on the routers so that people accessing the service I have hosted behind the routers can get to them.  When they hit the public IPs on the routers I have to forward them into the DMZ.  Nat is not the solution for the Dual ISPs its is being used as normal to allow people to get in from the outside.

My current situation from ISP to LAN is like this:
From ISP 1 Router to each Cisco Router (interface ETH0 on both Cisco routers). These interfaces are configured with an individual IP and a HSRP IP assigned from ISP1.

From ISP2 Router to each Cisco router (interface ETH1 on each router). These interfaces are configured the same as the ETH0.

Then on the interface ETH2 of each router (using a SVI) I have a IP assigned individually and a HSRP IP.
These ETH2 interfaces both connect to a watchguard firewall.

Here is a rough drawing (dont laugh I failed art class)

We are talking about the same thing.... the code you have looks exactly like what my I have in my routers.....

It be that there is no other way to do this...




net.bmp
0
 
LVL 4

Expert Comment

by:urgoll
ID: 22635074
Could you move the NAT to your watchguard firewall ? This way, only public IPs would be used between watchguard, routers and ISP and HSRP and policy-based routing would take care of all the issues of failover and ARP. As for wasting IPs, the best you can do is to subnet and use /29 on all subnets involved, which would consume two /29 of each ISP-provided netblock.

... Have you discussed being multi-homed with your current ISPs, or are you trying to do this on the sly ? Because with this setup, you are publicaly reachable by two distinct IP address, each address being bound to a single ISP. Probably doesn't matter much if this is for internal purposes, but not so great for services used by external entities (customers and such), as you then need to solve the issue of publishing your IP address, and DNS round-robin just doesn't cut it in case of failure of one of your ISP.
0
 
LVL 1

Author Comment

by:justinl525
ID: 22636319
I dont know how I would take NAT off the routers... I would like to though. I dont think I was clear enough on the picture...  The ISP dont terminate there circuits in my routers they terminate into the CPE equipment and deliver me an ethernet interface that servers as the gateway for the block of ips they give me....
So ISP1 give me a block of say 1.1.1.1-5  thier equipment is already assigned 1.1.1.5 and my cisco routers get 1.1.1.1, 1.1.1.2, and 1.1.1.3 (HSRP)

The ISPs dont want to play nice and I have to make what I have work if you know what I mean.

Currently I am using I have multiple DNS A records that point to the circuits. Some tests show this work ok... however that was from one ISP and DNS can be funny from one ISP/CUSTOMER/MACHINE to another.

Please tell me your thoughts on the DNS stuff...
How would you take nat off the routers or can you?
0
 
LVL 4

Expert Comment

by:urgoll
ID: 22636860
About the DNS issues, DNS is designed to be widely cached, so as the decrease the perceived latency, so the expectation is that DNS data does not change very often. Each DNS data has a TTL (Time To Live) value which tells DNS server how long they are allowed to keep the entry in cache before having to look it up again. In theory, you could set a small TTL for your entry (say 5 minutes) and use DNS round robin to provide both of your IP addresses. The problem is that many large consumer-oriented ISP disregard the TTL and just use a hard-coded TTL value of 1 day (or more) in order to decrease the overall amount of DNS traffic and increase cache efficiency. If one of your ISP goes down, and you take it out of the round-robin pool, those ISPs would still hand out the bad address to their customers, who would then fail to connect to you. Basically, DNS is not a good of providing network failover.

I'll get back to you about the NAT - the netblocks provided by your ISP and much much smaller than I expected, and I you are correct in stating that it is required on the routers.
0
 
LVL 1

Author Comment

by:justinl525
ID: 22637150
Let mw know what you find out about the nat...

The DNS stuff is not good....  Right i am (have to) rely on the client device trying all addresses returned in the DNS query for an A record however I know that this is not good either for multiple reasons....I think xp default to one day
0
 
LVL 4

Expert Comment

by:urgoll
ID: 22655822
Hello justinl525,
I've thought long and hard about your problem, and I can't find any better idea than do NAT on your routers the way you are already doing. The fact the the netblocks provided by your ISPs are so small, and that they are not willing to accomodate are really not playing in your favor.

Sorry for not being a great help,
Christophe
0
 
LVL 1

Author Comment

by:justinl525
ID: 22659831
Thats cool, I figured that might be the case but I wanted to see...
Out of curiosity, what would larger IP blocks allow for in terms of any alternative config you were thinking of...>?
0
 
LVL 4

Expert Comment

by:urgoll
ID: 22660005
With the small netblocks you have, you have to use private IP space for the subnet between your Watchguard and your two 1811 routers. If you had larger netblocks, you could subnet them and use public IP block between the routers and the Watchguard. You could therefore make it so the IP addresses used by end-user terminate on the Watchguard, which would then NAT to the internal server in the DMZ. On the way out, the watchguard could use a route map to determine which packets to route to which router.

The overall scheme would be the same, with all the drawbacks of not having a portable netblock and not using BGP, but the router configuration would be much simpler because they would not need to NAT anything. It would also allow your DMZ server to 'see' the real source IP address instead of our source NATed one.

Regards,
Christophe
0
 
LVL 1

Author Comment

by:justinl525
ID: 22662971
I got it...

I am going to leave this post here a day or 2 more see of there are any watchguard folks then I will I assign points.
0
 
LVL 4

Expert Comment

by:urgoll
ID: 22665164
Fine by me.


Sorry I couldn't find a perfect solution.

Regards,
Christophe
0
 
LVL 1

Author Closing Comment

by:justinl525
ID: 31502554
This is what I was expecting but hoping to find something dofferent... but it works and its really the only option.
0

Featured Post

PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

In the hope of saving someone else's sanity... About a year ago we bought a Cisco 1921 router with two ADSL/VDSL EHWIC cards to load balance local network traffic over the two broadband lines we have, but we couldn't get the routing to work consi…
Getting hacked is no longer a matter or "if you get hacked" — the 2016 cyber threat landscape is now titled "when you get hacked." When it happens — will you be proactive, or reactive?
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

837 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question