Link to home
Start Free TrialLog in
Avatar of justinl525
justinl525

asked on

Multi ISP with multiple routers.

I am being asked to setup 2 cisco 2811 routers to create a redundent connection to the internet and from the internet to hosted servers inside the dmz.  I have the outboud traffic handled with HSRP no problems it works.  

For inbound traffic to hosted services I have 2 ISP circuits with statis public IPs.  Currently I am running HSRP on the WAN interfaces for both these circuits on each of 2 2811s.  So each router is connected to the 2 ISPs (as you can imagine I am having to waste some of my public ips in this config) if the primary router fails the secondary router will take over for both isp circuits.... yes it actually works right now.  and inbound connections heal pretty quickly(RDP). Yes ARP is a problem sometimes because of the timeout on the downstream ISP routers but it works so far.  

The problem is as I stated before this wastes IPs and it is a complex setup and can be a bear to troubleshoot if there is a problem. What I would like to do is hook one isp up to one router and one isp up to the other router.  
Here is where I get stuck...  I know that if a packet comes in on ISP 1 it needs to go back out the ISP 1 and the same for ISP 2 traffic.  But the watchguard I have sitting behind the 2811s allows for only one gateway per interface to be configured as best as I can tell and it currently points to the HSRP virtual ip as the gateway.
I realise to some of you reading this that this is a problem with my current config and you are rght... I fixed by using a route map and configuring multiple IPs on the watchguard, so when a packet comes in on ISP 1 it goes to .5 and .6 for ISP 2 and on the way out the packet's next hop is set by this ip as well to make sure they go back out the correct circuit.

I could use this method to solve this problem with the new setup so that the packet's next hop would be out the ISP 1 circuit or set it to the router that is connected to ISP 2 so that is can send it on...  but I would like to not have to do that because that means the Primary router will be looking at every outbound packet and either forwarding it or passing back over to the other router to forward.

How would this affect performace for ISP 2 traffic?

Is there a better way to do this?
I dont have to have specific config commands but concepts or ideas on how to accomplish this with 2 cisco 2811 dual wan routers and a Watchguard 550e firewall.
Any suggestions from cisco or watchguard folks would be great!

NOTE: Cant use BGP we are a small company and probably could get approval for an ASN or IP block.
ASKER CERTIFIED SOLUTION
Avatar of urgoll
urgoll
Flag of Canada image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of justinl525
justinl525

ASKER

Hey urgoll,
I am with you on that... I am currently using the configuration you described... I was trying to get away from HSRP on the WAN side of the Cisco routers...
Is what you described typical for those of us who cant use BGP?
With HSRP and Static NAT there is an issue where the router responds to ARP for IPs that are configured with static nat with the BIA of the interface.  I fixed this for the LAN side of the routers so the host sees the Virtual MAC Addr in an arp reply, buton the WAN side when I look at the ISP routers arp table i still they see the BIA instead of the HSRP Virtual MAC address of my cisco routers... Can the routers configured so that the NAT "outside" interface will respond to arp requests with the virtual MAC instead of the BIA?
Yes, HSRP (or VRRP which is the non-proprietary equivalent) is common when BGP is not feasable. In most cases, NAT is not the solution because it breaks the end-to-end principle.

I'm not sure we're describing the same solution, as I would not be using NAT at all on the routers. I think it would clarify the situation if you could provide more details on your current situation, especially the issue of IP addresses in use and corresponding ISP.
I am using NAT on the routers so that people accessing the service I have hosted behind the routers can get to them.  When they hit the public IPs on the routers I have to forward them into the DMZ.  Nat is not the solution for the Dual ISPs its is being used as normal to allow people to get in from the outside.

My current situation from ISP to LAN is like this:
From ISP 1 Router to each Cisco Router (interface ETH0 on both Cisco routers). These interfaces are configured with an individual IP and a HSRP IP assigned from ISP1.

From ISP2 Router to each Cisco router (interface ETH1 on each router). These interfaces are configured the same as the ETH0.

Then on the interface ETH2 of each router (using a SVI) I have a IP assigned individually and a HSRP IP.
These ETH2 interfaces both connect to a watchguard firewall.

Here is a rough drawing (dont laugh I failed art class)

We are talking about the same thing.... the code you have looks exactly like what my I have in my routers.....

It be that there is no other way to do this...




net.bmp
Could you move the NAT to your watchguard firewall ? This way, only public IPs would be used between watchguard, routers and ISP and HSRP and policy-based routing would take care of all the issues of failover and ARP. As for wasting IPs, the best you can do is to subnet and use /29 on all subnets involved, which would consume two /29 of each ISP-provided netblock.

... Have you discussed being multi-homed with your current ISPs, or are you trying to do this on the sly ? Because with this setup, you are publicaly reachable by two distinct IP address, each address being bound to a single ISP. Probably doesn't matter much if this is for internal purposes, but not so great for services used by external entities (customers and such), as you then need to solve the issue of publishing your IP address, and DNS round-robin just doesn't cut it in case of failure of one of your ISP.
I dont know how I would take NAT off the routers... I would like to though. I dont think I was clear enough on the picture...  The ISP dont terminate there circuits in my routers they terminate into the CPE equipment and deliver me an ethernet interface that servers as the gateway for the block of ips they give me....
So ISP1 give me a block of say 1.1.1.1-5  thier equipment is already assigned 1.1.1.5 and my cisco routers get 1.1.1.1, 1.1.1.2, and 1.1.1.3 (HSRP)

The ISPs dont want to play nice and I have to make what I have work if you know what I mean.

Currently I am using I have multiple DNS A records that point to the circuits. Some tests show this work ok... however that was from one ISP and DNS can be funny from one ISP/CUSTOMER/MACHINE to another.

Please tell me your thoughts on the DNS stuff...
How would you take nat off the routers or can you?
About the DNS issues, DNS is designed to be widely cached, so as the decrease the perceived latency, so the expectation is that DNS data does not change very often. Each DNS data has a TTL (Time To Live) value which tells DNS server how long they are allowed to keep the entry in cache before having to look it up again. In theory, you could set a small TTL for your entry (say 5 minutes) and use DNS round robin to provide both of your IP addresses. The problem is that many large consumer-oriented ISP disregard the TTL and just use a hard-coded TTL value of 1 day (or more) in order to decrease the overall amount of DNS traffic and increase cache efficiency. If one of your ISP goes down, and you take it out of the round-robin pool, those ISPs would still hand out the bad address to their customers, who would then fail to connect to you. Basically, DNS is not a good of providing network failover.

I'll get back to you about the NAT - the netblocks provided by your ISP and much much smaller than I expected, and I you are correct in stating that it is required on the routers.
Let mw know what you find out about the nat...

The DNS stuff is not good....  Right i am (have to) rely on the client device trying all addresses returned in the DNS query for an A record however I know that this is not good either for multiple reasons....I think xp default to one day
Hello justinl525,
I've thought long and hard about your problem, and I can't find any better idea than do NAT on your routers the way you are already doing. The fact the the netblocks provided by your ISPs are so small, and that they are not willing to accomodate are really not playing in your favor.

Sorry for not being a great help,
Christophe
Thats cool, I figured that might be the case but I wanted to see...
Out of curiosity, what would larger IP blocks allow for in terms of any alternative config you were thinking of...>?
With the small netblocks you have, you have to use private IP space for the subnet between your Watchguard and your two 1811 routers. If you had larger netblocks, you could subnet them and use public IP block between the routers and the Watchguard. You could therefore make it so the IP addresses used by end-user terminate on the Watchguard, which would then NAT to the internal server in the DMZ. On the way out, the watchguard could use a route map to determine which packets to route to which router.

The overall scheme would be the same, with all the drawbacks of not having a portable netblock and not using BGP, but the router configuration would be much simpler because they would not need to NAT anything. It would also allow your DMZ server to 'see' the real source IP address instead of our source NATed one.

Regards,
Christophe
I got it...

I am going to leave this post here a day or 2 more see of there are any watchguard folks then I will I assign points.
Fine by me.


Sorry I couldn't find a perfect solution.

Regards,
Christophe
This is what I was expecting but hoping to find something dofferent... but it works and its really the only option.