asked on

Cisco Express Forwarding load balance scenario

I have configured a 2600 router for load balancing using CEF. The router has 2 Internet connections ( e1/0 and s0/0) from different providers with 2 equal cost default routes to the respective providers next hop address. I also have a SAA probe configured to track the state of one of the default routes for failover. NAT is configured on both external interfaces. There is one inside interface (Fa0/0) used for the internal network. The router is the default gateway for the 192.168.0.0 /24 network. CEF is enabled globally and on all the interfaces.

As is, I experiencing strange web browsing issues such as some pages loading fine and others not loading at all. If i shut down any one of the external interfaces then performance is normal in terms of web browsing.

On the 192.168.0.0 /24 network there are several sub-LANs seperated by a NAT device. For example:

Internet - router - >[ 192.168.0.0 / 24 ] - NATdeviceA - [ 10.10.10.0 /24 ]
- NATdeviceB - [ 10.10.20.0 /24 ]

I notice that web browsing behind any of the subsequent NAT devices (i.e. NATdeviceA) is particularly sporadic. There is a static nat entry on the 2600 router that associates NATdeviceA's external interface with a public IP from the s0/0 provider so that it can be publicly accessible. There is also a static NAT entry on the 2600 router that associates NATdeviceA's external interface with a public IP from the e1/0 provider so that is can be publicly accessible from that provider as well.

I was thinking that the issue could be realted to the static NAT entries for the NATdeviceA causing CEF to somehow get confused. I tried to make this as clear as possible and I have posted the running config below edited for public viewing. Any advice here is greatly appreciated.

Building configuration...

Current configuration : 2889 bytes
!
version 12.3
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname xxxxxxx
!
boot-start-marker
boot-end-marker
!
enable secret 5 xxxxxxxxxxxxxxxxxx
enable password xxxxxxxxxx
!
no aaa new-model
!
resource policy
!
no network-clock-participate slot 1
no network-clock-participate wic 0
ip subnet-zero
ip cef
!
!
no ip dhcp use vrf connected
no ip dhcp conflict logging
ip dhcp excluded-address 192.168.0.125 192.168.0.254
!
ip dhcp pool DHCP
network 192.168.0.0 255.255.255.0
domain-name xxxxxxxx.org
default-router 192.168.0.1
dns-server 65.x.x.66 65.x.x.67 209.x.x.2 209.x.x.2 66.x.x.8
netbios-node-type h-node
lease infinite
!
!
ip name-server 65.x.x.66
ip name-server 65.x.x.67
ip name-server 209.x.x.2
ip name-server 209.x.x.2
ip sla monitor 6
type echo protocol ipIcmpEcho 208.x.x.x <-- (next hop for e1/0 provider)
timeout 1000
threshold 250
frequency 5
ip sla monitor schedule 6 life forever start-time now
no ftp-server write-enable
!
!
!
track 100 rtr 6 reachability
!
!
interface FastEthernet0/0
description Connection to internal network
ip address 192.168.0.1 255.255.255.0
ip nat inside
no ip mroute-cache
duplex auto
speed auto
!
interface Serial0/0
ip address 65.x.x.x 255.255.255.252
ip nat outside
no dce-terminal-timing-enable
!
interface Serial0/1
no ip address
shutdown
no dce-terminal-timing-enable
!
interface Ethernet1/0
ip address 208.x.x.x 255.255.255.248
ip nat outside
shutdown
full-duplex
!
ip local policy route-map PrimarySAAPolicy
ip classless
ip route 0.0.0.0 0.0.0.0 208.x.x.x track 100 <---(e1/0 provider next hop)
ip route 0.0.0.0 0.0.0.0 65.x.x.x <---(s0/0 provider next hop)
!
no ip http server
ip nat inside source route-map BBVI interface Ethernet1/0 overload
ip nat inside source route-map t1map interface Serial0/0 overload
ip nat inside source static tcp 192.168.0.30 3389 65.x.x.x 3389 extendable
ip nat inside source static tcp 192.168.0.30 6881 65.x.x.x 6881 extendable
ip nat inside source static tcp 192.168.0.252 80 65.x.x.x 8800 extendable
ip nat inside source static tcp 192.168.0.253 80 65.x.x.x 8801 extendable
ip nat inside source static tcp 192.168.0.230 80 65.x.x.x 8802 extendable
ip nat inside source static 192.168.0.151 65.a.a.a <-------------------- (this NATdeviceA)
ip nat inside source static 192.168.0.161 65.x.x.x
ip nat inside source static 192.168.0.152 208.a.a.a <-------------------- (this NATdeviceA)
!
access-list 10 permit 192.168.0.0 0.0.0.255
access-list 12 permit 192.168.0.0 0.0.0.255
access-list 101 permit icmp any host 208.x.x.x echo <---(e1/0 provider next hop)
snmp-server community public RO
route-map BBVI permit 10
match interface Ethernet1/0
!
route-map t1map permit 10
match interface Serial0/0
!
route-map PrimarySAAPolicy permit 12
match ip address 101
set interface Null0
set ip next-hop 208.x.x.x <---(e1/0 provider next hop)
!
!
!
control-plane
!
!
line con 0
line aux 0
line vty 0 4
password xxxxxxxxx
login
!
!
end

xxxxxxxx#

mikebernhardt

I think the problem is because NAT is stateful. When an outbound packet is sent and translated through one of the ISP interfaces, the reply may not always come back through the same interface and if not, it will be dropped because there's no translation for the destination port. Usually when people use NAT on 2 interfaces as you are, one interface is primary and the other is backup.

andreacadia

ASKER

Thanks Mike,

So does this mean that i cannot get this scenario to fully work?? Can you suggest any workarounds?

mikebernhardt

If you're using NAT and 2 ISPs I think you need to just use one as a backup link in case the primary goes down. The only workaround I can think of is to put the NAT on a 2nd router behind the ISP-connected one. Then the outside router would use all public addressing and do the load-balancing and feed it all to the inside router that does the NAT.

noctot

You can get this to work easily. Mike is correct about the two NAT translations being the problem. I see people make this mistake often. The thing is, you don't need to be running NAT twice.

Internet - router - >[ 192.168.0.0 / 24 ] - NATdeviceA - [ 10.10.10.0 /24 ]
- NATdeviceB - [ 10.10.20.0 /24 ]

Just disable NAT on both NATdeviceA and NATdeviceB. Then change your NAT maps in the 2600 to point to the actual 10.10.10.x or 10.10.20.x addresses.

andreacadia

ASKER

Thanks noctot,

In this situation it would be preferred to have the NAT device for security purposes.

Does anyone have a suggestion for the as-is network layout to work properly?

mikebernhardt

The problem isn't that you have 2 levels of NAT, it's that your outbound traffic is being NATted through 2 interfaces to 2 different addresses before heading to the ISP.

I didn't notice that you are NATting twice though. What I would suggest then is to not NAT on the 2600. You can still use private addressing on the links to the inside NAT devices.

With no NAT on the 2600 you can load-balance to your heart's content and it should work fine.

mikebernhardt

Correction to above- you can use private addressing on the inside links as long as the addresses your NAT devices translate to are public. You'd set up static routes in the 2600 pointing to those addresses. Without knowing what you're doing I don't know if that would work. You would need a small block of public addressing from your ISPs aside from the interface addresses you have now.

andreacadia

ASKER

Hi Mike,

Not sure i am following you above...

Based on your comments above, if were to only have a 1 static NAT entry for one isp back to any given NatDevice my problem should disappear? Basically any NAT devices would only be accesible through one ISP publicly at that point. Does this solve the issue?

mikebernhardt

In your current situation you don't control which interface a given packet will NAT and leave through, CEF does. So if it forwards a packet to the wrong interface it will end up dropped. You would need to implement policy routing to ensure that a given source address exits a particular interface. Then CEF load-balancing wouldn't do anything for you, and failover gets more complicated as well.

That's why it usually works better to just use one ISP link at a time, with the other one as a hot standby.

noctot

I still like my suggestion of dropping one layer of NAT. The NAT devices aren't adding any additional security to outside intrusions. If someone gets into the Cisco then they will have access to the NAT translations in the config and know exactly how to get into the next level of devices. If you are looking for providing security between the two networks behind the NAT devices then a simple access list would be even more effective than NAT.

andreacadia

ASKER

I know that load balancing should not be beyond the scope of native cisco functionality. In my circumstance what really is the issue with load balancing and which part of the configuration needs to be changed in your opinions. The primary outcome of this configuration should be load balance across the two providers using CEF. Is there a better mechanism than CEF for my purposes?

ASKER CERTIFIED SOLUTION

mikebernhardt

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

andreacadia

ASKER

Thanks Mike that makes a lot of sense. It seems like the real issue is the CEF 'per destination' implementation not being truly session based. According to what i have read from Cisco CEF per destination should be able to guarantee the arrival of packets in order. The sacrifice is that the load balancing is not truly 50/50. Perhaps this does not pertain to a situation where there are two different ISPs.

What do you think about this solution:

Remove NAT altogether from the 2600 router. The external interfaces will remain the same. I obtain a block of IPs from my providers then assign public IPs on the inside interfaces so it looks like this:

ISPA ----- e1/0 (65.x.x.x) Fa0/0 (65.x.x.x)
Internet--- [2600 router] Fa1/1 (65.1.1.1) ---- [Firewall/NAT] (65.x.x.x)-----192.168.0.0/24
ISPB ------s0/0 (208.x.x.x) Fa0/1 (208.x.x.x)

- Assign the Firewall a default route to the Fa1/1 interface
- The router then has CEF configured with two equal cost default routes
- NAT happens only at firewall

See where i'm going?

mikebernhardt

Well it should work except that ISP B will not advertise the block you got from ISP A. So although you can load-balance outbound, all the inbound traffic will come back via ISP A. So if ISP A goes down you're dead in the water. If you can somehow get your self your own address block then you can have both ISPs advertise it (but you may need to use BGP at that point, it depends on how the ISPs want to work it).

andreacadia

ASKER

Sorry i took so long to award these points but we only recently concluded that this is not going to work for us. As you said, I believe the issue here is the fact the two ISPs are involved. When we test the config, it seems that packets would go out but sometimes not come back as web browsing was near impossible. As soon as one of the interfaces was shut down it would work no problem. I though that we could accomplish this without BGP involved but apparently only if 1 ISP is involved. I.e. two the T1s from the same ISP. We decided to proceed with a product from www.Fatpipeinc.com called the WARP. This will also give us the ability to due inbound load balancing. Thanks for your help.