[Last Call] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Cisco Express Forwarding load balance scenario

Posted on 2006-04-25
16
Medium Priority
?
543 Views
Last Modified: 2006-11-18
I have configured a 2600 router for load balancing using CEF.  The router has 2 Internet connections ( e1/0 and s0/0) from different providers with 2 equal cost default routes to the respective providers next hop address.  I also have a SAA probe configured to track the state of one of the default routes for failover.  NAT is configured on both  external interfaces.  There is one inside interface (Fa0/0) used for the internal network.  The router is the default gateway for the 192.168.0.0 /24 network.  CEF is enabled globally and on all the interfaces.

As is, I experiencing strange web browsing issues such as some pages loading fine and others not loading at all.  If i shut down any one of the external interfaces then performance is normal in terms of web browsing.  

On the 192.168.0.0 /24 network there are several sub-LANs seperated by a NAT device.  For example:

Internet - router - >[ 192.168.0.0 / 24 ] - NATdeviceA - [ 10.10.10.0 /24 ]
                                                          - NATdeviceB - [ 10.10.20.0 /24 ]

I notice that web browsing behind any of the subsequent NAT devices (i.e. NATdeviceA) is particularly sporadic.  There is a static nat entry on the 2600 router that associates NATdeviceA's external interface with a public IP from the s0/0 provider so that it can be publicly accessible.  There is also a static NAT entry on the 2600 router that associates NATdeviceA's external interface with a public IP from the e1/0 provider so that is can be publicly accessible from that provider as well.  

I was thinking that the issue could be realted to the static NAT entries for the NATdeviceA causing CEF to somehow get confused.  I tried to make this as clear as possible and I have posted the running config below edited for public viewing. Any advice here is greatly appreciated.




Building configuration...

Current configuration : 2889 bytes
!
version 12.3
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname xxxxxxx
!
boot-start-marker
boot-end-marker
!
enable secret 5 xxxxxxxxxxxxxxxxxx
enable password xxxxxxxxxx
!
no aaa new-model
!
resource policy
!
no network-clock-participate slot 1
no network-clock-participate wic 0
ip subnet-zero
ip cef
!
!
no ip dhcp use vrf connected
no ip dhcp conflict logging
ip dhcp excluded-address 192.168.0.125 192.168.0.254
!
ip dhcp pool DHCP
   network 192.168.0.0 255.255.255.0
   domain-name xxxxxxxx.org
   default-router 192.168.0.1
   dns-server 65.x.x.66 65.x.x.67 209.x.x.2 209.x.x.2 66.x.x.8
   netbios-node-type h-node
   lease infinite
!
!
ip name-server 65.x.x.66
ip name-server 65.x.x.67
ip name-server 209.x.x.2
ip name-server 209.x.x.2
ip sla monitor 6
 type echo protocol ipIcmpEcho 208.x.x.x <-- (next hop for e1/0 provider)
 timeout 1000
 threshold 250
 frequency 5
ip sla monitor schedule 6 life forever start-time now
no ftp-server write-enable
!
!
!
track 100 rtr 6 reachability
!
!
interface FastEthernet0/0
 description Connection to internal network
 ip address 192.168.0.1 255.255.255.0
 ip nat inside
 no ip mroute-cache
 duplex auto
 speed auto
!
interface Serial0/0
 ip address 65.x.x.x 255.255.255.252
 ip nat outside
 no dce-terminal-timing-enable
!
interface Serial0/1
 no ip address
 shutdown
 no dce-terminal-timing-enable
!
interface Ethernet1/0
 ip address 208.x.x.x 255.255.255.248
 ip nat outside
 shutdown
 full-duplex
!
ip local policy route-map PrimarySAAPolicy
ip classless
ip route 0.0.0.0 0.0.0.0 208.x.x.x track 100     <---(e1/0 provider next hop)
ip route 0.0.0.0 0.0.0.0 65.x.x.x                <---(s0/0 provider next hop)
!
no ip http server
ip nat inside source route-map BBVI interface Ethernet1/0 overload
ip nat inside source route-map t1map interface Serial0/0 overload
ip nat inside source static tcp 192.168.0.30 3389 65.x.x.x 3389 extendable
ip nat inside source static tcp 192.168.0.30 6881 65.x.x.x 6881 extendable
ip nat inside source static tcp 192.168.0.252 80 65.x.x.x 8800 extendable
ip nat inside source static tcp 192.168.0.253 80 65.x.x.x 8801 extendable
ip nat inside source static tcp 192.168.0.230 80 65.x.x.x 8802 extendable
ip nat inside source static 192.168.0.151 65.a.a.a  <-------------------- (this NATdeviceA)
ip nat inside source static 192.168.0.161 65.x.x.x
ip nat inside source static 192.168.0.152 208.a.a.a <-------------------- (this NATdeviceA)
!
access-list 10 permit 192.168.0.0 0.0.0.255
access-list 12 permit 192.168.0.0 0.0.0.255
access-list 101 permit icmp any host 208.x.x.x echo <---(e1/0 provider next hop)
snmp-server community public RO
route-map BBVI permit 10
 match interface Ethernet1/0
!
route-map t1map permit 10
 match interface Serial0/0
!
route-map PrimarySAAPolicy permit 12
 match ip address 101
 set interface Null0
 set ip next-hop 208.x.x.x   <---(e1/0 provider next hop)
!
!
!
control-plane
!
!
line con 0
line aux 0
line vty 0 4
 password xxxxxxxxx
 login
!
!
end

xxxxxxxx#
0
Comment
Question by:andreacadia
  • 7
  • 6
  • 2
15 Comments
 
LVL 28

Expert Comment

by:mikebernhardt
ID: 16536400
I think the problem is because NAT is stateful. When an outbound packet is sent and translated through one of the ISP interfaces, the reply may not always come back through the same interface and if not, it will be dropped because there's no translation for the destination port. Usually when people use NAT on 2 interfaces as you are, one interface is primary and the other is backup.
0
 

Author Comment

by:andreacadia
ID: 16537260
Thanks Mike,

So does this mean that i cannot get this scenario to fully work??  Can you suggest any workarounds?
0
 
LVL 28

Expert Comment

by:mikebernhardt
ID: 16538437
If you're using NAT and 2 ISPs I think you need to just use one as a backup link in case the primary goes down. The only workaround I can think of is to put the NAT on a 2nd router behind the ISP-connected one. Then the outside router would use all public addressing and do the load-balancing and feed it all to the inside router that does the NAT.
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
LVL 3

Expert Comment

by:noctot
ID: 16539392
  You can get this to work easily. Mike is correct about the two NAT translations being the problem. I see people make this mistake often. The thing is, you don't need to be running NAT twice.

Internet - router - >[ 192.168.0.0 / 24 ] - NATdeviceA - [ 10.10.10.0 /24 ]
                                                          - NATdeviceB - [ 10.10.20.0 /24 ]

Just disable NAT on both NATdeviceA and NATdeviceB. Then change your NAT maps in the 2600 to point to the actual 10.10.10.x or 10.10.20.x addresses.
0
 

Author Comment

by:andreacadia
ID: 16539747
Thanks noctot,

In this situation it would be preferred to have the NAT device for security purposes.  

Does anyone have a suggestion for the as-is network layout to work properly?
0
 
LVL 28

Expert Comment

by:mikebernhardt
ID: 16545437
The problem isn't that you have 2 levels of NAT, it's that your outbound traffic is being NATted through 2 interfaces to 2 different addresses before heading to the ISP.

I didn't notice that you are NATting twice though. What I would suggest then is to not NAT on the 2600. You can still use private addressing on the links to the inside NAT devices.

With no NAT on the 2600 you can load-balance to your heart's content and it should work fine.
0
 
LVL 28

Expert Comment

by:mikebernhardt
ID: 16545579
Correction to above- you can use private addressing on the inside links as long as the addresses your NAT devices translate to are public. You'd set up static routes in the 2600 pointing to those addresses. Without knowing what you're doing I don't know if that would work. You would need a small block of public addressing from your ISPs aside from the interface addresses you have now.
0
 

Author Comment

by:andreacadia
ID: 16546633
Hi Mike,

Not sure i am following you above...

Based on your comments above, if were to only have a 1 static NAT entry for one isp back to any given NatDevice my problem should disappear?  Basically any NAT devices would only be accesible through one ISP publicly at that point.  Does this solve the issue?
0
 
LVL 28

Expert Comment

by:mikebernhardt
ID: 16548117
In your current situation you don't control which interface a given packet will NAT and leave through, CEF does. So if it forwards a packet to the wrong interface it will end up dropped. You would need to implement policy routing to ensure that a given source address exits a particular interface. Then CEF load-balancing wouldn't do anything for you, and failover gets more complicated as well.

That's why it usually works better to just use one ISP link at a time, with the other one as a hot standby.
0
 
LVL 3

Expert Comment

by:noctot
ID: 16548371
  I still like my suggestion of dropping one layer of NAT. The NAT devices aren't adding any additional security to outside intrusions. If someone gets into the Cisco then they will have access to the NAT translations in the config and know exactly how to get into the next level of devices. If you are looking for providing security between the two networks behind the NAT devices then a simple access list would be even more effective than NAT.

0
 

Author Comment

by:andreacadia
ID: 16548486
I know that load balancing should not be beyond the scope of native cisco functionality.  In my circumstance what really is the issue with load balancing and which part of the configuration needs to be changed in your opinions.  The primary outcome of this configuration should be load balance across the two providers using CEF.  Is there a better mechanism than CEF for my purposes?
0
 
LVL 28

Accepted Solution

by:
mikebernhardt earned 2000 total points
ID: 16548972
I don't think the 2 layers of NAT are a problem, the 2600 still has to translate them. I do agree that they don't increase security and that good access lists are better. Or even better, a firewall. Cisco's firewall feature set can even provide a degree of stateful firewalling.

The problem is not dual NAT, or load balancing. The problem is using both together on the same router when 2 ISPs are involved. If it was one ISP you could ask them for a block of public IPs and do your NAT to public IPs on the inner NAT devices. Then the 2600 just load-balances.

 Look at this web session when using per-packet CEF load-balancing:

inside packet 1 (http get): 192.168.0.50:1025 --> 65.x.x.x:25001 to www.yahoo.com:80
data returns to 65.x.x.x:25001  and NATs to 192.168.0.50:1025
inside packet 2 (tcp ack): 192.168.0.50:1025 -->208.x.x.x:16233 to www.yahoo.com:80

So, what does yahoo do? They just received an ACK from an IP:Port with which they did not have a session. The ACK will be dropped, you're waiting for data, they're waiting for an ACK. Now, if you use per session CEF load-balancing it isn't quite as bad, but HTTP often opens multiple sockets and if they don't line up right at the server the same thing will happen.

Again, you can use policy routing to force certain source addresses or subnets to take one or the other path. Or try to use one ISP with the 2 different access methods for diversity- not as good as 2 ISPs but maybe it's good enough...
0
 

Author Comment

by:andreacadia
ID: 16549359
Thanks Mike that makes a lot of sense. It seems like the real issue is the CEF 'per destination' implementation not being truly session based.  According to what i have read from Cisco CEF per destination should be able to guarantee the arrival of packets in order.  The sacrifice is that the load balancing is not truly 50/50.  Perhaps this does not pertain to a situation where there are two different ISPs.

What do you think about this solution:

Remove NAT altogether from the 2600 router.  The external interfaces will remain the same.  I obtain a block of IPs from my providers then assign public IPs on the inside interfaces so it looks like this:

                ISPA ----- e1/0 (65.x.x.x)          Fa0/0 (65.x.x.x)
Internet---                                [2600 router]                   Fa1/1 (65.1.1.1) ---- [Firewall/NAT] (65.x.x.x)-----192.168.0.0/24
                ISPB ------s0/0 (208.x.x.x)         Fa0/1 (208.x.x.x)

- Assign the Firewall a default route to the Fa1/1 interface
- The router then has CEF configured with two equal cost default routes
- NAT happens only at firewall


See where i'm going?


0
 
LVL 28

Expert Comment

by:mikebernhardt
ID: 16555125
Well it should work except that ISP B will not advertise the block you got from ISP A. So although you can load-balance outbound, all the inbound traffic will come back via ISP A. So if ISP A goes down you're dead in the water. If you can somehow get your self your own address block then you can have both ISPs advertise it (but you may need to use BGP at that point, it depends on how the ISPs want to work it).
0
 

Author Comment

by:andreacadia
ID: 16784022
Sorry i took so long to award these points but we only recently concluded that this is not going to work for us.  As you said, I believe the issue here is the fact the two ISPs are involved.  When we test the config, it seems that packets would go out but sometimes not come back as web browsing was near impossible.  As soon as one of the interfaces was shut down it would work no problem.  I though that we could accomplish this without BGP involved but apparently only if 1 ISP is involved.  I.e. two the T1s from the same ISP.   We decided to proceed with a product from www.Fatpipeinc.com called the WARP.   This will also give us the ability to due inbound load balancing.  Thanks for your help.
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In the hope of saving someone else's sanity... About a year ago we bought a Cisco 1921 router with two ADSL/VDSL EHWIC cards to load balance local network traffic over the two broadband lines we have, but we couldn't get the routing to work consi…
In the world of WAN, QoS is a pretty important topic for most, if not all, networks. Some WAN technologies have QoS mechanisms built in, but others, such as some L2 WAN's, don't have QoS control in the provider cloud.
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

829 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question