Solved

Cisco 3550 with Multiple ISPs

Posted on 2010-09-15
19
839 Views
Last Modified: 2012-05-10
Here is the setup:

- Cisco 3550 doing core internal (VLAN) routing (10.1.1.1)
- ISP1 with 1841 router connected to ISPs equipment (10.1.1.2)
- ISP2 with 1841 router connected to ISPs equipment (10.1.1.3)
- ISP1 is reliable (but lower speed) bandwidth used for static NATs for web based services hosted on premises (email, web, TS, etc.)
- ISP2 is less-reliable (but faster, cheaper) bandwidth used to Internet surfing

My problem is that if I point my default router on the 3550 to the 1841 of ISP2, then my services that I have running through ISP1 (email, etc) no longer route correctly inbound. It's as if the packets can't return back out the interface from which they were sent. I've tried adding routes on the 3550 for the external subnets that have been assigned by ISP1 to the 1841 for ISP1, but this hasn't helped.

Thanks.
0
Comment
Question by:joshlamerritt
  • 8
  • 8
  • 3
19 Comments
 
LVL 57

Accepted Solution

by:
giltjr earned 500 total points
ID: 33682874
IP routing is based on destination IP address, not what interface the traffic came in on.

There are two options I can think of off hand.

1) Setup policy based routing where you route based on source IP address instead of destination IP address for outbound traffic.  Read:

http://www.cisco.com/en/US/tech/tk364/technologies_configuration_example09186a00802135d3.shtml

2) Setup your 1841's to do NAT'ing on the inbound traffic so that all external IP addresses get NAT'ed to one of your internal addresses.  Then setup the route tables on the 3550 based that.
0
 
LVL 2

Author Comment

by:joshlamerritt
ID: 33683061
Option 2 is already in place. Here is an example of a NAT statement that already exists on ISP1's 1841 (external IPs changed):

-----start snip-----
ip nat inside source list 1 interface FastEthernet0/0 overload
ip nat inside source static 10.2.1.22 1.1.1.1
ip nat inside source static 10.2.1.23 1.1.1.2
-----end snip-----

1.2.1.22 is an email server. When the default route of the 3550 points to 10.1.1.2 (internal interface of the ISP1 1841) then 10.2.1.22 can be contacted from the outside via 1.1.1.1. However, if the default route of the 3550 is changed to 10.1.1.3 (internal interface of ISP2 1841), inbound traffic to 10.2.1.22 no longer works.

From your comment above, it looks like I am missing a route on the 3550. What would that route be?
0
 
LVL 57

Expert Comment

by:giltjr
ID: 33684361
No, that is NAT'ing a host inside your network to an IP address outside of your network.

What option#2 is the other way around.  That is a public IP address appears to be an internal address on your network.

Something like:

ip nat pool net-10 10.0.1.0 10.0.1.255 prefix-length 24
ip nat outside source list 1 pool net-10

What this will do is all external IP addresses will appear to your internal network to be in the IP subnet 10.0.1.0/24.


Above taken from:

http://www.cisco.com/en/US/docs/ios/12_3/ipaddr/command/reference/ip1_i2g.html#wp1079462
0
 
LVL 10

Expert Comment

by:ddiazp
ID: 33684809
Could you think of these routes the following way?

All web traffic (port 80 and 443) to be sent to ISP2 through policy based routing
Everything else to ISP1

or

Traffic to specific pool of IP addresses sent to  ISP1 through static routes
Everything else to ISP2


I personally don't see how changing natting can make this work otherwise.

Also, is there a way that ISP2 can have static routes to your email's network through ISP1? That way your default gateway would stay to ISP2, ISP2 would go to ISP1 to contact servers, ISP1 back to ISP2 with response, ISP2 to your switch. This would work if ISP1, ISP2 and you are on the same subnet
0
 
LVL 57

Expert Comment

by:giltjr
ID: 33685058
If you do "reverse" natting, that is NAT the public IP address to an internal private, but setup two unique pools, one for each interface, you route back the way it came in.  That is you have:

         | eth0 <--> 1841#1
3550 |
         | eth1 <--> 1841#2

On 1841#1 you do:

ip nat pool net-88 10.0.88.0 10.0.88.255 prefix-length 24
ip nat outside source list 1 pool net-88

On 1841#2

ip nat pool net-99 10.0.99.0 10.0.99.255 prefix-length 24
ip nat outside source list 1 pool net-99

Now the 3550 will see IP addresses 10.0.88.1-254 for everything coming from ISP#1 and 10.0.99.1-254 for everything coming from ISP#2.

So the 3550 would have a route for 10.0.88.0/24 that points to 1841#1 and a route for 10.0.99.0/24 that points to 1841#2.


If 254 addresses are not enough, then you just change the pools to 10.88.0.0 10.88.255.255 prefix-length 16.
0
 
LVL 10

Expert Comment

by:ddiazp
ID: 33685614
Agree, but what happens when the traffic originates from behind the 3560? the default gateway will have to be ISP1 and nothing would have changed since ISP#2 would be unused. He would have to connect to his existing servers through the new NATed ip address, and create static nats for his servers on ISP#1 and ISP#2, update local DNS, etc.

Unless I'm missing the whole point.


Still, this sounds like an interesting scenario, I'll go ahead and test this out myself :)


0
 
LVL 57

Expert Comment

by:giltjr
ID: 33685852
ddiazp, good catch.

I either over thinking, under thinking, maybe not thinking at all. : )

The best solution is policy based routing to setup source IP address.  This allows you to direct outbound traffic based on what host it is coming from not what host it is going to.

I have to think about my second option some more.  At a minimum:

The reverse NAT setup would only need to be done on 1841#1 going to ISP#1.  This would handle all traffic that originates from the outside (Internet) into your network and allow it to go back via ISP#1.

The default route would then point to 1841#2.  This would handle all "web surfing" traffic.

The last thing is to figure out how to handle traffic that originates inside that must go out via ISP#1.  Either polciy based routing for source IP address or have static NAT's in both 1841's for both sets of IP addresses.  Doing static NAT's for addresses in ISP2's range would allow traffic originating internally to flow outbound via ISP2.  But would cause outbound e-mail and DNS queries to flow over ISP2.  Now this could free up bandwidth on ISP1 and could improve the performance of applications where the request originate from the Internet.
0
 
LVL 10

Expert Comment

by:ddiazp
ID: 33686100
I'd still suggest the easiest way to do this in my opinion which is policy based routing based on destination IP instead:

create ACL specifying destination IPs to servers on ISP#1 (email, DNS, whatever) on the switch
apply the ACL to a policy-map type inspect statement
enter ip next hop for ISP#1
apply policy on inside interface on layer3 switch

leave default gateway on ISP#2

done

when you say "less reliable" how reliable is it? is there an SLA on this link? NATing is great but i'd steer away from it in this scenario.



Also, remember these ISP routers may not be accessible to us
0
 
LVL 2

Author Comment

by:joshlamerritt
ID: 33691709
Thanks for you help ddizap and giltjr. It looks like a policy map is going to be my course of action. Here are a couple of more details and answers to questions:

ISP1 is a partial T1 (shared with voice). ISP2 is a cable internet provider.

With the policy mapping setup and the default gateway set to ISP2 (I do have access to both routers, btw), it would appear that this configuration would not allow for failover to ISP1 if/when ISP2 goes down. Is this correct?
0
Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

 
LVL 57

Expert Comment

by:giltjr
ID: 33692282
Actually you can have the router setup to fail over to ISP1 if ISP2 goes down.

You can use the SLA feature (I think the 3550 supports this).

ip sla 10
icmp-echo x.x.x.x
timeout 1000
frequency 3
ip sla schedule 10 life forever start-time now
ip route 0.0.0.0 0.0.0.0 10.0.0.1 track 10 (this should be your normal default route)
ip route 0.0.0.0 0.0.0.0 192.168.0.1 20    (this should be the route want to take take if the normal default fails).


The address x.x.x.x should be the 1st IP address outside your building on ISP2's connection.
0
 
LVL 2

Author Comment

by:joshlamerritt
ID: 33782102
Thanks for all of you help on this, guys. I am going to begin reconfiguring the network tonight using your recommendations. I'll let you know how it goes.
0
 
LVL 2

Author Comment

by:joshlamerritt
ID: 33782142
I spoke too soon. The 3550 is running SMI, and it looks like PBR requires EMI. Back to the drawing board...
0
 
LVL 57

Expert Comment

by:giltjr
ID: 33782203
Yes, PBR requires EMI.
0
 
LVL 2

Author Comment

by:joshlamerritt
ID: 33782316
Hmmm, now I've really confused myself. I'm routing with a 3550-12T, which appears to have the route-map commands, so it looks like I'm good to go. Back to the plan to deploy PBR.
0
 
LVL 57

Expert Comment

by:giltjr
ID: 33782548
12T's came with EMI standard.   No I did not just know this, I found it at:

http://www.cisco.com/en/US/products/sw/iosswrel/ps1833/prod_bulletin09186a00800ab81f.html
0
 
LVL 2

Author Comment

by:joshlamerritt
ID: 33783142
Ok, here is the plan. Current config as follows:

-----start snip-----
PCCA_North_MC_C3550-12T#sh run
Building configuration...

Current configuration : 5537 bytes
!
! Last configuration change at 14:58:29 EDT Tue Sep 28 2010
! NVRAM config last updated at 15:03:01 EDT Tue Sep 14 2010
!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname PCCA_North_MC_C3550-12T
!
enable secret <removed>
!
username provision privilege 15 password <removed>
no aaa new-model
clock timezone EST -5
clock summer-time EDT recurring
ip subnet-zero
ip routing
ip name-server 216.54.161.55
ip name-server 216.54.161.56
ip dhcp excluded-address 10.9.0.1 10.9.2.0
ip dhcp excluded-address 10.9.2.255 10.9.255.254
ip dhcp excluded-address 10.2.1.1 10.2.1.99
!
ip dhcp pool VOICE
   import all
   network 10.9.0.0 255.255.0.0
   domain-name pcca.local
   dns-server 10.1.1.1
   option 150 ip 10.9.1.10
   default-router 10.9.1.1
   lease 14
!
ip dhcp pool SERVERS
   import all
   network 10.2.1.0 255.255.255.0
   domain-name pcca-local
   dns-server 216.54.161.55 216.54.161.56
   default-router 10.2.1.1
   lease 8
!
!
!
<crypto commands removed>
!
!
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
!
!
!
!
!
interface GigabitEthernet0/1
 description ISCSI SAN PORTS
 switchport access vlan 22
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet0/2
 description ISCSI SAN PORTS
 switchport access vlan 23
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet0/3
 description ISCSI SAN PORTS
 switchport access vlan 22
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet0/4
 description ISCSI SAN PORTS
 switchport access vlan 23
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet0/5
 description ISCSI SAN PORTS
 switchport access vlan 22
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet0/6
 description ISCSI SAN PORTS
 switchport access vlan 23
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet0/7
 description ISCSI SAN PORTS
 switchport access vlan 22
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet0/8
 description ISCSI SAN PORTS
 switchport access vlan 23
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet0/9
 description LINK TO SERVER HOST2
 switchport access vlan 20
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet0/10
 description LINK TO SERVER HOST1
 switchport access vlan 20
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet0/11
 description LINK TO SOUTH BUILDING
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 10,51,90,250,251
 switchport mode trunk
!
interface GigabitEthernet0/12
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 10,20,22,51,90,250
 switchport mode trunk
!
interface Vlan1
 no ip address
 shutdown
!
interface Vlan10
 description PCCA LAN VLAN
 ip address 10.1.1.1 255.255.0.0
 ip helper-address 10.2.1.24
!
interface Vlan20
 description SERVER VLAN
 ip address 10.2.1.1 255.255.255.0
!
interface Vlan51
 description PCCA Private WLAN
 ip address 10.5.1.1 255.255.255.0
 ip helper-address 10.2.1.24
!
interface Vlan90
 description PCCA Voice VLAN
 ip address 10.9.1.1 255.255.0.0
!
interface Vlan250
 description Bogan Paging System
 ip address 10.254.10.1 255.255.255.0
 ip pim sparse-dense-mode
!
interface Vlan251
 description Access Control System
 ip address 10.254.11.1 255.255.255.0
!
ip classless
ip route 0.0.0.0 0.0.0.0 10.1.1.2
ip http server
ip http authentication local
ip http secure-server
!
!
!
control-plane
!
!
line con 0
 password <removed>
 login
line vty 0 4
 password <removed>
 login
line vty 5 15
 password <removed>
 login
!
ntp clock-period 17180481
ntp server 192.43.244.18
end
-----end snip-----

Here are the commands I intend to implement this evening:

-----start snip-----
(config)# sdm prefer routing
# wr mem
# reload

(config)# access-list 10 permit any any
(config)# route-map ISP1 permit 10
(config-route-map)# match ip address 10
(config-route-map)# set ip next-hop 10.1.1.2

(config)# access-list 10 permit any any
(config)# route-map ISP2 permit 10
(config-route-map)# match ip address 10
(config-route-map)# set ip next-hop 10.1.1.3
-----end snip-----

I gathered this info by reading through the document that was suggested earlier in this thread. Am I processing this correctly? Missing anything?

Thanks so much for your help.
0
 
LVL 2

Author Comment

by:joshlamerritt
ID: 33783213
Opps. The second access-list command set should have been access-list 11 for ISP2. So:

-----start snip-----
(config)# access-list 10 permit any any
(config)# route-map ISP1 permit 10
(config-route-map)# match ip address 10
(config-route-map)# set ip next-hop 10.1.1.2

(config)# access-list 11 permit any any
(config)# route-map ISP2 permit 11
(config-route-map)# match ip address 11
(config-route-map)# set ip next-hop 10.1.1.3
-----end snip-----

Am I on the right track?
0
 
LVL 2

Author Comment

by:joshlamerritt
ID: 33798260
Ok guys, we're almost there. PBR is working. I'll post my final config shortly. The last remaining issue is the IP SLA commands. The issue I'm running in to is that once it fails over to the secondary route, when the primary route comes back online, it doesn't switch back over to the primary route.

Here is my ip sla config:

-----start snip-----
ip sla 10
 icmp-echo 71.43.74.225
 timeout 1000
 frequency 3
ip sla schedule 10 life forever start-time now

ip route 0.0.0.0 0.0.0.0 10.1.1.3 track 10
ip route 0.0.0.0 0.0.0.0 10.1.1.2 2
-----end snip-----

Thoughts?
0
 
LVL 57

Expert Comment

by:giltjr
ID: 33810297
Thanks for the points.

As to the route fail over and recovery, you may want to read:

http://www.ciscoblog.com/archives/2008/08/dynamic_failove.html

I think after the ip slao schedule ... you need to issue:

track 10 rtr 1 reachability

You also need to make sure that if you defined your default route to 10.1.1.3 PRIOR to setting this up you that you need to delete it and then re-define with the track 10 on the end.
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

Meet the world's only “Transparent Cloud™” from Superb Internet Corporation. Now, you can experience firsthand a cloud platform that consistently outperforms Amazon Web Services (AWS), IBM’s Softlayer, and Microsoft’s Azure when it comes to CPU and …
ADCs have gained traction within the last decade, largely due to increased demand for legacy load balancing appliances to handle more advanced application delivery requirements and improve application performance.
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now