Cisco 3550 with Multiple ISPs
Here is the setup:
- Cisco 3550 doing core internal (VLAN) routing (10.1.1.1)
- ISP1 with 1841 router connected to ISPs equipment (10.1.1.2)
- ISP2 with 1841 router connected to ISPs equipment (10.1.1.3)
- ISP1 is reliable (but lower speed) bandwidth used for static NATs for web based services hosted on premises (email, web, TS, etc.)
- ISP2 is less-reliable (but faster, cheaper) bandwidth used to Internet surfing
My problem is that if I point my default router on the 3550 to the 1841 of ISP2, then my services that I have running through ISP1 (email, etc) no longer route correctly inbound. It's as if the packets can't return back out the interface from which they were sent. I've tried adding routes on the 3550 for the external subnets that have been assigned by ISP1 to the 1841 for ISP1, but this hasn't helped.
Thanks.
- Cisco 3550 doing core internal (VLAN) routing (10.1.1.1)
- ISP1 with 1841 router connected to ISPs equipment (10.1.1.2)
- ISP2 with 1841 router connected to ISPs equipment (10.1.1.3)
- ISP1 is reliable (but lower speed) bandwidth used for static NATs for web based services hosted on premises (email, web, TS, etc.)
- ISP2 is less-reliable (but faster, cheaper) bandwidth used to Internet surfing
My problem is that if I point my default router on the 3550 to the 1841 of ISP2, then my services that I have running through ISP1 (email, etc) no longer route correctly inbound. It's as if the packets can't return back out the interface from which they were sent. I've tried adding routes on the 3550 for the external subnets that have been assigned by ISP1 to the 1841 for ISP1, but this hasn't helped.
Thanks.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
No, that is NAT'ing a host inside your network to an IP address outside of your network.
What option#2 is the other way around. That is a public IP address appears to be an internal address on your network.
Something like:
ip nat pool net-10 10.0.1.0 10.0.1.255 prefix-length 24
ip nat outside source list 1 pool net-10
What this will do is all external IP addresses will appear to your internal network to be in the IP subnet 10.0.1.0/24.
Above taken from:
http://www.cisco.com/en/US/docs/ios/12_3/ipaddr/command/reference/ip1_i2g.html#wp1079462
What option#2 is the other way around. That is a public IP address appears to be an internal address on your network.
Something like:
ip nat pool net-10 10.0.1.0 10.0.1.255 prefix-length 24
ip nat outside source list 1 pool net-10
What this will do is all external IP addresses will appear to your internal network to be in the IP subnet 10.0.1.0/24.
Above taken from:
http://www.cisco.com/en/US/docs/ios/12_3/ipaddr/command/reference/ip1_i2g.html#wp1079462
Could you think of these routes the following way?
All web traffic (port 80 and 443) to be sent to ISP2 through policy based routing
Everything else to ISP1
or
Traffic to specific pool of IP addresses sent to ISP1 through static routes
Everything else to ISP2
I personally don't see how changing natting can make this work otherwise.
Also, is there a way that ISP2 can have static routes to your email's network through ISP1? That way your default gateway would stay to ISP2, ISP2 would go to ISP1 to contact servers, ISP1 back to ISP2 with response, ISP2 to your switch. This would work if ISP1, ISP2 and you are on the same subnet
All web traffic (port 80 and 443) to be sent to ISP2 through policy based routing
Everything else to ISP1
or
Traffic to specific pool of IP addresses sent to ISP1 through static routes
Everything else to ISP2
I personally don't see how changing natting can make this work otherwise.
Also, is there a way that ISP2 can have static routes to your email's network through ISP1? That way your default gateway would stay to ISP2, ISP2 would go to ISP1 to contact servers, ISP1 back to ISP2 with response, ISP2 to your switch. This would work if ISP1, ISP2 and you are on the same subnet
If you do "reverse" natting, that is NAT the public IP address to an internal private, but setup two unique pools, one for each interface, you route back the way it came in. That is you have:
| eth0 <--> 1841#1
3550 |
| eth1 <--> 1841#2
On 1841#1 you do:
ip nat pool net-88 10.0.88.0 10.0.88.255 prefix-length 24
ip nat outside source list 1 pool net-88
On 1841#2
ip nat pool net-99 10.0.99.0 10.0.99.255 prefix-length 24
ip nat outside source list 1 pool net-99
Now the 3550 will see IP addresses 10.0.88.1-254 for everything coming from ISP#1 and 10.0.99.1-254 for everything coming from ISP#2.
So the 3550 would have a route for 10.0.88.0/24 that points to 1841#1 and a route for 10.0.99.0/24 that points to 1841#2.
If 254 addresses are not enough, then you just change the pools to 10.88.0.0 10.88.255.255 prefix-length 16.
| eth0 <--> 1841#1
3550 |
| eth1 <--> 1841#2
On 1841#1 you do:
ip nat pool net-88 10.0.88.0 10.0.88.255 prefix-length 24
ip nat outside source list 1 pool net-88
On 1841#2
ip nat pool net-99 10.0.99.0 10.0.99.255 prefix-length 24
ip nat outside source list 1 pool net-99
Now the 3550 will see IP addresses 10.0.88.1-254 for everything coming from ISP#1 and 10.0.99.1-254 for everything coming from ISP#2.
So the 3550 would have a route for 10.0.88.0/24 that points to 1841#1 and a route for 10.0.99.0/24 that points to 1841#2.
If 254 addresses are not enough, then you just change the pools to 10.88.0.0 10.88.255.255 prefix-length 16.
Agree, but what happens when the traffic originates from behind the 3560? the default gateway will have to be ISP1 and nothing would have changed since ISP#2 would be unused. He would have to connect to his existing servers through the new NATed ip address, and create static nats for his servers on ISP#1 and ISP#2, update local DNS, etc.
Unless I'm missing the whole point.
Still, this sounds like an interesting scenario, I'll go ahead and test this out myself :)
Unless I'm missing the whole point.
Still, this sounds like an interesting scenario, I'll go ahead and test this out myself :)
ddiazp, good catch.
I either over thinking, under thinking, maybe not thinking at all. : )
The best solution is policy based routing to setup source IP address. This allows you to direct outbound traffic based on what host it is coming from not what host it is going to.
I have to think about my second option some more. At a minimum:
The reverse NAT setup would only need to be done on 1841#1 going to ISP#1. This would handle all traffic that originates from the outside (Internet) into your network and allow it to go back via ISP#1.
The default route would then point to 1841#2. This would handle all "web surfing" traffic.
The last thing is to figure out how to handle traffic that originates inside that must go out via ISP#1. Either polciy based routing for source IP address or have static NAT's in both 1841's for both sets of IP addresses. Doing static NAT's for addresses in ISP2's range would allow traffic originating internally to flow outbound via ISP2. But would cause outbound e-mail and DNS queries to flow over ISP2. Now this could free up bandwidth on ISP1 and could improve the performance of applications where the request originate from the Internet.
I either over thinking, under thinking, maybe not thinking at all. : )
The best solution is policy based routing to setup source IP address. This allows you to direct outbound traffic based on what host it is coming from not what host it is going to.
I have to think about my second option some more. At a minimum:
The reverse NAT setup would only need to be done on 1841#1 going to ISP#1. This would handle all traffic that originates from the outside (Internet) into your network and allow it to go back via ISP#1.
The default route would then point to 1841#2. This would handle all "web surfing" traffic.
The last thing is to figure out how to handle traffic that originates inside that must go out via ISP#1. Either polciy based routing for source IP address or have static NAT's in both 1841's for both sets of IP addresses. Doing static NAT's for addresses in ISP2's range would allow traffic originating internally to flow outbound via ISP2. But would cause outbound e-mail and DNS queries to flow over ISP2. Now this could free up bandwidth on ISP1 and could improve the performance of applications where the request originate from the Internet.
I'd still suggest the easiest way to do this in my opinion which is policy based routing based on destination IP instead:
create ACL specifying destination IPs to servers on ISP#1 (email, DNS, whatever) on the switch
apply the ACL to a policy-map type inspect statement
enter ip next hop for ISP#1
apply policy on inside interface on layer3 switch
leave default gateway on ISP#2
done
when you say "less reliable" how reliable is it? is there an SLA on this link? NATing is great but i'd steer away from it in this scenario.
Also, remember these ISP routers may not be accessible to us
create ACL specifying destination IPs to servers on ISP#1 (email, DNS, whatever) on the switch
apply the ACL to a policy-map type inspect statement
enter ip next hop for ISP#1
apply policy on inside interface on layer3 switch
leave default gateway on ISP#2
done
when you say "less reliable" how reliable is it? is there an SLA on this link? NATing is great but i'd steer away from it in this scenario.
Also, remember these ISP routers may not be accessible to us
ASKER
Thanks for you help ddizap and giltjr. It looks like a policy map is going to be my course of action. Here are a couple of more details and answers to questions:
ISP1 is a partial T1 (shared with voice). ISP2 is a cable internet provider.
With the policy mapping setup and the default gateway set to ISP2 (I do have access to both routers, btw), it would appear that this configuration would not allow for failover to ISP1 if/when ISP2 goes down. Is this correct?
ISP1 is a partial T1 (shared with voice). ISP2 is a cable internet provider.
With the policy mapping setup and the default gateway set to ISP2 (I do have access to both routers, btw), it would appear that this configuration would not allow for failover to ISP1 if/when ISP2 goes down. Is this correct?
Actually you can have the router setup to fail over to ISP1 if ISP2 goes down.
You can use the SLA feature (I think the 3550 supports this).
ip sla 10
icmp-echo x.x.x.x
timeout 1000
frequency 3
ip sla schedule 10 life forever start-time now
ip route 0.0.0.0 0.0.0.0 10.0.0.1 track 10 (this should be your normal default route)
ip route 0.0.0.0 0.0.0.0 192.168.0.1 20 (this should be the route want to take take if the normal default fails).
The address x.x.x.x should be the 1st IP address outside your building on ISP2's connection.
You can use the SLA feature (I think the 3550 supports this).
ip sla 10
icmp-echo x.x.x.x
timeout 1000
frequency 3
ip sla schedule 10 life forever start-time now
ip route 0.0.0.0 0.0.0.0 10.0.0.1 track 10 (this should be your normal default route)
ip route 0.0.0.0 0.0.0.0 192.168.0.1 20 (this should be the route want to take take if the normal default fails).
The address x.x.x.x should be the 1st IP address outside your building on ISP2's connection.
ASKER
Thanks for all of you help on this, guys. I am going to begin reconfiguring the network tonight using your recommendations. I'll let you know how it goes.
ASKER
I spoke too soon. The 3550 is running SMI, and it looks like PBR requires EMI. Back to the drawing board...
Yes, PBR requires EMI.
ASKER
Hmmm, now I've really confused myself. I'm routing with a 3550-12T, which appears to have the route-map commands, so it looks like I'm good to go. Back to the plan to deploy PBR.
12T's came with EMI standard. No I did not just know this, I found it at:
http://www.cisco.com/en/US/products/sw/iosswrel/ps1833/prod_bulletin09186a00800ab81f.html
http://www.cisco.com/en/US/products/sw/iosswrel/ps1833/prod_bulletin09186a00800ab81f.html
ASKER
Ok, here is the plan. Current config as follows:
-----start snip-----
PCCA_North_MC_C3550-12T#sh
Building configuration...
Current configuration : 5537 bytes
!
! Last configuration change at 14:58:29 EDT Tue Sep 28 2010
! NVRAM config last updated at 15:03:01 EDT Tue Sep 14 2010
!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname PCCA_North_MC_C3550-12T
!
enable secret <removed>
!
username provision privilege 15 password <removed>
no aaa new-model
clock timezone EST -5
clock summer-time EDT recurring
ip subnet-zero
ip routing
ip name-server 216.54.161.55
ip name-server 216.54.161.56
ip dhcp excluded-address 10.9.0.1 10.9.2.0
ip dhcp excluded-address 10.9.2.255 10.9.255.254
ip dhcp excluded-address 10.2.1.1 10.2.1.99
!
ip dhcp pool VOICE
import all
network 10.9.0.0 255.255.0.0
domain-name pcca.local
dns-server 10.1.1.1
option 150 ip 10.9.1.10
default-router 10.9.1.1
lease 14
!
ip dhcp pool SERVERS
import all
network 10.2.1.0 255.255.255.0
domain-name pcca-local
dns-server 216.54.161.55 216.54.161.56
default-router 10.2.1.1
lease 8
!
!
!
<crypto commands removed>
!
!
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
!
!
!
!
!
interface GigabitEthernet0/1
description ISCSI SAN PORTS
switchport access vlan 22
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/2
description ISCSI SAN PORTS
switchport access vlan 23
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/3
description ISCSI SAN PORTS
switchport access vlan 22
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/4
description ISCSI SAN PORTS
switchport access vlan 23
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/5
description ISCSI SAN PORTS
switchport access vlan 22
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/6
description ISCSI SAN PORTS
switchport access vlan 23
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/7
description ISCSI SAN PORTS
switchport access vlan 22
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/8
description ISCSI SAN PORTS
switchport access vlan 23
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/9
description LINK TO SERVER HOST2
switchport access vlan 20
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/10
description LINK TO SERVER HOST1
switchport access vlan 20
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/11
description LINK TO SOUTH BUILDING
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 10,51,90,250,251
switchport mode trunk
!
interface GigabitEthernet0/12
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 10,20,22,51,90,250
switchport mode trunk
!
interface Vlan1
no ip address
shutdown
!
interface Vlan10
description PCCA LAN VLAN
ip address 10.1.1.1 255.255.0.0
ip helper-address 10.2.1.24
!
interface Vlan20
description SERVER VLAN
ip address 10.2.1.1 255.255.255.0
!
interface Vlan51
description PCCA Private WLAN
ip address 10.5.1.1 255.255.255.0
ip helper-address 10.2.1.24
!
interface Vlan90
description PCCA Voice VLAN
ip address 10.9.1.1 255.255.0.0
!
interface Vlan250
description Bogan Paging System
ip address 10.254.10.1 255.255.255.0
ip pim sparse-dense-mode
!
interface Vlan251
description Access Control System
ip address 10.254.11.1 255.255.255.0
!
ip classless
ip route 0.0.0.0 0.0.0.0 10.1.1.2
ip http server
ip http authentication local
ip http secure-server
!
!
!
control-plane
!
!
line con 0
password <removed>
login
line vty 0 4
password <removed>
login
line vty 5 15
password <removed>
login
!
ntp clock-period 17180481
ntp server 192.43.244.18
end
-----end snip-----
Here are the commands I intend to implement this evening:
-----start snip-----
(config)# sdm prefer routing
# wr mem
# reload
(config)# access-list 10 permit any any
(config)# route-map ISP1 permit 10
(config-route-map)# match ip address 10
(config-route-map)# set ip next-hop 10.1.1.2
(config)# access-list 10 permit any any
(config)# route-map ISP2 permit 10
(config-route-map)# match ip address 10
(config-route-map)# set ip next-hop 10.1.1.3
-----end snip-----
I gathered this info by reading through the document that was suggested earlier in this thread. Am I processing this correctly? Missing anything?
Thanks so much for your help.
-----start snip-----
PCCA_North_MC_C3550-12T#sh
Building configuration...
Current configuration : 5537 bytes
!
! Last configuration change at 14:58:29 EDT Tue Sep 28 2010
! NVRAM config last updated at 15:03:01 EDT Tue Sep 14 2010
!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname PCCA_North_MC_C3550-12T
!
enable secret <removed>
!
username provision privilege 15 password <removed>
no aaa new-model
clock timezone EST -5
clock summer-time EDT recurring
ip subnet-zero
ip routing
ip name-server 216.54.161.55
ip name-server 216.54.161.56
ip dhcp excluded-address 10.9.0.1 10.9.2.0
ip dhcp excluded-address 10.9.2.255 10.9.255.254
ip dhcp excluded-address 10.2.1.1 10.2.1.99
!
ip dhcp pool VOICE
import all
network 10.9.0.0 255.255.0.0
domain-name pcca.local
dns-server 10.1.1.1
option 150 ip 10.9.1.10
default-router 10.9.1.1
lease 14
!
ip dhcp pool SERVERS
import all
network 10.2.1.0 255.255.255.0
domain-name pcca-local
dns-server 216.54.161.55 216.54.161.56
default-router 10.2.1.1
lease 8
!
!
!
<crypto commands removed>
!
!
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
!
!
!
!
!
interface GigabitEthernet0/1
description ISCSI SAN PORTS
switchport access vlan 22
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/2
description ISCSI SAN PORTS
switchport access vlan 23
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/3
description ISCSI SAN PORTS
switchport access vlan 22
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/4
description ISCSI SAN PORTS
switchport access vlan 23
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/5
description ISCSI SAN PORTS
switchport access vlan 22
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/6
description ISCSI SAN PORTS
switchport access vlan 23
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/7
description ISCSI SAN PORTS
switchport access vlan 22
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/8
description ISCSI SAN PORTS
switchport access vlan 23
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/9
description LINK TO SERVER HOST2
switchport access vlan 20
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/10
description LINK TO SERVER HOST1
switchport access vlan 20
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/11
description LINK TO SOUTH BUILDING
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 10,51,90,250,251
switchport mode trunk
!
interface GigabitEthernet0/12
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 10,20,22,51,90,250
switchport mode trunk
!
interface Vlan1
no ip address
shutdown
!
interface Vlan10
description PCCA LAN VLAN
ip address 10.1.1.1 255.255.0.0
ip helper-address 10.2.1.24
!
interface Vlan20
description SERVER VLAN
ip address 10.2.1.1 255.255.255.0
!
interface Vlan51
description PCCA Private WLAN
ip address 10.5.1.1 255.255.255.0
ip helper-address 10.2.1.24
!
interface Vlan90
description PCCA Voice VLAN
ip address 10.9.1.1 255.255.0.0
!
interface Vlan250
description Bogan Paging System
ip address 10.254.10.1 255.255.255.0
ip pim sparse-dense-mode
!
interface Vlan251
description Access Control System
ip address 10.254.11.1 255.255.255.0
!
ip classless
ip route 0.0.0.0 0.0.0.0 10.1.1.2
ip http server
ip http authentication local
ip http secure-server
!
!
!
control-plane
!
!
line con 0
password <removed>
login
line vty 0 4
password <removed>
login
line vty 5 15
password <removed>
login
!
ntp clock-period 17180481
ntp server 192.43.244.18
end
-----end snip-----
Here are the commands I intend to implement this evening:
-----start snip-----
(config)# sdm prefer routing
# wr mem
# reload
(config)# access-list 10 permit any any
(config)# route-map ISP1 permit 10
(config-route-map)# match ip address 10
(config-route-map)# set ip next-hop 10.1.1.2
(config)# access-list 10 permit any any
(config)# route-map ISP2 permit 10
(config-route-map)# match ip address 10
(config-route-map)# set ip next-hop 10.1.1.3
-----end snip-----
I gathered this info by reading through the document that was suggested earlier in this thread. Am I processing this correctly? Missing anything?
Thanks so much for your help.
ASKER
Opps. The second access-list command set should have been access-list 11 for ISP2. So:
-----start snip-----
(config)# access-list 10 permit any any
(config)# route-map ISP1 permit 10
(config-route-map)# match ip address 10
(config-route-map)# set ip next-hop 10.1.1.2
(config)# access-list 11 permit any any
(config)# route-map ISP2 permit 11
(config-route-map)# match ip address 11
(config-route-map)# set ip next-hop 10.1.1.3
-----end snip-----
Am I on the right track?
-----start snip-----
(config)# access-list 10 permit any any
(config)# route-map ISP1 permit 10
(config-route-map)# match ip address 10
(config-route-map)# set ip next-hop 10.1.1.2
(config)# access-list 11 permit any any
(config)# route-map ISP2 permit 11
(config-route-map)# match ip address 11
(config-route-map)# set ip next-hop 10.1.1.3
-----end snip-----
Am I on the right track?
ASKER
Ok guys, we're almost there. PBR is working. I'll post my final config shortly. The last remaining issue is the IP SLA commands. The issue I'm running in to is that once it fails over to the secondary route, when the primary route comes back online, it doesn't switch back over to the primary route.
Here is my ip sla config:
-----start snip-----
ip sla 10
icmp-echo 71.43.74.225
timeout 1000
frequency 3
ip sla schedule 10 life forever start-time now
ip route 0.0.0.0 0.0.0.0 10.1.1.3 track 10
ip route 0.0.0.0 0.0.0.0 10.1.1.2 2
-----end snip-----
Thoughts?
Here is my ip sla config:
-----start snip-----
ip sla 10
icmp-echo 71.43.74.225
timeout 1000
frequency 3
ip sla schedule 10 life forever start-time now
ip route 0.0.0.0 0.0.0.0 10.1.1.3 track 10
ip route 0.0.0.0 0.0.0.0 10.1.1.2 2
-----end snip-----
Thoughts?
Thanks for the points.
As to the route fail over and recovery, you may want to read:
http://www.ciscoblog.com/archives/2008/08/dynamic_failove.html
I think after the ip slao schedule ... you need to issue:
track 10 rtr 1 reachability
You also need to make sure that if you defined your default route to 10.1.1.3 PRIOR to setting this up you that you need to delete it and then re-define with the track 10 on the end.
As to the route fail over and recovery, you may want to read:
http://www.ciscoblog.com/archives/2008/08/dynamic_failove.html
I think after the ip slao schedule ... you need to issue:
track 10 rtr 1 reachability
You also need to make sure that if you defined your default route to 10.1.1.3 PRIOR to setting this up you that you need to delete it and then re-define with the track 10 on the end.
ASKER
-----start snip-----
ip nat inside source list 1 interface FastEthernet0/0 overload
ip nat inside source static 10.2.1.22 1.1.1.1
ip nat inside source static 10.2.1.23 1.1.1.2
-----end snip-----
1.2.1.22 is an email server. When the default route of the 3550 points to 10.1.1.2 (internal interface of the ISP1 1841) then 10.2.1.22 can be contacted from the outside via 1.1.1.1. However, if the default route of the 3550 is changed to 10.1.1.3 (internal interface of ISP2 1841), inbound traffic to 10.2.1.22 no longer works.
From your comment above, it looks like I am missing a route on the 3550. What would that route be?