cisco router secondary ip and nat problem

Cisco 1841 router 12.4(3i)

I have 2 disconnected public IP spaces - both from the same ISP.  I am trying to NAT an IP from the second IP space to a sub-interface on the inside.

I have a primary and seconday IP on the outside interface and several sub-interfaces on the inside - each on a different vlan with isolated servers/services on each vlan.

I can't get a NAT to pass anything *except* ICMP on the IP in the second block.  

That is to say, my access-list is tripped with a ping 66.111.111.107 and the ping replies.  I see the icmp on the inside host (wireshark), so I know it's not just the router replying.  however, the access-list for 66.111.111.107 is *not* hit when I try *any* other port - http, https, ssh, rdp..etc.

I am not mixing IP's from the public blocks into the same vlan on the inside (i.e., IP's from block 1 are NAT'd to VLAN 100, 101, 102, and I need IP's from block 2 NAT'd to vlan 103).

(public ip middle octets changed - first/last are actual, internal are actual)

outside: fa0/1
ip address 209.222.222.58 255.255.255.248
ip address 66.111.111.113 255.255.255.224
ip nat outside
ip access-group in.outside in

inside: fa0/0
fa0/0.100
dot1Q 100
ip address 10.0.100.1 255.255.255.0

fa0/1.103
dot1Q 103
ip address 10.0.103.1 255.255.255.0

ip nat inside source static tcp 10.0.100.10 443 interface fa0/1 443
ip nat inside source static 10.0.103.10 66.111.111.107 ext

ip access-list ext in.outside
permit ip any host 66.111.111.107
permit tcp any host 209.222.222.58 eq 443
permit icmp any any
snowdog_2112Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Garry GlendownConsulting and Network/Security SpecialistCommented:
Apart from the missing "secondary" on one of the outside IP addresses, I assume you also have the "ip nat inside" command on the inside sub-interfaces? Also assuming that is fa0/0.103 above, not fa0/1.103? (usually copying part of the config instead of re-typing saves a couple typing errors)
What I'm wondering about is why the ACL doesn't show a hit for the TCP connection - maybe because the TCP handshake isn't complete? Do you see any SYN packets on attempt of connecting to the NAT IP?
0
snowdog_2112Author Commented:
yes - sorry about the typo's.  I was condensing down the relevant parts.

66.111.111.113 is the secondary IP on fa0/1, and the sub-int is on fa0/0.103 and is ip nat inside.

Let me add another variable as well: there is another router with an external interface in the same switch as the external interface as this 1841.  Router #2 is NAT'ing some of the IP's in Block #2 as well.  The ARP table on both show the IP I am trying to NAT on router 1 matches the MAC on router 1 (i.e., I don't think it's router 2 advertising the same IP).

                    [ ISP ]
                        |
             [switch (L2 mgd)]
                 |                  |
         [router 1]        [router 2]

I tried running a network-traffic profile on the outside interface, and I did see some packets from the external source IP I was testing from (I used an acl with permit ip <test ip> 66.111.111.107> in the network-traffic profile).

How do I look for SYN packets?  Would they be in the wireshark sniff I have from the network-traffic capture?

I have other NAT's that are working, and I tried the connection from several different external sources with same results.
0
Garry GlendownConsulting and Network/Security SpecialistCommented:
Yes, the SYN packets must be visible in the Wireshark capture ... you should see the typical 3-way-handshake, with SYN coming from the remote site, SYN/ACK returning there, and finally the ACK from the remote site again ... check whether the handshake fails somewhere in between ...
As for the external interface, doing the packet debug does help too ... just make sure you get the ACL right, otherwise the Cisco router might be gone in the blink of the eye ;)
Maybe you can post a sanitized capture from the router outside interface of the incoming TCP connection, with "detail" used in the debug command ... ("debug ip packet ACL detail", replace ACL with the actual ACL name/number) This should give a hint as to which packets actually arrive, plus maybe run the same on the inside interface with just a filter on the test system, to see whether the NAT works ...
0
The Ultimate Tool Kit for Technolgy Solution Provi

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy for valuable how-to assets including sample agreements, checklists, flowcharts, and more!

snowdog_2112Author Commented:
This is weird.  I put an acl "permit ip host <remote host> host <ip in 2nd block>"

Then "deb ip packet <acl> det"

Then I tried a tcp connection from <remote host>.  Nothing...no packets registered.

Here's a sanitized config (all public ip's and crypto keys changed)

version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname myrouter1
!
boot-start-marker
boot system flash c1841-advipservicesk9-mz.124-3i.bin
boot-end-marker
!
no aaa new-model
!
resource policy
!
mmi polling-interval 60
no mmi auto-configure
no mmi pvc
mmi snmp-timeout 180
ip subnet-zero
ip cef
!
!
!
!
ip domain name myrouter.com
ip name-server 8.8.8.8
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
username mycompany privilege 15 password 7 115B4E171A16521F28
archive
 log config
  hidekeys
!
!
!
crypto isakmp policy 10
 encr 3des
 hash md5
 authentication pre-share
 group 2
crypto isakmp key cYD7apkJDXw28Q address 67.55.22.66 no-xauth
crypto isakmp key cYD7apk address 68.19.19.111 no-xauth
crypto isakmp keepalive 10 periodic
!
!
crypto ipsec transform-set esp_3des_md5 esp-3des esp-md5-hmac
!
!
crypto map map_outside local-address FastEthernet0/1
crypto map map_outside 10 ipsec-isakmp
 set peer 67.55.22.66
 set security-association lifetime seconds 86400
 set transform-set esp_3des_md5
 match address 104
crypto map map_outside 20 ipsec-isakmp
 set peer 68.19.19.111
 set security-association lifetime seconds 86400
 set transform-set esp_3des_md5
 match address 105
!
!
!
!
interface Tunnel0
 description tunnel to KC
 ip address 192.168.254.2 255.255.255.252
 ip nat inside
 ip virtual-reassembly
 keepalive 5 4
 tunnel source 209.222.233.58
 tunnel destination 66.155.199.98
!
interface FastEthernet0/0
 no ip address
 speed auto
 full-duplex
 no mop enabled
!
interface FastEthernet0/0.99
 description VLAN - LabTech 99
 encapsulation dot1Q 99
 ip address 10.0.99.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly
 no snmp trap link-status
!
interface FastEthernet0/0.100
 description VLAN - myrouter exchange
 encapsulation dot1Q 100
 ip address 10.0.100.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly
 no snmp trap link-status
!
interface FastEthernet0/0.101
 description VLAN - hosted customer 101
 encapsulation dot1Q 101
 ip address 10.0.101.1 255.255.255.0
 ip access-group in.customer.101 in
 ip nat inside
 ip virtual-reassembly
 no snmp trap link-status
!
interface FastEthernet0/0.102
 description VLAN - hosted customer 102
 encapsulation dot1Q 102
 ip address 10.0.102.1 255.255.255.0
 ip access-group in.customer.102 in
 ip nat inside
 ip virtual-reassembly
 no snmp trap link-status
!
interface FastEthernet0/0.103
 description VLAN - hosted customer 103
 encapsulation dot1Q 103
 ip address 10.0.103.1 255.255.255.0
 ip nat inside
 no ip virtual-reassembly
 no snmp trap link-status
!
interface FastEthernet0/0.104
 description VLAN - hosted customer 104
 encapsulation dot1Q 104
 ip address 10.0.104.1 255.255.255.0
 ip access-group in.customer.104 in
 ip nat inside
 ip virtual-reassembly
 no snmp trap link-status
!
interface FastEthernet0/0.192
 description VLAN - myrouter
 encapsulation dot1Q 192
 ip address 192.168.252.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly
 no snmp trap link-status
!
interface FastEthernet0/1
 ip address 66.155.199.113 255.255.255.224 secondary
 ip address 209.222.233.58 255.255.255.248
 ip access-group in.outside in
 ip nat outside
 ip virtual-reassembly
 duplex auto
 speed auto
 crypto map map_outside
!
interface Serial0/0/0
 no ip address
 shutdown
!
ip classless
ip route 0.0.0.0 0.0.0.0 209.222.233.57
ip route 192.168.250.0 255.255.255.0 192.168.254.1
!
!
ip http server
no ip http secure-server
ip nat pool nat_pool_101 209.222.233.60 209.222.233.60 netmask 255.255.255.248
ip nat pool kc_hosted_web1 192.168.252.15 192.168.252.15 netmask 255.255.255.0 type rotary
ip nat inside source list 110 interface FastEthernet0/1 overload
ip nat inside source list customer_nat pool nat_pool_101 overload
ip nat inside source static tcp 10.0.100.11 443 interface FastEthernet0/1 443
ip nat inside source static tcp 10.0.100.11 25 interface FastEthernet0/1 25
ip nat inside source static tcp 10.0.100.11 80 interface FastEthernet0/1 80
ip nat inside source static tcp 192.168.252.10 3389 interface FastEthernet0/1 3389
ip nat inside source static 10.0.103.10 66.155.199.112 extendable
ip nat inside source static tcp 192.168.252.15 53 66.155.199.113 53 extendable
ip nat inside source static udp 192.168.252.15 53 66.155.199.113 53 extendable
ip nat inside source static tcp 192.168.252.15 20 209.222.233.59 20 extendable
ip nat inside source static tcp 192.168.252.15 21 209.222.233.59 21 extendable
ip nat inside source static tcp 192.168.252.15 80 209.222.233.59 80 extendable
ip nat inside source static tcp 192.168.252.15 443 209.222.233.59 443 extendable
ip nat inside source static tcp 192.168.252.100 70 209.222.233.60 70 extendable
ip nat inside source static tcp 192.168.252.100 80 209.222.233.60 80 extendable
ip nat inside source static tcp 192.168.252.100 3306 209.222.233.60 3306 extendable
ip nat inside source static tcp 10.0.101.11 3389 209.222.233.60 3390 extendable
ip nat inside source static tcp 10.0.102.10 3389 209.222.233.60 3391 extendable
ip nat inside source static 10.0.99.10 209.222.233.61 extendable
ip nat inside destination list 115 pool kc_hosted_web1
!
ip access-list extended customer_nat
 permit ip 10.0.101.0 0.0.0.255 any
 permit ip 10.0.102.0 0.0.0.255 any
 deny   ip 10.0.103.0 0.0.0.255 any
 deny   ip 10.0.104.0 0.0.0.255 192.168.0.0 0.0.0.255
 deny   ip 10.0.104.0 0.0.0.255 192.168.1.0 0.0.0.255
 permit ip 10.0.104.0 0.0.0.255 any
 permit ip 10.0.105.0 0.0.0.255 any
 permit ip 10.0.106.0 0.0.0.255 any
 permit ip 10.0.107.0 0.0.0.255 any
 permit ip 10.0.108.0 0.0.0.255 any
 permit ip 10.0.109.0 0.0.0.255 any
 permit ip 10.0.110.0 0.0.0.255 any
ip access-list extended in.customer.101
 permit icmp 10.0.101.0 0.0.0.255 192.168.252.0 0.0.0.255
 permit icmp 10.0.101.0 0.0.0.255 10.0.100.0 0.0.0.255
 permit tcp 10.0.101.0 0.0.0.255 10.0.100.0 0.0.0.255 eq smtp
 permit tcp 10.0.101.0 0.0.0.255 10.0.100.0 0.0.0.255 eq 443
 permit tcp 10.0.101.0 0.0.0.255 192.168.252.0 0.0.0.255 eq smtp
 permit tcp 10.0.101.0 0.0.0.255 192.168.252.0 0.0.0.255 eq 443
 permit tcp 10.0.101.0 0.0.0.255 any eq domain
 permit udp 10.0.101.0 0.0.0.255 any eq domain
 deny   ip 10.0.101.0 0.0.0.255 10.0.100.0 0.0.0.255
 deny   ip 10.0.101.0 0.0.0.255 192.168.252.0 0.0.0.255
 permit ip 10.0.101.0 0.0.0.255 any
ip access-list extended in.customer.102
 permit icmp 10.0.102.0 0.0.0.255 192.168.252.0 0.0.0.255
 permit icmp 10.0.102.0 0.0.0.255 10.0.100.0 0.0.0.255
 permit tcp 10.0.101.0 0.0.0.255 10.0.100.0 0.0.0.255 eq smtp
 permit tcp 10.0.101.0 0.0.0.255 10.0.100.0 0.0.0.255 eq 443
 permit tcp 10.0.102.0 0.0.0.255 10.0.100.0 0.0.0.255 eq smtp
 permit tcp 10.0.102.0 0.0.0.255 10.0.100.0 0.0.0.255 eq 443
 permit tcp 10.0.102.0 0.0.0.255 any eq domain
 permit udp 10.0.102.0 0.0.0.255 any eq domain
 deny   ip 10.0.102.0 0.0.0.255 10.0.100.0 0.0.0.255
 deny   ip 10.0.102.0 0.0.0.255 192.168.252.0 0.0.0.255
 permit ip 10.0.102.0 0.0.0.255 any
ip access-list extended in.customer.103
 permit icmp 10.0.103.0 0.0.0.255 10.0.103.0 0.0.0.255
 permit icmp 10.0.103.0 0.0.0.255 10.0.100.0 0.0.0.255
 deny   icmp 10.0.103.0 0.0.0.255 10.0.0.0 0.0.255.255
 permit icmp any any
 permit tcp 10.0.103.0 0.0.0.255 10.0.100.0 0.0.0.255 eq smtp
 permit tcp 10.0.103.0 0.0.0.255 10.0.100.0 0.0.0.255 eq 443
 permit tcp 10.0.103.0 0.0.0.255 any eq domain
 permit udp 10.0.103.0 0.0.0.255 any eq domain
 permit ip 10.0.103.0 0.0.0.255 10.0.103.0 0.0.0.255
 deny   ip 10.0.103.0 0.0.0.255 10.0.0.0 0.0.255.255
 deny   ip 10.0.103.0 0.0.0.255 192.168.0.0 0.0.255.255
 permit ip 10.0.103.0 0.0.0.255 any
ip access-list extended in.customer.104
 permit icmp 10.0.104.0 0.0.0.255 10.0.104.0 0.0.0.255
 deny   icmp 10.0.104.0 0.0.0.255 10.0.0.0 0.0.255.255
 permit icmp any any
 permit udp 10.0.104.0 0.0.0.255 any eq domain
 permit ip 10.0.104.0 0.0.0.255 10.0.104.0 0.0.0.255
 deny   ip 10.0.104.0 0.0.0.255 10.0.0.0 0.0.255.255
 permit ip 10.0.104.0 0.0.0.255 192.168.1.0 0.0.0.255
 permit ip 10.0.104.0 0.0.0.255 192.168.0.0 0.0.0.255
 deny   ip 10.0.104.0 0.0.0.255 192.168.0.0 0.0.255.255
 permit ip 10.0.104.0 0.0.0.255 any
ip access-list extended in.outside
 permit ip any host 66.155.199.112
 permit tcp any any eq smtp
 permit tcp any any eq 443
 permit tcp any host 209.222.233.60 range 3389 3900
 permit tcp any any eq www
 permit tcp any any established
 permit udp any eq domain any
 permit tcp any host 209.222.233.60 eq www
 permit tcp any host 209.222.233.60 eq gopher
 permit tcp any host 209.222.233.60 eq 3306
 permit udp any any eq ntp
 permit icmp any any
 permit tcp any any eq domain
 permit tcp any eq domain any
 permit udp any any eq domain
 permit tcp any any eq 1064
 permit tcp any host 209.222.233.61 eq www
 permit tcp any host 209.222.233.61 eq 443
 permit tcp any host 209.222.233.61 eq 3306
 permit tcp any host 209.222.233.61 range gopher 75
 permit udp any host 209.222.233.61 range 70 75
 permit tcp any host 209.222.233.61 eq 5500
 permit tcp any host 209.222.233.61 eq 5901
 permit tcp any host 209.222.233.61 range 8000 8999
 permit udp any host 209.222.233.61 range 8000 8999
 permit tcp any host 209.222.233.61 range 40000 41000
 permit udp any host 209.222.233.61 range 40000 41000
 permit tcp 66.155.199.96 0.0.0.31 any eq 3389
 permit tcp any host 209.222.233.59 eq ftp
 permit tcp any host 209.222.233.59 eq ftp-data
 permit tcp any host 209.222.233.59 range 2048 2080
 permit tcp any host 209.222.233.58 eq 22
 permit ip 66.155.199.96 0.0.0.31 host 209.222.233.58
 permit gre any any
 permit tcp any any eq 500
 permit udp any any eq non500-isakmp
 permit udp any any eq isakmp
 permit esp any any
 permit tcp any host 209.222.233.61 eq 3389
!
access-list 1 permit 192.168.252.0 0.0.0.255
access-list 5 permit 10.0.101.0 0.0.0.255
access-list 5 permit 10.0.102.0 0.0.0.255
access-list 5 permit 10.0.104.0 0.0.0.255
access-list 104 permit ip 10.0.104.0 0.0.0.255 192.168.1.0 0.0.0.255
access-list 105 permit ip 10.0.104.0 0.0.0.255 192.168.0.0 0.0.0.255
access-list 110 deny   ip 10.0.100.0 0.0.0.255 192.168.250.0 0.0.0.255
access-list 110 permit ip 10.0.100.0 0.0.0.255 any
access-list 115 permit tcp any host 209.222.233.59 range 2048 2080
access-list 199 permit ip host 67.63.234.203 host 66.155.199.112
!
!
!
!
control-plane
!
!
!
!
!
!
!
!
!
line con 0
 session-timeout 120
 exec-timeout 60 0
 logging synchronous
 login local
line aux 0
line vty 0 4
 session-timeout 120
 exec-timeout 60 0
 logging synchronous
 login local
!
scheduler allocate 20000 1000
end
0
Garry GlendownConsulting and Network/Security SpecialistCommented:
Doing a ping from the remote host did list packets with the debug, though?
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
snowdog_2112Author Commented:
Thanks again for the help - I think I'm getting closer to a solution.

I get *no* detail in the "deb ip packet"!  I assume I'm applying it wrong...

I'm testing ping and RDP from a remote host (different ISP, different city)

ip access-list ext 199
permit ip 67.66.22.0 0.0.0.255 any

ip debug packet 199 detail

From remote host: ping 66.155.199.112, and RDP to 66.155.199.112
The ping replies, RDP fails.

Here's the completely weird thing.  I have wireshark running on the internal host I'm NAT'ing to - 10.0.103.10.  When I RDP to 66.155.199.112, it times out and I get nothing in wireshark.  When I "Cancel" the error on the remote host, *then* I get some packets in Wireshark on the internal host!!!

So...it seems *something* is coming through.

This is just odd...
0
snowdog_2112Author Commented:
Still looking for assistance on this...
0
snowdog_2112Author Commented:
still searching...I have a workaround in place (using another router), but will need a solution at some point...
0
snowdog_2112Author Commented:
no love on this...
0
snowdog_2112Author Commented:
no solution found...have an alternate configuration, but will have to face this at some point...
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Routers

From novice to tech pro — start learning today.