cyberfinancials
asked on
Cisco 1841 dropping NAT translations when internal host tries to connect to non-existing external IP
We have a SDSL internet connection connected via a Cisco 1841 router running IOS 12.4(17) provided by our ISP.
We have two separate public IP address ranges 61.129.120.88/29 and 61.129.65.224/30. We are mapping these IPs into our LAN 192.168.0.0/24 by NAT to make some inside servers (SMTP, POP3, Web, VPN dialin etc.) available over the Internet and internal hosts are allowed to gain access to the Internet.
For some time we experience the following phenomenon:
When an internal host, which is statically natted, tries to connect to an invalid IP (e.g. port 25/tcp), the router drops the static NAT for that host so that it is offline (i.e. no access to or from Internet any more, not even NS lookups etc.). If I clear the NAT table entering "clear ip nat trans *" on the Cisco, it is immediately online again. A symptom I noticed is that when this happens (before clearing the NAT table), a "show run" shows the configuration without the respective "ip nat inside..." entry for the host concerned. But the configuration entry is still there because after clearing the NAT table it is back again, it just becomes kind of ineffective.
Background info: This especially happens to our mail server running at internal IP 192.168.0.28, natted by "ip nat inside source static 192.168.0.28 61.129.120.90 extendable" in the case when it tries to deliver mail to an e-mail-adress whose MX points to an non-existing IP. Unfortunately this can not be avoided 100% because sometimes there are out-of-office replys or message-bounced-replys to spam, so the mail server goes offline every few days.
But it has nothing to do with the mail server itself. I found out that I can trigger the problem from any internal server (with different Linux kernel versions) immediately by connecting via telnet command line:
[root@mailserver]# telnet 128.138.140.100 25
Trying 128.138.140.100...
telnet: connect to address 128.138.140.100: No route to host
telnet: Unable to connect to remote host: No route to host
where 128.138.140.100 is one of those invalid IPs I found in the SMTP's log files. After the connection attempt the server immediately goes offline. These hosts are not existing or at least they are offline. One gets a "no route to host" when traced via a "normal" Internet connection where no Cisco router is involved.
Fact is that everything worked fine for over 2 years before the problems started.
I suspect it might have something to do with the additional public IP address range but that's just a rough guess. Originally we had just one (the .88/29), later the .224/30 was added and the router re-configured by the ISP, but I cannot tell if this was the exact start of the problems or what could have changed elsewhere. It is difficult to pinpoint the start of the problems because originally we knew absolutely nothing about it, it was just a spooky phenomenon solved by unplugging the router.
The admin people at our ISP cannot solve the issue. I am not a Cisco pro, I started reading about it when the ISP failed. They also replaced the router box itself, the flash card, they lowered NAT translation timeouts and they tried a different IOS version. I strongly suspect that the reason is just a configuration issue, so I hope anyone can solve this mystery.
Here is the config (I altered digits of the external IPs for privacy reasons):
version 12.4
no service pad
service timestamps debug datetime localtime
service timestamps log datetime localtime
service password-encryption
!
hostname xxx
!
boot-start-marker
boot-end-marker
!
logging buffered 12000 debugging
enable secret level 10 5 xxx
enable secret 5 xxx
!
no aaa new-model
clock timezone GMT+1 1
clock summer-time GMT+1 recurring last Sun Mar 2:00 last Sun Oct 3:00
ip cef
!
!
!
!
ip vrf mgmt
rd 123:321
!
no ip domain lookup
ip inspect name firewall fragment maximum 256 timeout 1
ip inspect name firewall ftp
ip inspect name firewall tftp
ip inspect name firewall tcp
ip inspect name firewall udp
ip inspect name firewall echo
ip inspect name firewall pptp
!
!
!
username xxx password 7 xxx
username xxx privilege 15 password 7 xxx
username xxx privilege 15 password 7 xxx
!
!
!
!
!
!
interface Loopback99
ip vrf forwarding mgmt
ip address 61.129.120.90 255.255.255.255
!
interface Loopback127
ip vrf forwarding mgmt
ip address 172.31.57.25 255.255.255.255 secondary
ip address 172.31.57.24 255.255.255.255
!
interface FastEthernet0/0
description +++ Clients LAN +++ 01-00-144201-1015 IP Range: 61.129.120.88/29
ip address 192.168.0.1 255.255.255.0
no ip unreachables
no ip proxy-arp
ip nat inside
no ip virtual-reassembly
load-interval 30
duplex auto
speed auto
!
interface FastEthernet0/1
description +++ Clients LAN +++
no ip address
shutdown
duplex auto
speed auto
!
interface ATM0/0/0
description Data Interface for Upstream
no ip address
load-interval 30
no atm ilmi-keepalive
dsl equipment-type CPE
dsl operating-mode GSHDSL symmetric annex A
dsl linerate AUTO
!
interface ATM0/0/0.35 point-to-point
description [Internet-Direkt:200516269
bandwidth 2048
no snmp trap link-status
pvc 0/35
vbr-nrt 2048 2047 100
oam-pvc manage 5
oam ais-rdi 3 3
encapsulation aal5snap
protocol ppp Virtual-Template1
!
!
interface ATM0/0/0.127 point-to-point
description *** Management 4 ISP ***
ip vrf forwarding mgmt
ip unnumbered Loopback127
no snmp trap link-status
pvc 0/127
encapsulation aal5snap
!
!
interface ATM0/1/0
description Data Interface for Upstream
no ip address
load-interval 30
no atm ilmi-keepalive
dsl equipment-type CPE
dsl operating-mode GSHDSL symmetric annex A
dsl linerate AUTO
!
interface ATM0/1/0.35 point-to-point
description [Internet-Direkt:200516269
bandwidth 2048
no snmp trap link-status
pvc 0/35
vbr-nrt 2048 2047 100
oam-pvc manage 5
oam ais-rdi 3 3
encapsulation aal5snap
protocol ppp Virtual-Template1
!
!
interface ATM0/1/0.127 point-to-point
description *** Management 4 Tele2 ***
ip vrf forwarding mgmt
ip unnumbered Loopback127
no snmp trap link-status
pvc 0/127
encapsulation aal5snap
!
!
interface Virtual-Template1
ip address 61.129.65.226 255.255.255.252
ip access-group 102 in
no ip unreachables
ip nat outside
ip inspect firewall out
ip virtual-reassembly
ppp multilink
!
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 61.129.65.225
ip route vrf mgmt 172.31.58.254 255.255.255.255 ATM0/0/0.127
ip route vrf mgmt 172.31.58.254 255.255.255.255 ATM0/1/0.127 200
!
!
ip http server
no ip http secure-server
ip nat inside source list 98 interface Virtual-Template1 overload
ip nat inside source static 192.168.0.104 61.129.120.89 extendable
ip nat inside source static 192.168.0.28 61.129.120.90 extendable
ip nat inside source static 192.168.0.100 61.129.120.91 extendable
ip nat inside source static 192.168.0.205 61.129.120.92 extendable
ip nat inside source static tcp 192.168.0.102 80 61.129.120.93 80 extendable
ip nat inside source static tcp 192.168.0.102 143 61.129.120.93 143 extendable
ip nat inside source static tcp 192.168.0.102 443 61.129.120.93 443 extendable
ip nat inside source static tcp 192.168.0.41 5900 61.129.120.93 5900 extendable
ip nat inside source static 192.168.0.106 61.129.120.94 extendable
!
access-list 45 permit any
access-list 98 permit 192.168.0.0 0.0.0.255
access-list 99 permit 195.70.224.0 0.0.0.255
access-list 99 permit 61.129.65.224 0.0.0.3
access-list 99 permit 61.129.65.220 0.0.0.3
access-list 99 permit 172.31.0.0 0.0.255.255
access-list 99 permit 192.168.0.0 0.0.0.255
access-list 102 permit tcp 195.70.224.0 0.0.0.255 host 61.129.65.226 eq telnet
access-list 102 permit tcp 61.129.65.224 0.0.0.3 host 61.129.65.226 eq telnet
access-list 102 permit icmp any 61.129.120.88 0.0.0.7 echo-reply
access-list 102 permit icmp any 61.129.120.88 0.0.0.7 administratively-prohibite
access-list 102 permit icmp any 61.129.120.88 0.0.0.7 packet-too-big
access-list 102 permit icmp any 61.129.120.88 0.0.0.7 echo
access-list 102 permit icmp any 61.129.120.88 0.0.0.7 time-exceeded
access-list 102 permit tcp any host 61.129.120.90 eq www
access-list 102 permit tcp any host 61.129.120.90 eq ftp
access-list 102 permit tcp any host 61.129.120.90 eq smtp
access-list 102 permit tcp any host 61.129.120.90 eq pop3
access-list 102 permit tcp any host 61.129.120.90 range 60000 65535
access-list 102 permit tcp any host 61.129.120.91 eq www
access-list 102 permit tcp any host 61.129.120.91 eq ftp
access-list 102 permit tcp any host 61.129.120.91 eq 116
access-list 102 permit tcp any host 61.129.120.91 range 60000 65535
access-list 102 permit tcp any host 61.129.120.92 eq www
access-list 102 permit tcp any host 61.129.120.92 eq 8080
access-list 102 permit tcp any host 61.129.120.92 eq 443
access-list 102 permit tcp any host 61.129.120.92 eq ftp
access-list 102 permit tcp any host 61.129.120.92 range 60000 65535
access-list 102 permit tcp any host 61.129.120.94 eq www
access-list 102 permit tcp any host 61.129.120.94 eq 8080
access-list 102 permit tcp any host 61.129.120.94 eq 443
access-list 102 permit tcp any host 61.129.120.94 eq ftp
access-list 102 permit tcp any host 61.129.120.94 range 60000 65535
access-list 102 permit tcp any host 61.129.120.95 eq www
access-list 102 permit tcp any host 61.129.120.90 eq 143
access-list 102 permit tcp any host 61.129.120.95 eq 4500
access-list 102 permit tcp any host 61.129.120.95 eq 500
access-list 102 permit tcp any host 61.129.120.93 eq 1723
access-list 102 permit gre any host 61.129.120.93
access-list 102 permit tcp any host 61.129.120.90 eq 443
access-list 102 permit tcp any host 61.129.120.93 eq 5900
access-list 102 permit tcp any host 61.129.120.91 eq 443
access-list 102 permit tcp any host 61.129.120.94 eq smtp
access-list 102 permit tcp any host 61.129.120.94 eq smtp
access-list 102 permit tcp any host 61.129.120.93 eq 143
access-list 102 permit tcp any host 61.129.120.93 eq 443
access-list 102 permit tcp any host 61.129.120.93 eq www
access-list 102 permit tcp any host 61.129.120.93 eq www
access-list 102 permit tcp any host 61.129.120.89 eq 443
access-list 102 permit tcp any host 61.129.120.92 eq 81
access-list 102 permit icmp any any echo
access-list 102 permit ip host 195.70.224.26 any
access-list 102 permit gre any host 61.129.120.89
access-list 102 permit icmp any 61.129.65.224 0.0.0.3 echo-reply
access-list 102 permit icmp any 61.129.65.224 0.0.0.3 administratively-prohibite
access-list 102 permit icmp any 61.129.65.224 0.0.0.3 packet-too-big
access-list 102 permit icmp any 61.129.65.224 0.0.0.3 echo
access-list 102 permit icmp any 61.129.65.224 0.0.0.3 time-exceeded
access-list 102 permit tcp any host 61.129.120.94 eq 1723
access-list 102 permit gre any host 61.129.120.94
access-list 102 permit icmp any 61.129.120.88 0.0.0.7 unreachable
access-list 102 permit icmp any 61.129.65.224 0.0.0.3 unreachable
snmp-server community xxx RO 98
!
!
!
!
control-plane
!
!
!
line con 0
exec-timeout 0 0
login local
stopbits 1
line aux 0
line vty 0 4
access-class 99 in vrf-also
exec-timeout 120 0
logging synchronous
login local
!
scheduler max-task-time 5000
scheduler allocate 20000 1000
ntp clock-period 17178236
ntp server 195.70.224.26
end
We have two separate public IP address ranges 61.129.120.88/29 and 61.129.65.224/30. We are mapping these IPs into our LAN 192.168.0.0/24 by NAT to make some inside servers (SMTP, POP3, Web, VPN dialin etc.) available over the Internet and internal hosts are allowed to gain access to the Internet.
For some time we experience the following phenomenon:
When an internal host, which is statically natted, tries to connect to an invalid IP (e.g. port 25/tcp), the router drops the static NAT for that host so that it is offline (i.e. no access to or from Internet any more, not even NS lookups etc.). If I clear the NAT table entering "clear ip nat trans *" on the Cisco, it is immediately online again. A symptom I noticed is that when this happens (before clearing the NAT table), a "show run" shows the configuration without the respective "ip nat inside..." entry for the host concerned. But the configuration entry is still there because after clearing the NAT table it is back again, it just becomes kind of ineffective.
Background info: This especially happens to our mail server running at internal IP 192.168.0.28, natted by "ip nat inside source static 192.168.0.28 61.129.120.90 extendable" in the case when it tries to deliver mail to an e-mail-adress whose MX points to an non-existing IP. Unfortunately this can not be avoided 100% because sometimes there are out-of-office replys or message-bounced-replys to spam, so the mail server goes offline every few days.
But it has nothing to do with the mail server itself. I found out that I can trigger the problem from any internal server (with different Linux kernel versions) immediately by connecting via telnet command line:
[root@mailserver]# telnet 128.138.140.100 25
Trying 128.138.140.100...
telnet: connect to address 128.138.140.100: No route to host
telnet: Unable to connect to remote host: No route to host
where 128.138.140.100 is one of those invalid IPs I found in the SMTP's log files. After the connection attempt the server immediately goes offline. These hosts are not existing or at least they are offline. One gets a "no route to host" when traced via a "normal" Internet connection where no Cisco router is involved.
Fact is that everything worked fine for over 2 years before the problems started.
I suspect it might have something to do with the additional public IP address range but that's just a rough guess. Originally we had just one (the .88/29), later the .224/30 was added and the router re-configured by the ISP, but I cannot tell if this was the exact start of the problems or what could have changed elsewhere. It is difficult to pinpoint the start of the problems because originally we knew absolutely nothing about it, it was just a spooky phenomenon solved by unplugging the router.
The admin people at our ISP cannot solve the issue. I am not a Cisco pro, I started reading about it when the ISP failed. They also replaced the router box itself, the flash card, they lowered NAT translation timeouts and they tried a different IOS version. I strongly suspect that the reason is just a configuration issue, so I hope anyone can solve this mystery.
Here is the config (I altered digits of the external IPs for privacy reasons):
version 12.4
no service pad
service timestamps debug datetime localtime
service timestamps log datetime localtime
service password-encryption
!
hostname xxx
!
boot-start-marker
boot-end-marker
!
logging buffered 12000 debugging
enable secret level 10 5 xxx
enable secret 5 xxx
!
no aaa new-model
clock timezone GMT+1 1
clock summer-time GMT+1 recurring last Sun Mar 2:00 last Sun Oct 3:00
ip cef
!
!
!
!
ip vrf mgmt
rd 123:321
!
no ip domain lookup
ip inspect name firewall fragment maximum 256 timeout 1
ip inspect name firewall ftp
ip inspect name firewall tftp
ip inspect name firewall tcp
ip inspect name firewall udp
ip inspect name firewall echo
ip inspect name firewall pptp
!
!
!
username xxx password 7 xxx
username xxx privilege 15 password 7 xxx
username xxx privilege 15 password 7 xxx
!
!
!
!
!
!
interface Loopback99
ip vrf forwarding mgmt
ip address 61.129.120.90 255.255.255.255
!
interface Loopback127
ip vrf forwarding mgmt
ip address 172.31.57.25 255.255.255.255 secondary
ip address 172.31.57.24 255.255.255.255
!
interface FastEthernet0/0
description +++ Clients LAN +++ 01-00-144201-1015 IP Range: 61.129.120.88/29
ip address 192.168.0.1 255.255.255.0
no ip unreachables
no ip proxy-arp
ip nat inside
no ip virtual-reassembly
load-interval 30
duplex auto
speed auto
!
interface FastEthernet0/1
description +++ Clients LAN +++
no ip address
shutdown
duplex auto
speed auto
!
interface ATM0/0/0
description Data Interface for Upstream
no ip address
load-interval 30
no atm ilmi-keepalive
dsl equipment-type CPE
dsl operating-mode GSHDSL symmetric annex A
dsl linerate AUTO
!
interface ATM0/0/0.35 point-to-point
description [Internet-Direkt:200516269
bandwidth 2048
no snmp trap link-status
pvc 0/35
vbr-nrt 2048 2047 100
oam-pvc manage 5
oam ais-rdi 3 3
encapsulation aal5snap
protocol ppp Virtual-Template1
!
!
interface ATM0/0/0.127 point-to-point
description *** Management 4 ISP ***
ip vrf forwarding mgmt
ip unnumbered Loopback127
no snmp trap link-status
pvc 0/127
encapsulation aal5snap
!
!
interface ATM0/1/0
description Data Interface for Upstream
no ip address
load-interval 30
no atm ilmi-keepalive
dsl equipment-type CPE
dsl operating-mode GSHDSL symmetric annex A
dsl linerate AUTO
!
interface ATM0/1/0.35 point-to-point
description [Internet-Direkt:200516269
bandwidth 2048
no snmp trap link-status
pvc 0/35
vbr-nrt 2048 2047 100
oam-pvc manage 5
oam ais-rdi 3 3
encapsulation aal5snap
protocol ppp Virtual-Template1
!
!
interface ATM0/1/0.127 point-to-point
description *** Management 4 Tele2 ***
ip vrf forwarding mgmt
ip unnumbered Loopback127
no snmp trap link-status
pvc 0/127
encapsulation aal5snap
!
!
interface Virtual-Template1
ip address 61.129.65.226 255.255.255.252
ip access-group 102 in
no ip unreachables
ip nat outside
ip inspect firewall out
ip virtual-reassembly
ppp multilink
!
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 61.129.65.225
ip route vrf mgmt 172.31.58.254 255.255.255.255 ATM0/0/0.127
ip route vrf mgmt 172.31.58.254 255.255.255.255 ATM0/1/0.127 200
!
!
ip http server
no ip http secure-server
ip nat inside source list 98 interface Virtual-Template1 overload
ip nat inside source static 192.168.0.104 61.129.120.89 extendable
ip nat inside source static 192.168.0.28 61.129.120.90 extendable
ip nat inside source static 192.168.0.100 61.129.120.91 extendable
ip nat inside source static 192.168.0.205 61.129.120.92 extendable
ip nat inside source static tcp 192.168.0.102 80 61.129.120.93 80 extendable
ip nat inside source static tcp 192.168.0.102 143 61.129.120.93 143 extendable
ip nat inside source static tcp 192.168.0.102 443 61.129.120.93 443 extendable
ip nat inside source static tcp 192.168.0.41 5900 61.129.120.93 5900 extendable
ip nat inside source static 192.168.0.106 61.129.120.94 extendable
!
access-list 45 permit any
access-list 98 permit 192.168.0.0 0.0.0.255
access-list 99 permit 195.70.224.0 0.0.0.255
access-list 99 permit 61.129.65.224 0.0.0.3
access-list 99 permit 61.129.65.220 0.0.0.3
access-list 99 permit 172.31.0.0 0.0.255.255
access-list 99 permit 192.168.0.0 0.0.0.255
access-list 102 permit tcp 195.70.224.0 0.0.0.255 host 61.129.65.226 eq telnet
access-list 102 permit tcp 61.129.65.224 0.0.0.3 host 61.129.65.226 eq telnet
access-list 102 permit icmp any 61.129.120.88 0.0.0.7 echo-reply
access-list 102 permit icmp any 61.129.120.88 0.0.0.7 administratively-prohibite
access-list 102 permit icmp any 61.129.120.88 0.0.0.7 packet-too-big
access-list 102 permit icmp any 61.129.120.88 0.0.0.7 echo
access-list 102 permit icmp any 61.129.120.88 0.0.0.7 time-exceeded
access-list 102 permit tcp any host 61.129.120.90 eq www
access-list 102 permit tcp any host 61.129.120.90 eq ftp
access-list 102 permit tcp any host 61.129.120.90 eq smtp
access-list 102 permit tcp any host 61.129.120.90 eq pop3
access-list 102 permit tcp any host 61.129.120.90 range 60000 65535
access-list 102 permit tcp any host 61.129.120.91 eq www
access-list 102 permit tcp any host 61.129.120.91 eq ftp
access-list 102 permit tcp any host 61.129.120.91 eq 116
access-list 102 permit tcp any host 61.129.120.91 range 60000 65535
access-list 102 permit tcp any host 61.129.120.92 eq www
access-list 102 permit tcp any host 61.129.120.92 eq 8080
access-list 102 permit tcp any host 61.129.120.92 eq 443
access-list 102 permit tcp any host 61.129.120.92 eq ftp
access-list 102 permit tcp any host 61.129.120.92 range 60000 65535
access-list 102 permit tcp any host 61.129.120.94 eq www
access-list 102 permit tcp any host 61.129.120.94 eq 8080
access-list 102 permit tcp any host 61.129.120.94 eq 443
access-list 102 permit tcp any host 61.129.120.94 eq ftp
access-list 102 permit tcp any host 61.129.120.94 range 60000 65535
access-list 102 permit tcp any host 61.129.120.95 eq www
access-list 102 permit tcp any host 61.129.120.90 eq 143
access-list 102 permit tcp any host 61.129.120.95 eq 4500
access-list 102 permit tcp any host 61.129.120.95 eq 500
access-list 102 permit tcp any host 61.129.120.93 eq 1723
access-list 102 permit gre any host 61.129.120.93
access-list 102 permit tcp any host 61.129.120.90 eq 443
access-list 102 permit tcp any host 61.129.120.93 eq 5900
access-list 102 permit tcp any host 61.129.120.91 eq 443
access-list 102 permit tcp any host 61.129.120.94 eq smtp
access-list 102 permit tcp any host 61.129.120.94 eq smtp
access-list 102 permit tcp any host 61.129.120.93 eq 143
access-list 102 permit tcp any host 61.129.120.93 eq 443
access-list 102 permit tcp any host 61.129.120.93 eq www
access-list 102 permit tcp any host 61.129.120.93 eq www
access-list 102 permit tcp any host 61.129.120.89 eq 443
access-list 102 permit tcp any host 61.129.120.92 eq 81
access-list 102 permit icmp any any echo
access-list 102 permit ip host 195.70.224.26 any
access-list 102 permit gre any host 61.129.120.89
access-list 102 permit icmp any 61.129.65.224 0.0.0.3 echo-reply
access-list 102 permit icmp any 61.129.65.224 0.0.0.3 administratively-prohibite
access-list 102 permit icmp any 61.129.65.224 0.0.0.3 packet-too-big
access-list 102 permit icmp any 61.129.65.224 0.0.0.3 echo
access-list 102 permit icmp any 61.129.65.224 0.0.0.3 time-exceeded
access-list 102 permit tcp any host 61.129.120.94 eq 1723
access-list 102 permit gre any host 61.129.120.94
access-list 102 permit icmp any 61.129.120.88 0.0.0.7 unreachable
access-list 102 permit icmp any 61.129.65.224 0.0.0.3 unreachable
snmp-server community xxx RO 98
!
!
!
!
control-plane
!
!
!
line con 0
exec-timeout 0 0
login local
stopbits 1
line aux 0
line vty 0 4
access-class 99 in vrf-also
exec-timeout 120 0
logging synchronous
login local
!
scheduler max-task-time 5000
scheduler allocate 20000 1000
ntp clock-period 17178236
ntp server 195.70.224.26
end
Turn on audit trail if not already on:
(config)# ip inspect audit-trail (watch logs)
Can you post "show ip inspect all"
CBAC will detect amd bock SMTP attacks or attackes in general !!
harbor235 ;}
ASKER
Thank you for your reply!
That firewall cause sounds reasonable.
I activated audit-trail as you suggested. I attached the outputs of show ip inspect all before and after trying to connect to 128.138.140.100 (one of those bogus IPs causing the troubles), as well as the output of "show logg".
Unfortunately I am not able to derive a solution from that output... do you have any suggestions? Should we turn off inspection of tcp? Thank you very much for an answer in advance!
show-ip-inspec-all-before.txt
show-ip-inspect-after.txt
show-logg.txt
That firewall cause sounds reasonable.
I activated audit-trail as you suggested. I attached the outputs of show ip inspect all before and after trying to connect to 128.138.140.100 (one of those bogus IPs causing the troubles), as well as the output of "show logg".
Unfortunately I am not able to derive a solution from that output... do you have any suggestions? Should we turn off inspection of tcp? Thank you very much for an answer in advance!
show-ip-inspec-all-before.txt
show-ip-inspect-after.txt
show-logg.txt
I did not look at your symtoms close enough the first time.
You should be able to initiate traffic from inside to 128.138.140.100 because established connectionsare allowed. Access-list 102 is used to punch holes into CBAC allowing traffic from outside to inisde.
Why are you trying to connect to ivalid IPs? When this phenomenon happens what does "show ip nat trans " look like? is the translations still there?
Are you using a up to date version of code to mitigate potential bugs? Not the newest version
but the most stable for your platform and feature set?
You are not inspecting SMTP but CBAC still perfroms basic IDS/IDP, so if the invalid SMTP server
is flagged it could be that your mailserver is being protected. However, I am not positive because
I am not sure what invalid IP and a MX record pointing to a non-existent IP are. Do these problems
only happen when looking at invalid IPS?
harbor235 ;}
ASKER
Thank you for your reply!
The server is connecting to invalid IPs because it gets spam mail with a From-header containing a certain sender domain (e.g. @lcd.colorado.edu) and then tries to reply by looking up the DNS MX record for the domain and trying to connect to the IP port 25 in the MX record. Unfortunately we cannot avoid this case under any circumstances since there are cases where a reply is needed (not to spam but to regular mails, but how to tell the difference for 100%?).
This situation is the only case I could identify for causing our problems since it happened quite often up to now, but there may be further I did not notice so far.
By "invalid IP" I mean IPs where a connection attempt returns a "no route to host" error. I do not know what is special about these IPs, I always found them in our mailserver's log as MX for any spam domain (128.138.140.100 is a real example). If I make up fantasy IPs which really do not exist, I just get a timeout, not a "no route to host" error. Also, I just found out, that the problem depends on the port I try to connect to. For instance, connecting to port 25 or 80 of 128.138.140.100 triggers the problem, port 23 or port 110 does not. Obviously these IPs are not non-existing. They must be listening on port 25 and return maybe some disturbing packages for the Cisco...?
The router software is not the most recent version, but very new. The most recent version unfortunately causes other problems, according to our ISP. The inside server machines are running on different Linux kernel versions (2.6.xxx) and on different hardware, I doubt that these are the cause because I can trigger the same phenomenon from different inside machines.
I attached the nat translation table before and after triggering the problem. It is the output of "show ip nat trans verbose" and I cleared the nat table before to reduce the number of entries which have nothing to do with the problem (a lot of connections are established within seconds). Also, I deleted all entries related to other internal servers. As you see, the static nat entry disappears from the table when the problem is triggered.
show-ip-nat-trans-before.txt
show-ip-nat-trans-after.txt
The server is connecting to invalid IPs because it gets spam mail with a From-header containing a certain sender domain (e.g. @lcd.colorado.edu) and then tries to reply by looking up the DNS MX record for the domain and trying to connect to the IP port 25 in the MX record. Unfortunately we cannot avoid this case under any circumstances since there are cases where a reply is needed (not to spam but to regular mails, but how to tell the difference for 100%?).
This situation is the only case I could identify for causing our problems since it happened quite often up to now, but there may be further I did not notice so far.
By "invalid IP" I mean IPs where a connection attempt returns a "no route to host" error. I do not know what is special about these IPs, I always found them in our mailserver's log as MX for any spam domain (128.138.140.100 is a real example). If I make up fantasy IPs which really do not exist, I just get a timeout, not a "no route to host" error. Also, I just found out, that the problem depends on the port I try to connect to. For instance, connecting to port 25 or 80 of 128.138.140.100 triggers the problem, port 23 or port 110 does not. Obviously these IPs are not non-existing. They must be listening on port 25 and return maybe some disturbing packages for the Cisco...?
The router software is not the most recent version, but very new. The most recent version unfortunately causes other problems, according to our ISP. The inside server machines are running on different Linux kernel versions (2.6.xxx) and on different hardware, I doubt that these are the cause because I can trigger the same phenomenon from different inside machines.
I attached the nat translation table before and after triggering the problem. It is the output of "show ip nat trans verbose" and I cleared the nat table before to reduce the number of entries which have nothing to do with the problem (a lot of connections are established within seconds). Also, I deleted all entries related to other internal servers. As you see, the static nat entry disappears from the table when the problem is triggered.
show-ip-nat-trans-before.txt
show-ip-nat-trans-after.txt
So the translation is still in the table, have you tried to actually inpect smtp so the firewall does deep packet inspection of smtp?
ip inspect name FW smtp
see if anything changes
harbor235 ;}
ASKER
I just tried and it did not solve the problem... also, there is nothing suspicious in the router log.
ok, so if you turn of CBAC does that correct the problem?
router(config)# interface Virtual-Template1
router(config-if) #no ip inspect firewall out
harbor235 ;}
Leave it off long enough to test then turn it back on
harbor235 ;}
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
You are running CBAC (IOS firewall), the firewall is inspecting tcp and is smtp aware. The firewall is most likely blocking the traffic.
CBAC can be configured to generate syslog messages if it has not already.
Did you look in the log to see if there were CBAC messages describing the traffic being filtered?
harbor235 ;}