Solved

Cisco 3845 Router / dropping packets or resetting sockets

Posted on 2006-11-21
11
1,433 Views
Last Modified: 2008-01-09
I have a Cisco 3845 router acting as our main gateway for a small hosting setup.  It has two interfaces facing the public network and one facing the private network, hosting a few boxes.  It seems to be resetting socket connections, especially during peak hours.  I sometimes can't open a session from the inside out at all, no matter if it's a simple web browser or a telnet to a mail server somewhere.  There are boxes hosted on the inside, so I have a pretty long ACL.  I'm noticing passive ftp sessions from the outside->in sometimes take several attempts before they connect - at any hour.  SMTP sessions from the inside->out are often reset by the router, especially during peak loads.  I've tried increasing "ip inspect max-incomplete high 1500" etc. (see the config below).   The router is running c3845-advipservicesk9-mz.124-10.bin IOS and has 1GB of RAM.  The main Internet circuit is a 1Gb.  Things seem to slow down during peek hours, presumably due to the dropped sessions.  I have pasted all the relevant portions of the config below.  Our IP address has been change to 123.123.23.*.  The gateway has been changed to 222.222.222.*.  thank you

!This is the running config of the router: 123.123.123.1
!----------------------------------------------------------------------------
!version 12.4
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
service sequence-numbers
!
hostname C3845
!
boot-start-marker
boot-end-marker
!
security authentication failure rate 3 log
security passwords min-length 6
no logging buffered
no logging console
no logging monitor
enable secret 5 [removed]
!
no aaa new-model
clock timezone UTC -8
clock summer-time UTC recurring
no ip source-route
ip cef
ip tcp synwait-time 5
!
!
no ip dhcp use vrf connected
ip dhcp excluded-address 123.123.123.1 123.123.123.2
ip dhcp excluded-address 123.123.123.5 123.123.123.254
!
ip dhcp pool sdm-pool1
   import all
   network 123.123.123.0 255.255.255.0
   dns-server 123.123.123.6 123.123.123.7
   default-router 123.123.123.1
   domain-name SOMEDOMAINNAME.COM
!
!
no ip bootp server
ip domain name SOMEDOMAINNAME.COM
ip name-server 123.123.123.7
ip name-server 123.123.123.6
ip name-server 123.123.123.11
ip name-server 123.123.123.12
ip ssh time-out 60
ip ssh authentication-retries 2
ip inspect max-incomplete high 1500
ip inspect one-minute high 1500
ip inspect tcp max-incomplete host 150 block-time 0
ip inspect name Inside->Out tcp
ip inspect name Inside->Out udp
ip inspect name Inside->Out dns
ip inspect name Inside->Out ntp
ip inspect name Inside->Out time
ip inspect name Inside->Out icmp
ip inspect name Inside->Out fragment maximum 256 timeout 1
ip inspect name Inside->Out echo
ip inspect name Inside->Out finger
ip inspect name Inside->Out ftp
ip inspect name Inside->Out telnet
ip inspect name Inside->Out ssh
ip inspect name Inside->Out netstat
ip inspect name Inside->Out h323
ip inspect name Inside->Out h323callsigalt
ip inspect name Inside->Out h323gatestat
ip inspect name Inside->Out appleqtc
ip inspect name Inside->Out netshow
ip inspect name Inside->Out rtsp
ip inspect name Outside->In ftp
ip ips sdf location flash://256MB.sdf autosave
ip ips notify SDEE
ip ips signature 3051 0 delete
ip ips signature 3050 0 delete
ip ips signature 3051 1 delete
ip ips signature 11228 0 delete
ip ips signature 11225 0 delete
ip ips signature 11224 0 delete
ip ips signature 11222 0 delete
ip ips signature 11221 0 delete
ip ips signature 11218 0 delete
ip ips signature 11217 0 delete
ip ips signature 11212 0 delete
ip ips signature 11211 0 delete
ip ips signature 11210 0 delete
ip ips signature 11209 0 delete
ip ips signature 11208 0 delete
ip ips signature 11207 0 delete
ip ips signature 11202 0 delete
ip ips signature 11201 0 delete
ip ips signature 11200 0 delete
!
voice-card 0
 no dspfarm
!
!
crypto pki trustpoint TP-self-signed-2502529158
 enrollment selfsigned
 subject-name cn=IOS-Self-Signed-Certificate-2502529158
 revocation-check none
 rsakeypair TP-self-signed-2502529158
!
!
crypto pki certificate chain TP-self-signed-2502529158
 certificate self-signed 01
  [removed]
username [removed]
!
!
!
crypto isakmp policy 1
 encr 3des
 authentication pre-share
 group 2
crypto isakmp key [removed]
!
!
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto ipsec transform-set ESP-3DES-SHA1 esp-3des esp-sha-hmac
crypto ipsec transform-set ESP-3DES-SHA2 esp-3des esp-sha-hmac
crypto ipsec transform-set ESP-3DES-SHA3 esp-3des esp-sha-hmac
crypto ipsec transform-set ESP-3DES-SHA4 esp-3des esp-sha-hmac
crypto ipsec transform-set ESP-3DES-SHA5 esp-3des esp-sha-hmac
crypto ipsec transform-set ESP-3DES-SHA6 esp-3des esp-sha-hmac
crypto ipsec transform-set ESP-3DES-SHA7 esp-3des esp-sha-hmac
!
crypto dynamic-map SDM_DYNMAP_1 1
 set transform-set ESP-3DES-SHA1
 set pfs group2
 match address 100
!
!
crypto map SDM_CMAP_1 65535 ipsec-isakmp dynamic SDM_DYNMAP_1
!
!
!
!
interface GigabitEthernet0/0
 description $ETH-SW-LAUNCH$$INTF-INFO-GE 0/0$$ES_WAN$$FW_OUTSIDE$$ETH-WAN$
 ip address 111.222.121.26 255.255.255.248
 ip access-group 101 in
 ip verify unicast reverse-path
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip virtual-reassembly
 ip route-cache flow
 duplex auto
 speed auto
 media-type sfp
 no mop enabled
 crypto map SDM_CMAP_1
 crypto ipsec df-bit clear
!
interface GigabitEthernet0/1
 description DSL$ETH-WAN$
 ip address 22.11.150.142 255.255.255.252
 ip access-group sdm_gigabitethernet0/1_in_100 in
 ip verify unicast reverse-path
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip virtual-reassembly
 ip route-cache flow
 duplex auto
 speed auto
 media-type rj45
 no mop enabled
!
interface FastEthernet0/0/0
!
interface FastEthernet0/0/1
!
interface FastEthernet0/0/2
!
interface FastEthernet0/0/3
!
interface GigabitEthernet0/1/0
 ip address 123.123.123.1 255.255.255.0
 ip access-group sdm_gigabitethernet0/1_in in
 ip access-group sdm_gigabitethernet0/1_out out
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip inspect Inside->Out in
 ip inspect Outside->In out
 ip virtual-reassembly
 ip route-cache flow
 negotiation auto
 no mop enabled
!
interface Vlan1
 no ip address
!
ip route 0.0.0.0 0.0.0.0 111.222.121.25
ip route 10.1.1.0 255.255.255.0 123.123.123.29
ip route 10.1.2.0 255.255.255.0 123.123.123.55
ip route 10.69.0.0 255.255.0.0 123.123.123.29
ip route 65.126.210.0 255.255.255.0 22.11.150.141
ip route 22.11.128.0 255.255.224.0 22.11.150.141
ip route 22.11.140.0 255.255.255.0 111.222.121.25
ip route 22.11.141.0 255.255.255.0 111.222.121.25
ip route 207.158.33.160 255.255.255.224 22.11.150.141
!
!
ip http server
ip http authentication local
ip http secure-server
ip http timeout-policy idle 5 life 86400 requests 10000
!
ip access-list extended inbound_IPS_filter
 remark SDM_ACL Category=1
 remark deny private 10.5.1.x
 deny   ip 10.5.1.0 0.0.0.255 any
 permit ip any any
ip access-list extended sdm_gigabitethernet0/1_in
 remark SDM_ACL Category=1
 permit ip any any
ip access-list extended sdm_gigabitethernet0/1_in_100
 remark auto generated by SDM firewall configuration
 remark SDM_ACL Category=1
 deny   ip host 255.255.255.255 any
 deny   ip host 0.0.0.0 any
 deny   ip 127.0.0.0 0.255.255.255 any
 deny   ip 172.16.0.0 0.15.255.255 any
 deny   ip 192.168.0.0 0.0.255.255 any
 deny   ip 10.0.0.0 0.255.255.255 any
 deny   ip 123.123.123.0 0.0.0.255 any
 remark permit to cns only
 permit ip any 123.123.123.0 0.0.0.255
ip access-list extended sdm_gigabitethernet0/1_out
 remark SDM_ACL Category=1
 remark trusted networks
 permit ip 10.5.1.0 0.0.0.31 any
 permit ip 10.5.1.32 0.0.0.31 host 123.123.123.247
 permit ip 10.5.1.64 0.0.0.31 host 123.123.123.250
 permit ip 10.5.1.64 0.0.0.31 host 123.123.123.236
 permit ip 10.5.1.64 0.0.0.31 host 123.123.123.248
 remark DNS-NS1-TCP
 permit tcp any host 123.123.123.6 eq domain
 remark DNS-NS1-UDP
 permit udp any host 123.123.123.6 eq domain
 remark DNS-NS2-TCP
 permit tcp any host 123.123.123.7 eq domain
 remark DNS-NS2-UDP
 permit udp any host 123.123.123.7 eq domain
 remark DNS-NS3-TCP
 permit tcp any host 123.123.123.11 eq domain
 remark DNS-NS3-UDP
 permit udp any host 123.123.123.11 eq domain
 remark DNS-NS4-TCP
 permit tcp any host 123.123.123.12 eq domain
 remark DNS-NS4-UDP
 permit udp any host 123.123.123.12 eq domain
 remark block to these few hosts first
 deny   ip any host 123.123.123.10
 deny   ip any host 123.123.123.248
 deny   ip any host 123.123.123.247
 remark TWC-DNS-NS1-TCP
 permit tcp any host 123.123.123.251 eq domain
 remark TWC-DNS-NS1-UDP
 permit udp any host 123.123.123.251 eq domain
 remark TWC-DNS-NS2-TCP
 permit tcp any host 123.123.123.252 eq domain
 remark TWC-DNS-NS2-UDP
 permit udp any host 123.123.123.252 eq domain
 remark WWB BOX - DVW IPSEC/P2P
 permit tcp any host 123.123.123.250 eq 1723
 remark WWB BOX - DVW IPSEC/P2P
 permit gre any host 123.123.123.250
 remark WWB BOX - DVW IPSEC/P2P
 permit esp any host 123.123.123.250
 remark WWB BOX - DVW IPSEC/P2P
 permit ahp any host 123.123.123.250
 remark Enigma
 permit udp any host 123.123.123.250 eq isakmp
 remark WWB BOX - DVW IPSEC/P2P
 permit udp any host 123.123.123.250 eq 1700
 remark WWB BOX - DVW IPSEC/P2P
 permit udp any host 123.123.123.250 eq 1701
 remark WWB BOX - DVW IPSEC/P2P
 permit udp any host 123.123.123.250 eq non500-isakmp
 remark WWB BOX - DVW IPSEC/P2P
 permit udp any host 123.123.123.250
 remark WWB BOX - DVW
 deny   ip any host 123.123.123.250
 remark vic
 permit tcp any host 123.123.123.225 eq 22
 remark vic
 permit tcp any host 123.123.123.226 eq 22
 remark vic
 permit tcp any host 123.123.123.227 eq 22
 remark vic
 deny   tcp any host 123.123.123.225 log
 remark vic
 deny   tcp any host 123.123.123.226 log
 remark vic
 deny   tcp any host 123.123.123.227 log
 remark time server
 permit udp any host 123.123.123.1 eq ntp
 remark all / ftp, ftp-data
 permit tcp any any range ftp-data ftp
 remark ALL / www
 permit tcp any any eq www
 remark all / https
 permit tcp any any eq 443
 remark terminal services
 permit tcp any any eq 3389
 remark ssh
 permit tcp any any eq 22
 remark IMAPI4
 permit tcp any any eq 143
 remark SMTP
 permit tcp any any eq smtp
 remark SMTP - 587
 permit tcp any any eq 587
 remark POP3
 permit tcp any any eq pop3
 permit icmp any any echo
 permit icmp any any echo-reply
 permit icmp any any time-exceeded log
 permit icmp any any unreachable log
 deny   ip any any log
!
logging trap notifications
logging 123.123.123.38
access-list 1 remark SDM_ACL Category=1
access-list 1 permit 123.123.123.244
access-list 100 remark SDM_ACL Category=4
access-list 100 remark IPSec Rule
access-list 100 permit ip 123.123.123.0 0.0.0.255 10.5.1.0 0.0.0.255
access-list 101 remark auto generated by SDM firewall configuration
access-list 101 remark SDM_ACL Category=1
access-list 101 permit udp any host 111.222.121.26 eq non500-isakmp
access-list 101 permit udp any host 111.222.121.26 eq isakmp
access-list 101 permit esp any host 111.222.121.26
access-list 101 permit ahp any host 111.222.121.26
access-list 101 remark PL&OB IPSEC
access-list 101 permit ip 10.5.1.0 0.0.0.31 123.123.123.0 0.0.0.255
access-list 101 deny   ip host 255.255.255.255 any
access-list 101 deny   ip host 0.0.0.0 any
access-list 101 deny   ip 127.0.0.0 0.255.255.255 any
access-list 101 deny   ip 172.16.0.0 0.15.255.255 any
access-list 101 deny   ip 192.168.0.0 0.0.255.255 any
access-list 101 deny   ip 10.0.0.0 0.255.255.255 any
access-list 101 deny   ip 123.123.123.0 0.0.0.255 any log
access-list 101 permit ip any any
snmp-server community communityname RO
snmp-server host 123.123.123.30 communityname
no cdp run
!
!
!
!
control-plane
!
!
!
!
!
!
!
!
!
banner login ^CAuthorized access only!
 Disconnect IMMEDIATELY if you are not an authorized user!^C
!
line con 0
 login local
 transport output telnet
 stopbits 1
line aux 0
 login local
 transport output telnet
 stopbits 1
line vty 0 4
 privilege level 15
 login local
 transport input telnet ssh
line vty 5 15
 privilege level 15
 login local
 transport input telnet ssh
!
scheduler allocate 20000 1000
ntp clock-period 17179922
ntp source GigabitEthernet0/1/0
ntp update-calendar
ntp server 123.123.123.1 key 0 prefer
ntp peer 204.34.198.40 prefer
ntp peer 204.34.198.41
!
end




0
Comment
Question by:BarrySDCA
  • 6
  • 4
11 Comments
 
LVL 6

Expert Comment

by:nexissteve
ID: 17985915
Hi,

Is this a new implemtation?

One thing - shutdown all ports that are not in use.

interface FastEthernet0/0/0
!
interface FastEthernet0/0/1
!
interface FastEthernet0/0/2
!
interface FastEthernet0/0/3
!

I know it wont affect the performance issue on the router - but it is good practice none the less.

Its hard to see if you have a routing issue here. Have you checked CPU utilisation , Mem utilisation and error counters on the ports?
0
 

Author Comment

by:BarrySDCA
ID: 17987215
if you can send me a few commands to throw at it I will and post it here.  I am certain it is the router.  I can open sites and connect to services on the same swich.  When I go through the router (from the inside), I notice all kinds of problems.  It is not a new install, but it is a new IOS.  The flash fried, taking everything down several weeks ago.  All I could do was replace the flash and re-load.  thank you
0
 
LVL 1

Expert Comment

by:martin_wilkinson
ID: 17992678
Can you post the output of 'show logging'?  You may need to enable logging to the internal buffer with 'logging buffered' in your config.

The firewall feature set will block connections under certain circumstances and the reason for this will show up in the log if this is the cause of your troubles.
0
 

Author Comment

by:BarrySDCA
ID: 17998771
well that was interesting and I should have done it before.  My mind is so tangled up with this issue that I'm not even thinking straight any more.

What I found was CBAC is *not* working.  Outside hosts on port 25 are being blocked.  It's obvious a mail server here is reaching out to a mail server somewhere else, and the packet is being dropped.  Now why would CBAC not be working???  thank you!
0
 
LVL 1

Expert Comment

by:martin_wilkinson
ID: 17999194
How do you know CBAC is not working?  Don't be fooled (like I was) by the change in CBAC's behaviour somewhere along the 12.3T train.  You used to be able to see the entries created by CBAC in the ACL by doing a 'show access-list xxx' - this is no longer the case.

The only way to see the CBAC entries now is with a 'sh ip inspect session' or 'sh ip inspect stat'.


0
Control application downtime with dependency maps

Visualize the interdependencies between application components better with Applications Manager's automated application discovery and dependency mapping feature. Resolve performance issues faster by quickly isolating problematic components.

 

Author Comment

by:BarrySDCA
ID: 18000251
I don't see how CBAC can be working because I clearly see sessions being blocked by the log.  

For example:

353838: Nov 22 16:23:17.684 UTC: %SEC-6-IPACCESSLOGP: list sdm_gigabitethernet0/
1_out denied tcp 207.249.96.82(25) -> 123.123.123.127(2691), 1 packet

This is obviously a message from the inside trying to get out and the outside server (207.249.96.82) is simply responding to a packet initiated from a mail server on the inside.  I have since added an ACL on g0/0 outbound with permit all.  I remember a while back CBAC would not be criticized if there was no ACL on the interface.  I also enabled CBAC on the outbound of this interface, the main Internet interface.  So...if CBAC was working, it should not be stopping this type of traffic.....right?

We are also having problems with email users using their email client....seems the same thing....packets are being dropped and I have no idea why!

ya...still major problems over here and no turkey day until I resolve it.    I am open to any and all input.

Thank you!
0
 

Author Comment

by:BarrySDCA
ID: 18011616
Martin...You have helped tremendously.  The log was showing packets being dropped, but CBAC should have opened the ports.  I removed CBAC code and re-applied it.  It appears to be working.   I have NO IDEA why that was happening.  We are under a light load, but I think it may be fixed now.  I will follow-up.  thank you!
0
 
LVL 1

Expert Comment

by:martin_wilkinson
ID: 18015518
Hi Barry, How's it looking?

I've never seen that before with CBAC, but i've had similar issues with NAT and IPSec.
0
 

Author Comment

by:BarrySDCA
ID: 18015649
so far so good.  We still have a light load, so I would like to wait until tomorrow (Monday) to give it a real test.  But so far, so good!

We are running IPSEC in the router, but I am positive one has nothing to do with the other in this case.  I've been watching the CBAC sessions constantly (I wish there was a way to make it keep displaying open sessions every x seconds...like nestat

anyway...so far so good.  I will definitely follow-up Monday afternoon, after we push it a bit.

Thank you!
0
 
LVL 1

Accepted Solution

by:
martin_wilkinson earned 250 total points
ID: 18051157
Hi Barry - all good?
0
 

Author Comment

by:BarrySDCA
ID: 18051223
You know...it's good!  I've been swamped by another issue, one client was receiving 200 spam messages/hr.  So since that didn't have any connectivity issues (haha), ya...all good!

The script was correct, but the router had something corrupt.  I basically entered 'no ip inspect' and then entered the *same* exact config again.  bam!  it worked.  I should have known better to look at the logs, but I was just going crazy with it because I knew my script was good!  Anyway, points well earned and very much appreciated.  Thank you!
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Imagine you have a shopping list of items you need to get at the grocery store. You have two options: A. Take one trip to the grocery store and get everything you need for the week, or B. Take multiple trips, buying an item at a time, to achieve t…
Meet the world's only “Transparent Cloud™” from Superb Internet Corporation. Now, you can experience firsthand a cloud platform that consistently outperforms Amazon Web Services (AWS), IBM’s Softlayer, and Microsoft’s Azure when it comes to CPU and …
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're looking for how to monitor bandwidth using netflow or packet s…

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now