Solved

Cisco routers and maximum addreses we can NAT

Posted on 2015-01-13
15
89 Views
Last Modified: 2015-02-18
Hi,

We have a customer that is using a 3845 for NAT and is almost at its limit with 55 customers (1 Class C address apparently). Can you recommend another router (preferably Cisco) that can NAT more?

Thanks,
Matt
0
Comment
Question by:Vardata
  • 9
  • 6
15 Comments
 
LVL 50

Expert Comment

by:Don Johnston
ID: 40547482
Why do you think that you're at the limit?

There is no "limit" on the number of address a router can NAT.
0
 

Author Comment

by:Vardata
ID: 40551696
Sorry I should have worded that better...the CPU utilization is at 75% with that many users doing NAT. Our DRAM is maxed at 1GB.
0
 

Author Comment

by:Vardata
ID: 40553934
I attached the CPU history . The 72 hour graph shows some spikes into the 90% range and a fair amount of max readings int the 50-70% range. Is this normal or do you think there should be concern with the spikes?  I also had them run a 'show processes' command and there weren't any processes that were greater than 10%. Again these are only performing NAT on ~70 customers.



natrter2#show process cpu history
 
natrter2   02:36:00 PM Thursday Jan 15 2015 UTC
 
 
    111222221111122222111112222211111222224444433333444443333333
    555777777777722222888889999999999000000000011111444446666699
100
 90
 80
 70
 60
 50
 40                                       *****     ************
 30    *****               *****          **********************
 20 ************************************************************
 10 ************************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....6
             0    5    0    5    0    5    0    5    0    5    0
               CPU% per second (last 60 seconds)
 
 
    423232234222323433233234222233443333432233433332233443434424
    490202343656084423446612945723044517038558033037925121242430
100
 90
 80
 70
 60
 50
 40 *       *      #    *  *      ** * **   **#       *** * ** *
 30 #** *  ********##* *#**** **######*###**#######**########* *
 20 ############################################################
 10 ############################################################
   0....5....1....1....2....2....3....3....4....4....5....5....6
             0    5    0    5    0    5    0    5    0    5    0
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%
 
 
    465664232242444676678746445653622133456988875576665555432312226889668654
    818864849151008715002700345984557550159324869814262662052295332580050985
100
 90                                        *  *                    ***
 80                     **                 *****                   ***  *
 70    **          *** ***     *  *       ******  *  *             *** ***
 60  ****          ******* *  *** *      ************* **         *###*****
 50 *****     *   ******** *  *** *      **###************        *###******
 40 ******    * ********#******** *   * ***#####************      *###**#***
 30 **###**** * ***#######*****#***** ****######************ * *  #####*##**
 20 #####********#########*##*###********#################********##########
 10 ########**#*####################**#*###################*******##########
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
             0    5    0    5    0    5    0    5    0    5    0    5    0
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%
0
 
LVL 50

Expert Comment

by:Don Johnston
ID: 40555042
Definitely some spiking going on.  I assume that you've looked at the CPU utilization detail and that's how you determined it's a NAT issue, right?

What speed is your internet connection?  And do you have a firewall inline?
0
 

Author Comment

by:Vardata
ID: 40557862
The only reason I mentioned NAT was because its the only thing this box is doing. when i ran the 'sh proceses cpu' there weren't any services over 10%. Of course that wasnt during a spike either.

Do you know of a command to show a history of the processes like the 'sh processes cpu history' command shows? I'm not sure about the firewall. I will ask and let you know. Thanks for your help so far...much appreciated
0
 
LVL 50

Expert Comment

by:Don Johnston
ID: 40558007
No, I dont think there's a command that lets you see the history of individual processes.  

I've always checked during the spikes.
0
 

Author Comment

by:Vardata
ID: 40558013
Ok thank you for the quick response! Wasnt sure if maybe a newer version IOS had that command.

do you think the spikes could be due to a virus somewhere on the network? Any other commands you think would be beneficial to run?
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 

Author Comment

by:Vardata
ID: 40568177
After further discussion with customer they also have a Sup720-3b in a 65060performing NAT. This also has spikes in CPU performance into the 90% level more often than the 3845's. We were going to take them to a Gig of Dram but they are now looking into purchasing a Linux server to perform NAT. Not familiar with this and was wondering if you had any experience with this? He is quite concerned and after looking at the CPU history on his Sup I can understand. I have his config if that would help shed some light on the issue. Any feedback is appreciated.

4163457002865807181947886722966776866199141405835339036975

100

 90                       *

 80                       *

 70        *   ****** *   **  **        *    *             ****

 60    ****************** ******* ******** ***#** ** * *** *#**

 50  ***#########################*###*###########**###*#*######

 40  ##########################################################

 30  ##########################################################

 20  ##########################################################

 10  ##########################################################

    0....5....1....1....2....2....3....3....4....4....5....5....

              0    5    0    5    0    5    0    5    0    5




               CPU% per minute (last 60 minutes)

              * = maximum CPU%   # = average CPU%







     8778543323456789787665565666653334446788889766677966555433556899785665

     7671964678556880387598619009326281091164005395954215332024382750106305

100                                            *                   *

 90  *             ** *                    *   *      *           ***

 80  ****         *** **                   *****    * *           *** *

 70  ****        *********      *         ************* *         *****

 60  *****      ******************       ****************       ***********

 50  ###***    ****##**************     ****###*******#*****   ************

 40  ####** * ****######*****#****** *****###############****  **######*#**

 30  #####******##################******###################*****###########

 20  ######################################################################

 10  ######################################################################

    0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.

              0    5    0    5    0    5    0    5    0    5    0    5    0




                   CPU% per hour (last 72 hours)

                  * = maximum CPU%   # = average CPU%
0
 
LVL 50

Accepted Solution

by:
Don Johnston earned 500 total points
ID: 40568196
I don't know that I would start adding RAM and swapping things out because there are spikes in the CPU utilization.  

In the 60 minute graph, there was one instance where it hit 90%.  The rest of the time the max was 70% or lower with averages at 50%.

On the 72-hour graph, there is a regular interval of high utilization where the averages and max rise  (which I'm guessing corresponds to the workday).  In that graph, there were a couple instances where it hit a high of 100%.

Think about it like this:  Whenever the CPU utilization is below 100%, the CPU is not fully utilized and has excess processing capability.

And more importantly, there's still no certainty that NAT is causing this symptom.

Now the 3845 has been EOS for about 3 years, but it's a very serviceable platform with pretty good performance.  I simply can't see 55 users putting a significant load on that router.

Let me ask you this:  What are the connections on this router?  Specifically, what speed is the internet connection and what speed is the LAN connection?

You said that NAT is "the only thing this box is doing".  Is there also a firewall somewhere doing packet inspection?

It would help to see a topology diagram of the network and the config of this router.

As for the 6500, if you're seeing a heavy load on that with 55 users, then something definitely up.  The 3845 performance is rated at 500,000pps (using CEF).  The Sup 720 is rated at up to 30 million pps.  So the 6500 shouldn't even be breathing hard.

Is it possible that CEF is disabled?  "show ip interface" will display if CEF is enabled (which it is by default).
0
 

Author Comment

by:Vardata
ID: 40581435
Sorry Don...Im down a guy in our lab and I have been busy keeping up. I agree that there has to be something else going on. Currently he has no firewall inline with the router. I do not know the connection speed, but will ask him. The 6500 has more than 55 users, but certainly not so many that it should be spiking like it is. I do have the configs on the 6500's and will forward to you as soon as I get the IP's and other private info masked. I will also get the 3845 configs. Thanks again for your input. Its much appreciated.
0
 

Author Comment

by:Vardata
ID: 40584773
Don,

Attached is the config of the 6500. Waiting on the 3845 config from customer.

BTW -  they are looking into a router from a mfg I have not heard of...Routerboard? Specifically at this model CCR1036-12G-4S. According to the specs it can do 24 million pps. Any experience with this platform?
6500Config.txt
0
 
LVL 50

Expert Comment

by:Don Johnston
ID: 40584844
I don't see anything on the 6500 config that could cause performance issues.  But... I have never seen that many secondary IP addresses on an interface before.  I know there is no theoretical limit on the number of secondary addresses but 75 seems a bit excessive. :-)

That could be the cause but I really can't say since I've never seen more than 3 secondary addresses on an interface.

Never heard of Routerboard. They appear to be a Latvian company that was established in 1995.  But there doesn't seem to be much about them until around 2008.  As for a $1,100 router that has 24mpps?  Anything is possible... I guess.  I don't think that I would be comfortable recommending this product to any of my consulting customers.
0
 

Author Comment

by:Vardata
ID: 40603995
Hi Don,

Sorry it took me so long, but here is the config on the 3845. Response from the customer regarding the connection speed..

"this cisco 3845 is connected at 1 gig --- but is processing peak traffic of about 35 Meg."

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2015.02.11 11:57:31 =~=~=~=~=~=~=~=~=~=~=~=
show run
Building configuration...

Current configuration : 10645 bytes
!
! Last configuration change at 11:52:00 UTC Tue Jan 20 2015
! NVRAM config last updated at 11:52:04 UTC Tue Jan 20 2015
!
version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec localtime
no service password-encryption
!
hostname natrter2
!
boot-start-marker
boot-end-marker
!
enable secret 5 $1$tmRv$g90CrlyAnti/1414BT6Ne/
enable password open4btc
!
no aaa new-model
clock timezone UTC -7
clock summer-time UTC recurring
dot11 syslog
 --More--         !
!
ip cef
!
!
ip name-server xxx.168.40.6
ip auth-proxy max-nodata-conns 3
ip admission max-nodata-conns 3
!
multilink bundle-name authenticated
!
mls qos map cos-dscp 0 10 16 26 32 46 48 56
voice-card 0
 no dspfarm
!
!
!
!
!
!
!
!
!
 --More--         !
!
!
!
!
!
!
!
!
!
!
!
username btc secret 5 $1$AsIw$Sz/Yydjzq2G8UcaT0SQ5P0
username dp secret 5 $1$ICCg$iTfTvOavQjTLkvBkEDrwQ.
archive
 log config
  hidekeys
!
!
!
!
!
class-map match-any AutoQoS-VoIP-Remark
 --More--          match ip dscp ef
 match ip dscp cs3
 match ip dscp af31
 match ip dscp af11
class-map match-any AutoQoS-VoIP-Control-UnTrust
 match access-group name AutoQoS-VoIP-Control
 match protocol rtcp
class-map match-any AutoQoS-VoIP-RTP-UnTrust
 match access-group name AutoQoS-VoIP-RTCP
!
!
policy-map AutoQoS-Policy-UnTrust
 class AutoQoS-VoIP-RTP-UnTrust
  priority percent 10
 class AutoQoS-VoIP-Control-UnTrust
  bandwidth percent 10
 class class-default
  fair-queue
!
!
!
!
!
 --More--         interface Loopback0
 ip address 10.3.3.8 255.255.255.0
 no ip redirects
 ip nat inside
 ip virtual-reassembly
!
interface GigabitEthernet0/0
 no ip address
 ip nat outside
 ip virtual-reassembly
 duplex auto
 speed auto
 media-type rj45
 service-policy output AutoQoS-Policy-UnTrust
!
interface GigabitEthernet0/0.1
 encapsulation dot1Q 1 native
 ip address xxx.168.40.8 255.255.255.224
 ip flow ingress
 ip nat outside
 ip virtual-reassembly
!
interface GigabitEthernet0/0.2
 --More--          encapsulation dot1Q 2
 ip address 10.235.2.1 255.255.255.0 secondary
 ip address 10.235.1.1 255.255.255.0
 ip access-group 111 out
 no ip redirects
 no ip unreachables
 ip nat inside
 ip virtual-reassembly
 ip policy route-map NAT
 no ip mroute-cache
 arp timeout 1200
!
interface GigabitEthernet0/1
 no ip address
 shutdown
 duplex auto
 speed auto
 media-type rj45
!
router ospf 8
 log-adjacency-changes
 network 10.235.1.0 0.0.0.255 area 1
 network xxx.168.40.0 0.0.0.255 area 0
 --More--         !
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 xxx.168.40.1
ip route 10.235.1.18 255.255.255.255 xxx.168.68.101
ip route 10.235.2.9 255.255.255.255 xxx.168.40.23
ip route 10.235.2.13 255.255.255.255 xxx.168.40.84
ip route 10.235.254.0 255.255.255.0 xxx.168.40.2
ip route 10.236.1.0 255.255.255.0 xxx.168.68.102
ip route 10.236.2.0 255.255.255.0 xxx.168.68.102
ip route 10.237.2.9 255.255.255.255 xxx.168.40.23
ip route 10.237.2.13 255.255.255.255 xxx.168.40.84
ip route 10.243.1.0 255.255.255.0 xxx.168.40.9
ip route 10.247.1.0 255.255.255.0 xxx.168.68.100
ip route 10.247.2.0 255.255.255.0 xxx.168.68.100
ip route 172.31.1.0 255.255.255.0 xxx.168.41.182
ip route xxx.168.40.0 255.255.255.0 GigabitEthernet0/0.1
ip route xxx.168.40.6 255.255.255.255 GigabitEthernet0/0.1
ip route xxx.168.40.80 255.255.255.248 xxx.168.40.81
ip route xxx.168.40.80 255.255.255.248 xxx.168.40.1
ip route xxx.168.40.160 255.255.255.224 xxx.168.40.2
ip route xxx.168.41.4 255.255.255.252 xxx.168.40.2
ip route xxx.168.41.24 255.255.255.252 xxx.168.40.74
ip route xxx.168.41.108 255.255.255.252 xxx.168.40.2
 --More--         ip route xxx.168.41.116 255.255.255.252 xxx.168.40.2
ip route xxx.168.41.180 255.255.255.252 xxx.168.41.181
ip route xxx.168.41.180 255.255.255.252 xxx.168.40.2
ip route xxx.168.41.184 255.255.255.252 xxx.168.40.2
ip route xxx.168.41.208 255.255.255.240 xxx.168.40.4
ip route xxx.168.42.40 255.255.255.248 xxx.168.40.2
ip route xxx.168.68.0 255.255.255.0 xxx.168.40.1
!
ip flow-cache timeout inactive 10
ip flow-cache timeout active 1
ip flow-export source GigabitEthernet0/0.1
ip flow-export version 5
ip flow-export destination xxx.118.14.30 2058
!
ip http server
no ip http secure-server
ip nat translation tcp-timeout 150
ip nat translation udp-timeout 60
ip nat translation icmp-timeout 45
ip nat pool btcw xxx.168.43.81 xxx.168.43.82 netmask 255.255.255.252
ip nat inside source list 110 pool btcw overload
!
ip access-list extended AutoQoS-VoIP-RTCP
 --More--          permit udp any any range 16384 32767
!
access-list 4 permit xxx.168.40.0 0.0.0.255
access-list 4 permit xxx.168.41.0 0.0.0.255
access-list 4 permit xxx.168.43.0 0.0.0.255
access-list 4 permit xxx.168.42.0 0.0.0.255
access-list 4 permit xxx.168.44.0 0.0.0.255
access-list 99 remark < prohibit unauthorized access >
access-list 99 permit xxx.168.40.0 0.0.0.255
access-list 99 permit xxx.168.41.0 0.0.0.255
access-list 99 permit xxx.168.68.0 0.0.0.255
access-list 110 permit ip 10.235.1.0 0.0.0.255 any
access-list 111 permit tcp any any eq ftp
access-list 111 permit tcp any eq ftp any
access-list 111 permit udp any any eq 1023
access-list 111 permit tcp any any eq 1723
access-list 111 permit udp any any eq 1723
access-list 111 permit icmp xxx.168.40.0 0.0.0.255 any
access-list 111 permit udp xxx.168.40.0 0.0.0.255 any
access-list 111 permit icmp xxx.168.41.0 0.0.0.255 any
access-list 111 permit udp xxx.168.41.0 0.0.0.255 any
access-list 111 permit icmp xxx.168.42.0 0.0.0.255 any
access-list 111 permit udp xxx.168.42.0 0.0.0.255 any
 --More--         access-list 111 permit icmp xxx.168.43.0 0.0.0.255 any
access-list 111 permit udp xxx.168.43.0 0.0.0.255 any
access-list 111 permit udp any xxx.168.40.0 0.0.0.255
access-list 111 permit ip host 72.166.82.62 any
access-list 111 permit tcp any any eq 443
access-list 111 permit udp any any eq 443
access-list 111 permit icmp any any echo-reply
access-list 111 permit icmp any any time-exceeded
access-list 111 permit tcp any any eq 500
access-list 111 permit ip any 10.242.2.0 0.0.0.255
access-list 111 remark <   SIP VOIP ports>
access-list 111 permit udp any any range 5060 5080
access-list 111 permit tcp any any range 5060 5080
access-list 111 remark <   Deny RFC 1918>
access-list 111 deny   ip 192.168.0.0 0.0.255.255 any
access-list 111 deny   ip host 0.0.0.0 any
access-list 111 remark <   DNS Amplification Exploited ports>
access-list 111 deny   tcp any any eq domain
access-list 111 remark <   Exploited ports>
access-list 111 deny   tcp any any eq smtp
access-list 111 deny   udp any any eq 25
access-list 111 deny   udp any any eq tftp
access-list 111 deny   tcp any any eq 135
 --More--         access-list 111 deny   udp any any eq 135
access-list 111 deny   tcp any any eq 137
access-list 111 deny   udp any any eq netbios-ns
access-list 111 deny   udp any any eq netbios-dgm
access-list 111 deny   tcp any any eq 138
access-list 111 deny   tcp any any eq 139
access-list 111 deny   udp any any eq snmp
access-list 111 deny   udp any any eq snmptrap
access-list 111 deny   udp any any eq netbios-ss
access-list 111 deny   tcp any any eq 193
access-list 111 deny   tcp any any eq 445
access-list 111 deny   tcp any any eq 587
access-list 111 deny   tcp any any eq 593
access-list 111 deny   tcp any any eq 707
access-list 111 deny   udp any any range 995 999
access-list 111 remark <   added Jan 29>
access-list 111 deny   udp any any eq 1389
access-list 111 deny   udp any any eq 2983
access-list 111 remark <  end  added Jan 29>
access-list 111 remark < deny Bittorrent P2P downloads>
access-list 111 deny   tcp any any range 1027 1039
access-list 111 deny   tcp any any eq 1040
access-list 111 deny   tcp any any eq 1111
 --More--         access-list 111 deny   tcp any any eq 1143
access-list 111 deny   tcp any any eq 1293
access-list 111 deny   tcp any any eq 1303
access-list 111 deny   tcp any any eq 1426
access-list 111 deny   tcp any any eq 1842
access-list 111 deny   tcp any any eq 1900
access-list 111 deny   tcp any any eq 1955
access-list 111 deny   tcp any any eq 2084
access-list 111 deny   tcp any any eq 2085
access-list 111 deny   tcp any any eq 2086
access-list 111 deny   tcp any any eq 2087
access-list 111 deny   tcp any any eq 2088
access-list 111 deny   tcp any any eq 2089
access-list 111 deny   tcp any any eq 2754
access-list 111 deny   tcp any any eq 3410
access-list 111 deny   tcp any any eq 3826
access-list 111 deny   tcp any any eq 4444
access-list 111 deny   tcp any any eq 4540
access-list 111 deny   tcp any any eq 5370
access-list 111 deny   tcp any any eq 6881
access-list 111 deny   tcp any any eq 6882
access-list 111 deny   tcp any any eq 6883
access-list 111 deny   tcp any any eq 6884
 --More--         access-list 111 deny   tcp any any eq 6885
access-list 111 deny   tcp any any eq 6886
access-list 111 deny   tcp any any eq 6887
access-list 111 deny   tcp any any eq 6888
access-list 111 deny   tcp any any eq 6969
access-list 111 deny   tcp any any eq 17148
access-list 111 deny   tcp any any eq 18656
access-list 111 deny   tcp any any eq 22276
access-list 111 deny   tcp any any eq 25675
access-list 111 deny   tcp any any eq 29249
access-list 111 deny   tcp any any eq 32100
access-list 111 deny   tcp any any eq 32851
access-list 111 deny   tcp any any eq 35415
access-list 111 deny   tcp any any eq 40743
access-list 111 deny   tcp any any eq 43356
access-list 111 deny   tcp any any eq 44789
access-list 111 deny   tcp any any eq 45807
access-list 111 deny   tcp any any eq 46059
access-list 111 deny   tcp any any eq 46984
access-list 111 deny   tcp any any eq 48195
access-list 111 deny   tcp any any eq 48716
access-list 111 deny   tcp any any eq 51236
access-list 111 deny   tcp any any eq 51760
 --More--         access-list 111 deny   tcp any any eq 54027
access-list 111 deny   tcp any any eq 54312
access-list 111 deny   tcp any any eq 55289
access-list 111 deny   tcp any any eq 58719
access-list 111 deny   tcp any any eq 62020
access-list 111 remark < end Bittorrent>
access-list 111 deny   udp any any eq 1900
access-list 111 deny   udp any any eq 8998
access-list 111 deny   udp any any eq 16470
access-list 111 deny   udp any any eq 16464
access-list 111 deny   udp any any eq 16465
access-list 111 deny   udp any any eq 16471
access-list 111 deny   udp any any eq 51236
access-list 111 deny   udp any any eq 59355
access-list 111 permit ip any any
snmp-server community fonehm2btc RO
snmp-server community 4novam2get RO
snmp-server host xxx.118.1.20 4novam2get
snmp-server host xxx.118.14.30 4novam2get
snmp-server host xxx.168.40.23 fonehm2btc
snmp-server host xxx.168.40.27 fonehm2btc
!
!
 --More--         !
route-map NAT permit 10
 match ip address 110
 set ip next-hop 10.3.3.2
!
!
!
!
control-plane
!
!
!
!
!
!
!
!
!
!
line con 0
line aux 0
line vty 0 4
 access-class 99 in
 --More--          password open4btc
 login
!
scheduler allocate 20000 1000
ntp clock-period 17180393
ntp server xxx.168.40.6 version 2
!
end

natrter2#               exit
0
 
LVL 50

Expert Comment

by:Don Johnston
ID: 40604372
I don't see anything in the router config that would cause any performance problems.
0
 

Author Comment

by:Vardata
ID: 40617127
Hey Don,

I'm going to ask him to put a firewall inline to see if we can determine what is reeking havoc with his routers. Hopefully this will shed some light on what is going on. I'm going to close this Post and give you the points. If you want, I will keep you informed as to what happens. Thank you for all your efforts. They are much appreciated!
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

When posting a question about a Cisco ASA, Cisco Router or Cisco Switch, it can aid diagnosis if a suitably sanitised copy of the config is provided. It is much better to leave as much of the configuration as original as possible, as it could be tha…
There are two basic ways to configure a static route for Cisco IOS devices. I've written this article to highlight a case study comparing the configuration of a static route using the next-hop IP and the configuration of a static route using an outg…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now