Cisco high CPU and NAT issue

Greetings,

I have a C2921-SEC/K9 router with a Gigabit Ethernet Service Module.  It provides an Internet gateway to about 1500 concurrent wireless users (coming from a WLAN controller).  I provide bandwidth limiting on the WLAN controller to ensure our 100M/100M Internet pipe does not become too saturated.

I have noticed that CPU typically hangs out between 40-60% during average use, but when we start running large downloads/uploads, CPU spikes to 98-99%.

Additionally, Internet dies from the LAN frequently, and the only way to fix is to "clear ip nat trans *"

I've noticed there is a decent amount of process switching going on:

Interface Embedded-Service-Engine0/0 is disabled

GigabitEthernet0/0
          Throttle count         19
                   Drops         RP         21         SP          0
             SPD Flushes       Fast      47825        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs     495779      Drops          0

    Protocol  IP                  
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process   11267820 1890300056   12813781 1087915318
            Cache misses          0          -          -          -
                    Fast  595196163 3438320894  544169576 3155494271
               Auton/SSE          0          0          0          0

    Protocol  ARP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     495779   29746740        163       9780
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process       5062    2060201      30286    1817160
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    NOTE: all counts are cumulative and reset only after a reload.
Interface GigabitEthernet0/1 is disabled

Interface GigabitEthernet0/2 is disabled

GigabitEthernet1/0
          Throttle count          0
                   Drops         RP          0         SP          0
             SPD Flushes       Fast       5331        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs     579918      Drops          0

    Protocol  IP                  
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process    1965821  333642552      52452   18166643
            Cache misses          0          -          -          -
                    Fast      28637    6408179          0          0
               Auton/SSE          0          0          0          0

    Protocol  DEC MOP            
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process        430      33110        510      39270
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Spanning Tree      
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     304510   15834520          0          0
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  ARP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     579918   34795080          6        360
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  CDP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process       5648    2795570       5652    2539774
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  SCP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0          0 4294743006
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process      13708    4468304          0          0
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    NOTE: all counts are cumulative and reset only after a reload.
GigabitEthernet1/1 Internal switch interface connected to Service Module
          Throttle count          0
                   Drops         RP          0         SP          0
             SPD Flushes       Fast        176        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs          0      Drops          0

    Protocol  DEC MOP            
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0        510      39270
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Spanning Tree      
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     152096    7908992        111       6660
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  CDP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process       5642    2764400       5642    2534938
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process      10131     486288          0          0
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    NOTE: all counts are cumulative and reset only after a reload.
Loopback0

    All statistics for this interface are zero.
Interface NVI0 is disabled

Vlan1 LAN
          Throttle count          0
                   Drops         RP        524         SP          0
             SPD Flushes       Fast     223085        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs    3601935      Drops          0
         
    Protocol  IP                  
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process   16174521 1435194246    9747182 1731922240
            Cache misses          0          -          -          -
                    Fast  550018749 3067325560  600924038 2699446987
               Auton/SSE          0          0          0          0

    Protocol  DEC MOP            
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process        430      33110        509      39193
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Spanning Tree      
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process         74       6312          0          0
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  ARP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process    2325262  139515732    1980047  118802820
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process      18445    4996035          0          0
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0


I do have CEF enabled.

interface GigabitEthernet0/0
 ip address xxx.xxx.xxx.xxx xxx.xxx.xxx.xxx
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip multicast boundary 30
 ip nat outside
 ip virtual-reassembly in
 load-interval 30
 duplex auto
 speed auto
 no cdp enable
 no mop enabled

interface Vlan1
 description LAN
 ip address 192.168.200.1 255.255.252.0 secondary
 ip flow ingress
 ip flow egress
 ip nat inside
 ip virtual-reassembly in

ip nat inside source list NAT interface GigabitEthernet0/0 overload

ip access-list standard NAT
 permit 192.168.0.0 0.0.255.255

Do I need to do anything with the Gigabit interfaces associated with the ethernet service module?
David BaumanAsked:
Who is Participating?
 
Joey YungSenior Network EngineerCommented:
New deployment or issue after change?
Can you share the basic diagram?

Show more report for further troubleshoot the issue:

show processes memory
show processes cpu
show ip nat  statistics
show ip cef switching statistics
show ip cache flow
show ver
0
 
JustInCaseCommented:
If your high CPU usage is caused by NAT, and you can confirm that by commands that Joey gave.

You can check Cisco's Troubleshoot high CPU usage

And if there is really high CPU utilization caused by natting you have two options:
limit number of sessions per host
buy router with better CPU

First is recommended, since you can kill any router with large number of hosts and many opened connections.

Verifying NAT Operation and Basic NAT Troubleshooting

Large Translation Table Increases the CPU

A host can send hundreds of translations, which in turn leads to high CPU usage. In other words, it can make the table so large that it causes the CPU to run at 100 percent. The ip nat translation max-entries 300 command makes the 300 per host limit or an aggregate limit of the amount of translations on the router. The workaround is to use the ip nat translation max-entries all-hosts 300 command.
0
 
David BaumanAuthor Commented:
Sorry for the delay.

New deployment.

I won't be able to check this again for about a month unfortunately
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

 
David BaumanAuthor Commented:
Thanks guys.

What happens when the limit is met?  Does it deny any new translations?  Delete the older ones?  It seems that when I run these commands, it seriously lags DNS requests
0
 
JustInCaseCommented:
If older entries are deleted, then you would have problem with internet - like pages don't load etc (not just serious DNS lags).
That command limits NAT entry table to 300 entries per host, so it deny any new translations. You can try to play around with value, and allow bigger NAT table per host and see where is safe limit. Max value is 2147483647 so you can play around a lot with that.
But, if that command solved your problem with CPU overload, solution it most likely temporary (network usually grow over time, and more and more resources are on internet) so you probably need to do think about some changes, either to network or to edge router.
0
 
David BaumanAuthor Commented:
Ahh ok.  So once this limit is reached, I still need to clear ip nat tran * for new requests to be processed.

Thanks!
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.