asked on

Cisco high CPU and NAT issue

Greetings,

I have a C2921-SEC/K9 router with a Gigabit Ethernet Service Module. It provides an Internet gateway to about 1500 concurrent wireless users (coming from a WLAN controller). I provide bandwidth limiting on the WLAN controller to ensure our 100M/100M Internet pipe does not become too saturated.

I have noticed that CPU typically hangs out between 40-60% during average use, but when we start running large downloads/uploads, CPU spikes to 98-99%.

Additionally, Internet dies from the LAN frequently, and the only way to fix is to "clear ip nat trans *"

I've noticed there is a decent amount of process switching going on:

Interface Embedded-Service-Engine0/0 is disabled

GigabitEthernet0/0
Throttle count 19
Drops RP 21 SP 0
SPD Flushes Fast 47825 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 495779 Drops 0

Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 11267820 1890300056 12813781 1087915318
Cache misses 0 - - -
Fast 595196163 3438320894 544169576 3155494271
Auton/SSE 0 0 0 0

Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 495779 29746740 163 9780
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 5062 2060201 30286 1817160
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

NOTE: all counts are cumulative and reset only after a reload.
Interface GigabitEthernet0/1 is disabled

Interface GigabitEthernet0/2 is disabled

GigabitEthernet1/0
Throttle count 0
Drops RP 0 SP 0
SPD Flushes Fast 5331 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 579918 Drops 0

Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 1965821 333642552 52452 18166643
Cache misses 0 - - -
Fast 28637 6408179 0 0
Auton/SSE 0 0 0 0

Protocol DEC MOP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 430 33110 510 39270
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

Protocol Spanning Tree
Switching path Pkts In Chars In Pkts Out Chars Out
Process 304510 15834520 0 0
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 579918 34795080 6 360
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

Protocol CDP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 5648 2795570 5652 2539774
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

Protocol SCP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 0 0 0 4294743006
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 13708 4468304 0 0
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

NOTE: all counts are cumulative and reset only after a reload.
GigabitEthernet1/1 Internal switch interface connected to Service Module
Throttle count 0
Drops RP 0 SP 0
SPD Flushes Fast 176 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 0 Drops 0

Protocol DEC MOP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 0 0 510 39270
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

Protocol Spanning Tree
Switching path Pkts In Chars In Pkts Out Chars Out
Process 152096 7908992 111 6660
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

Protocol CDP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 5642 2764400 5642 2534938
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 10131 486288 0 0
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

NOTE: all counts are cumulative and reset only after a reload.
Loopback0

All statistics for this interface are zero.
Interface NVI0 is disabled

Vlan1 LAN
Throttle count 0
Drops RP 524 SP 0
SPD Flushes Fast 223085 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 3601935 Drops 0

Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 16174521 1435194246 9747182 1731922240
Cache misses 0 - - -
Fast 550018749 3067325560 600924038 2699446987
Auton/SSE 0 0 0 0

Protocol DEC MOP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 430 33110 509 39193
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

Protocol Spanning Tree
Switching path Pkts In Chars In Pkts Out Chars Out
Process 74 6312 0 0
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 2325262 139515732 1980047 118802820
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 18445 4996035 0 0
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0

I do have CEF enabled.

interface GigabitEthernet0/0
ip address xxx.xxx.xxx.xxx xxx.xxx.xxx.xxx
no ip redirects
no ip unreachables
no ip proxy-arp
ip multicast boundary 30
ip nat outside
ip virtual-reassembly in
load-interval 30
duplex auto
speed auto
no cdp enable
no mop enabled

interface Vlan1
description LAN
ip address 192.168.200.1 255.255.252.0 secondary
ip flow ingress
ip flow egress
ip nat inside
ip virtual-reassembly in

ip nat inside source list NAT interface GigabitEthernet0/0 overload

ip access-list standard NAT
permit 192.168.0.0 0.0.255.255

Do I need to do anything with the Gigabit interfaces associated with the ethernet service module?

ASKER CERTIFIED SOLUTION

Joey Yung

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

SOLUTION

Predrag Jovic

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

David Bauman

ASKER

Sorry for the delay.

New deployment.

I won't be able to check this again for about a month unfortunately

David Bauman

ASKER

Thanks guys.

What happens when the limit is met? Does it deny any new translations? Delete the older ones? It seems that when I run these commands, it seriously lags DNS requests

SOLUTION

Predrag Jovic

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

David Bauman

ASKER

Ahh ok. So once this limit is reached, I still need to clear ip nat tran * for new requests to be processed.

Thanks!