High CPU on Cisco router 3925
Our router is suffering the anomaly traffic, and its CPU raised to 40-50% from last night (normally is is < 10%) Can you check out this? Is there any way to mitigate the impact ?
This is the result of show process:
This is the CPU chart from last night until now:
Please help! Are we under attack?
This is the result of show process:
R1-ILL#sh process
CPU utilization for five seconds: 49%/46%; one minute: 30%; five minutes: 18%
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
1 Cwe 246DD68 3378 65842 5111196/12000 0 Chunk Manager
2 Csp 8AA7A5 2274 2417984 0 5448/6000 0 Load Meter
3 M* 0 168 1074 15619592/24000578 SSH Process
4 Mwe 3F55D1F 0 1 031272/32000 0 EDDRI_MAIN
5 Mwe 24CCD29 0 1 011812/12000 0 RO Notify Timers
6 Lst 2488205 11132240 1755668 634011236/12000 0 Check heaps
7 Cwe 2483E02 476 6876 6911116/12000 0 Pool Manager
8 Mwe 2483CCD 0 1 011564/12000 0 DiscardQ Backgro
9 Mst 12EF583 0 2 011312/12000 0 Timers
10 Mwe 1346E54 4 1653 2 5716/6000 0 WATCH_AFS
11 Mwe 13FBFA 0 1 011572/12000 0 License Client N
12 Mwe 12DDA0 0 1 011488/12000 0 Image License br
13 Msi 1469C1 3973070 201434 19723 7460/12000 0 Licensing Auto U
14 Mwe 247DDA 0 1 023544/24000 0 OIR Handler
15 Mwe 460BC8A 4 908 411564/12000 0 CRYPTO IKMP IPC
16 Mwe 44E016 0 1 031440/32000 0 Crash writer
17 Mwe 44DE5C 0 1 011268/12000 0 Exception contro
18 Msi A32BD5 3233064 12085727 26711172/12000 0 Environmental mo
19 Mwe A6BAB1 660 2417191 011560/12000 0 IPC Event Notifi
20 Mwe A7E66C 570 201434 211744/12000 0 IPC Dynamic Cach
21 Mwe A84597 0 1 011588/12000 0 IPC Session Serv
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
22 Mwe A7DF92 0 1 011580/12000 0 IPC Zone Manager
23 Mwe A7E5B3 180 11805941 011572/12000 0 IPC Periodic Tim
24 Mwe A7EAF6 150 11805941 011536/12000 0 IPC Deferred Por
25 Mwe A7E539 0 1 011588/12000 0 IPC Process leve
26 Mwe A7E246 0 1 011340/12000 0 IPC Seat Manager
27 Mwe A7E135 0 1 011552/12000 0 IPC Seat RX Cont
28 Mwe A7E05D 0 1 011548/12000 0 IPC Seat TX Cont
29 Mwe A7DD85 32 1208996 011708/12000 0 IPC Keep Alive M
30 Hsi A78FC5 142 2417962 011708/12000 0 IPC Loadometer
31 Mwe FC6101 188260 4789526 3910420/12000 0 ARP Input
32 Mwe FC24E8 1628 12607145 010448/12000 0 ARP Background
33 Mwe FF83C7 0 2 011300/12000 0 ATM Idle Timer
34 Mwe FE724C 0 1 011536/12000 0 ATM ASYNC PROC
35 Lwe 11DDFC1 0 1 011540/12000 0 CEF MIB API
36 Lwe 12C8A89 0 1 011824/12000 0 AAA_SERVER_DEADT
37 Mwe 1320C8B 0 1 023480/24000 0 Policy Manager
38 Mwe 141F803 0 2 011260/12000 0 DDR Timers
39 Lwe 14C412D 0 3 011344/12000 0 Entity MIB API
40 Mwe 16B69D4 332 34425 914608/16000 0 EEM ED Syslog
41 Mst 1EAF753 9560 42 22761910192/12000 0 PrstVbl
42 Mwe 2247B8D 0 2 011288/12000 0 Serial Backgroun
43 Mwe 24D2F64 0 1 0 5540/6000 0 RMI RM Notify Wa
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
44 Mwe 29FE7F 0 2 011344/12000 0 SMART
45 Msp 137AC85 386 12089894 011352/12000 0 GraphIt
46 Mwe 1447C69 0 2 023300/24000 0 Dialer event
47 Mwe 225153F 0 1 011580/12000 0 SERIAL A'detect
48 Mwe 2688C1F 0 2 023344/24000 0 XML Proxy Client
49 Cwe 24A0542 0 1 011584/12000 0 Critical Bkgnd
50 Mwe 478E01 12480 1234958 1022816/24000 0 Net Background
51 Mwe 478CD2 0 9 023292/24000 0 IDB Work
52 Lwe 1396769 22 66014 022312/24000 0 Logger
53 Mwe 130F295 1000 12085729 010668/12000 0 TTY Background
54 Mwe 22A6408 0 3 011144/12000 0 IF-MGR control p
55 Mwe 22B05D1 2 20 10011580/12000 0 IF-MGR event pro
56 Mwe 2F5157 0 1 0 5580/6000 0 Inode Table Dest
57 Mwe ED3D60 0 2 011528/12000 0 cpf_msg_holdq_pr
58 Mwe ED3F3F 0 1 011544/12000 0 cpf_msg_rcvq_pro
59 Mwe F54A62 0 1 023572/24000 0 Crypto PKI-HA
60 Mwe 4352A3F 0 1 011564/12000 0 IKE HA Mgr
61 Mwe 4354AA3 0 1 011564/12000 0 IPSEC HA Mgr
62 Mwe 79EF32 6 106049 011300/12000 0 TDM Management
63 Mwe EC71D1 0 2 010600/12000 0 rf task
64 Hwe EC711F 0 1 011828/12000 0 RF High Priority
65 Hwe 485B05 4422 1055232 411460/12000 0 Net Input
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
66 Msp 47C9AE 380 2418164 011344/12000 0 Compute load avg
67 Msp 485D5A 10166 206419 4911136/12000 0 Per-minute Jobs
68 Msp 485DC2 48633312 12090716 402214696/16000 0 Per-Second Jobs
69 Mwe 3768C9 0 1 011808/12000 0 AggMgr Process
70 Mwe F49E2D 0 1 011584/12000 0 Token Daemon
71 Mst 1C6347E 16 945290 011168/12000 0 Transport Port A
72 Mwe 22BAC2C 9816 2417191 411552/12000 0 HC Counter Timer
73 Mwe 187A21 7740 805724 911224/12000 0 SM Monitor
74 Hwe 1994F4 0 2 011304/12000 0 Bryce I2C CMD Qu
75 Mwe 36489F 0 1 0 5584/6000 0 dev_device_inser
76 Mwe 3646CD 0 1 0 5584/6000 0 dev_device_remov
77 Mwe 3A1FDD 0 1 023584/24000 0 sal_dpc_process
78 Mwe 3A63B0 0 1 011564/12000 0 ARL Table Manage
79 Hwe 55D414 0 2 011344/12000 0 ESWPPM
80 Mwe 5678E0 0 2 011292/12000 0 Eswilp Storm Con
81 Hwe 57F5E1 0 2 011344/12000 0 ESWILPPM
82 Mwe 5A4DB4 0 2 0 5144/6000 0 Eswilp Storm Con
83 Hwe 649E33 0 2 011348/12000 0 DXMRVL
84 Mwe 25E533 588 12085730 011340/12000 0 UHCI Periodic Ta
85 Hwe A08FA2 6 3 200010776/12000 0 USB Startup
86 Mwe EDD382 0 1 011816/12000 0 RF_INTERDEV_DELA
87 Mwe EDED3C 0 1 031556/32000 0 RF_INTERDEV_SCTP
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
88 Msi A86AEB 272 12085740 011344/12000 0 Ether-Switch RBC
90 Hwe 80597C 0 1 011556/12000 0 IGMP Snooping Pr
91 Hwe 8054EE 0 1 011560/12000 0 IGMP Snooping Re
92 Msi AA0C68 486 402866 111292/12000 0 Call Management
93 Mwe ED9A3F 0 1 031564/32000 0 CF_INTERDEV_SCTP
94 Msp 7A1D42 34398 48358765 011228/12000 0 Netclock Backgro
95 Hwe 5C5B23 15870 12091769 111320/12000 0 BPSM stat Proces
96 Lsi 5753AC 1130 20140772 011004/12000 0 ILPM
97 Hwe 302C8F 0 2 023344/24000 0 Ethernet CFM
98 Hwe 302BE0 694 64619120 023340/24000 0 Ethernet Timer C
99 Hwe 302B3C 13102 1524021985 023340/24000 0 Ethernet Msec Ti
100 Mwe 4E99E4 0 2 011340/12000 0 Dot1x Mgr Proces
101 Mwe 4FADAC 0 1 011872/12000 0 MAB Framework
102 Mwe 513244 0 1 011848/12000 0 EAP Framework
103 Mwe 549409 0 2 011240/12000 0 DTP Protocol
104 Msi 7AB686 386 12085732 011124/12000 0 PI MATM Aging Pr
105 Msi 862482 558 1208595 011356/12000 0 EtherChnl
106 Lwe ABA3BC 0 2 011308/12000 0 call_home_les_oi
107 Mwe D95F59 0 1 011760/12000 0 IPv6 ping proces
108 Mwe F9E942 17612 3482047 511176/12000 0 AAA Server
109 Mwe F9AC52 0 1 011856/12000 0 AAA ACCT Proc
110 Mwe F9ABCA 0 1 011276/12000 0 ACCT Periodic Pr
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
111 Mwe FB02D4 0 1 011556/12000 0 AAA System Acct
112 Mwe 11D3A32 41064 2159212 1910904/12000 0 CDP Protocol
113 Mwe 12A0F6A 0 2 011336/12000 0 AAA Dictionary R
114 Mwe 14DE194 0 2 023348/24000 0 Ethernet LMI
115 Mwe 14F09BD 0 2 015348/16000 0 Ethernet OAM Pro
116 Lwe 159D289 0 2 023336/24000 0 CEF switching ba
117 Lwe 159D289 0 1 0 5336/6000 0 ADJ NSF process
118 Mwe 159D289 13120 733477 1722060/24000 0 ADJ resolve proc
119 Hwe 100C1DF 0 2 023268/24000 0 ATM OAM Input
120 Hwe 10078E4 0 2 023292/24000 0 ATM OAM TIMER
121 Mwe 24B3C22 0 2 011324/12000 0 IPAM/ODAP Events
122 Mwe 18CF67B 4916 377316050 023252/24000 0 IPAM Manager
123 Mwe 24B3C22 0 2 023328/24000 0 IPAM Events
124 Mwe 1931B25 2 8 250 8560/12000 0 IP ARP Adjacency
125 Mwe 19335E8 2 1 200011316/12000 0 IP ARP Retry Age
126 Mwe 190751D 494648436 4056992487 12120148/24000 0 IP Input
127 Mwe 1938E45 0 1 011776/12000 0 ICMP event handl
128 Mwe 19D8A88 0 3 011244/12000 0 PIM register asy
129 Mwe 1CCA332 2 20104 011328/12000 0 MOP Protocols
130 Mwe 1E5330C 0 2 011260/12000 0 PPP SIP
131 Mwe 24B3C22 0 2 011336/12000 0 PPP Bind
132 Mwe 24B3C22 0 2 023340/24000 0 PPP IP Route
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
134 Mwe 22E0AE4 0 1 023908/24000 0 SSS Manager
135 Mwe 22EC05C 0 1 023860/24000 0 SSS Policy Manag
136 Mwe 22CEDC3 0 1 011540/12000 0 SSS Feature Mana
137 Mwe 22CECB5 894 47225548 011684/12000 0 SSS Feature Time
138 Mwe 2371911 45920 6032180 723348/24000 0 Spanning Tree
139 Mwe 25B5FA5 0 2 023288/24000 0 SSM connection m
140 Lwe 263A76C 0 1 011216/12000 0 X.25 Encaps Mana
141 Mwe 435925D 2 40300 011588/12000 0 Authentication P
142 Mwe 4367125 0 1 011800/12000 0 Auth-proxy AAA B
143 Hwe 43759D3 0 2 015320/16000 0 EAPoUDP Process
144 Mwe 437E22A 0 2 015276/16000 0 IP Host Track Pr
145 Mwe 45A3336 0 2 023344/24000 0 KRB5 AAA
146 Lwe 159D289 1336 241743 522276/24000 0 CEF background p
147 Hwe 159D289 0 1 023336/24000 0 fib_fib_bfd_sb e
148 Hwe 1C1B199 0 1 011824/12000 0 Socket Timers
149 Mwe 50B0A3 0 2 011276/12000 0 Dot1x Supplicant
150 Mwe 50F72B 0 2 011268/12000 0 Dot1x Supplicant
151 Mwe 50AE77 0 2 011300/12000 0 Dot1x Supplicant
152 Mwe 690080 4 2 2000 9856/12000 0 L2MM
153 Mwe 697D9A 0 1 011512/12000 0 MRD
154 Mwe 686810 0 1 011524/12000 0 IGMPSN
155 Mwe 24B3C22 0 2 0 5340/6000 0 L2X Switching Ev
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
156 Mwe A8557A 0 17 011148/12000 0 IPC UDP Input
157 Lwe 1609442 5848 15744342 021800/24000 0 CEF: IPv4 proces
158 Lwe 159D289 0 5 011336/12000 0 ADJ background
159 Mwe 1B90D42 1406 201451 614176/16000 0 IP Background
160 Mwe 1BEDAE0 0 33 014408/16000 0 IP Connected Rou
161 Mwe 1BDFE70 0 36 020712/24000 0 IP RIB Update
162 Mwe 1958F4C 0 1 011732/12000 0 IP Traceroute
163 Mwe 11A07CC 1638 201434 811368/12000 0 Call Home Timer
164 Lwe 159D289 0 6 023140/24000 0 Collection proce
165 Mwe 21F28F4 0 5 010884/12000 0 SCTP Main Proces
166 Mwe 1C4AF94 396998 7469180 5321092/24000 0 TCP Timer
167 Lwe 1C507C8 140980 2727256 5121012/24000 0 TCP Protocols
168 Mwe 1869502 0 40301 015248/16000 0 HTTP CORE
169 Mwe 228D78B 0 2 011748/12000 0 SNMP Timers
170 Mwe 1132785 0 1 011568/12000 0 IUA Main Process
171 Mwe 21E5DB1 750 12085741 011356/12000 0 RUDPV1 Main Proc
172 Mwe 113B19A 0 1 011776/12000 0 bsm_timers
173 Msi 113800B 334 12085745 011712/12000 0 bsm_xmt_proc
174 Hwe 24B3C22 0 2 011336/12000 0 PPP Compress Inp
175 Hwe 24B3C22 0 2 011336/12000 0 PPP Compress Res
176 Mwe 125F06F 0 1 059256/60000 0 COPS
177 Mwe 144350D 0 2 011296/12000 0 Dialer Forwarder
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
178 Mwe 172D53B 0 3 011304/12000 0 Flow Exporter Ti
179 Lwe 159D289 0 3 023336/24000 0 MFIB Master back
180 Mwe 1AA4DB3 0 2 015296/16000 0 Multicast Offloa
181 Mwe 1B84BB3 0 1 011580/12000 0 RARP Input
182 Mwe 1BF98CE 0 30 014540/16000 0 static
183 Mwe 1C6ECC3 0 1 015768/16000 0 IP IRDP
184 Mwe 1C7E0E8 0 1 011908/12000 0 LAPB Process
185 Hwe 1E05B10 0 1 011500/12000 0 PAD InCall
186 Mwe 2646B49 0 2 023308/24000 0 X.25 Background
187 Hwe 1FA9A60 0 1 011580/12000 0 MQC Flow Event B
188 Mwe 2050EFC 0 2 011804/12000 0 HQF Shaper Backg
189 Mwe 2530790 0 1 023732/24000 0 VPDN call manage
190 Mwe 2A64846 0 2 011312/12000 0 PPP NBF
191 Mwe 43804C8 0 2 011072/12000 0 SDEE Management
192 Mwe 43A33ED 302 23610386 011324/12000 0 Inspect process
194 Mwe 4483C46 70 779537 011344/12000 0 FW DP Inspect pr
195 Mwe 44CF53E 182 23610386 011336/12000 0 CCE DP URLF cach
196 Mwe 45BBD77 0 2 011320/12000 0 URL filter proc
197 Mwe 47A4F48 0 1 023512/24000 0 IPS Process
198 Mwe 4812CC0 0 2 023284/24000 0 IPS Auto Update
199 Hwe 48F548A 0 1 011828/12000 0 Select Timers
200 Mwe 48B023F 26 2 1300011040/12000 0 HTTP Process
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
201 Mwe 48E5542 0 2 011180/12000 0 CIFS API Process
202 Mwe 48E4F9A 0 2 011300/12000 0 CIFS Proxy Proce
203 Mwe 4946143 0 3 010856/12000 0 Crypto HW Proc
204 Mwe 4979F2D 0 1 011824/12000 0 IPv6 Inspect Tim
205 Mwe CD5F80 28 483599 011096/12000 0 CRM_CALL_UPDATE_
207 Mwe 118203A 0 2 011348/12000 0 AAA Cached Serve
208 Mwe 14591CC 0 2 011344/12000 0 ENABLE AAA
209 Mwe 14F4F88 0 1 011564/12000 0 EM Background Pr
210 Mwe 188F916 0 1 011904/12000 0 IDMGR CORE
211 Mwe 1C70F8A 0 1 011820/12000 0 Key chain liveke
212 Mwe 1C9315C 0 2 011348/12000 0 LINE AAA
213 Mwe 1CA24C6 26954 3716595 710936/12000 0 LOCAL AAA
214 Mwe 241F810 2 2 100010936/12000 0 TPLUS
215 Mwe 288A572 0 3 011220/12000 0 LDAP process
216 Mwe 2FD4A27 0 3 012952/14000 0 Crypto WUI
217 Mwe 44F00B7 0 1 030244/32000 0 FW_TEST_TRP
218 Mwe 46B8CE3 12690 1084176 1111012/12000 0 Crypto Support
219 Mwe 483548E 0 1 011868/12000 0 EPM MAIN PROCESS
220 Mwe 496F07F 0 1 011576/12000 0 IPSECv6 PS Proc
221 Lwe 27F92B2 0 170 011324/12000 0 crypto engine pr
222 Mwe 26F8150 0 4 022444/24000 0 Crypto CA
223 Mwe 26F747F 0 1 023576/24000 0 Crypto PKI-CRL
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
224 Mwe 281BA8B 0 1 023576/24000 0 encrypt proc
225 Lwe 281C283 10192 814792 1210136/12000 0 crypto sw pk pro
226 Mwe 432A830 9146 12086091 020244/24000 0 Crypto IKEv2
227 Mwe 432BAAE 0 1 011540/12000 0 IKEv2 AAA handle
228 Mwe 466D251 0 1 011532/12000 0 Crypto INT
229 Mwe 4656C74 28 865 3222572/24000 0 Crypto IKE Dispa
230 Mwe 46402FC 304 2012 15120268/24000 0 Crypto IKMP
231 Mwe 46CEA6C 0 1 011868/12000 0 IPSEC manual key
232 Mwe 467E501 1336 604621 221516/24000 0 IPSEC key engine
233 Mwe 46ABDAA 0 2 011792/12000 0 CRYPTO QoS proce
234 Mwe 45DDE13 4 383 1029924/32000 0 Crypto ACL
235 Mwe 45DBE1D 0 1 011576/12000 0 Crypto PAS Proc
236 Mwe 46E982E 0 1 011776/12000 0 GDOI GM Process
237 Mwe 471A6F4 0 1 011792/12000 0 UNICAST REKEY
238 Mwe 471A663 0 1 011580/12000 0 UNICAST REKEY AC
239 Msp 27EC052 274 12089911 015712/16000 0 Crypto Device Up
240 Mwe 494D374 0 2 011452/12000 0 Multi-ISA Event
241 Mwe 494D0F6 0 1 011580/12000 0 Multi-ISA Cleanu
242 Lwe 16369E 0 1 015584/16000 0 Licensing MIB pr
243 Mwe 4DA023 0 2 011228/12000 0 Control-plane ho
244 Mwe 839B33 0 1 011564/12000 0 PM Callback
246 Mwe 127E6A6 4882 790467 611180/12000 0 AAA SEND STOP EV
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
247 Mwe 1287428 0 1 023836/24000 0 Test AAA Client
248 Mwe 169632D 0 2 014948/16000 0 EEM ED Track
249 Mwe 169632D 0 3 014944/16000 0 EEM ED Resource
250 Mwe 16C08EB 0 2 014984/16000 0 EEM ED Routing
251 Msi 211206E 634 1208595 011352/12000 0 RMON Recycle Pro
252 Mwe 211CB43 0 2 011340/12000 0 RMON Deferred Se
253 Mwe 2384F59 0 1 011588/12000 0 Syslog Traps
254 Mwe 468CE53 78 377810 011404/12000 0 Crypto cTCP proc
256 Mwe 8699AE 2 2 1000 8768/12000 0 VLAN Manager
257 Mwe 16D60C 0 1 031076/32000 0 LICENSE AGENT
258 Mwe 164DF47 4 566 710680/12000 0 EEM Server
259 Mwe 11A028E 0 2 010724/12000 0 Call Home proces
260 Mwe 167D108 0 2 011152/12000 0 EEM Policy Direc
261 Mwe 169632D 0 2 014940/16000 0 EEM ED CLI
262 Mwe 169632D 0 3 014944/16000 0 EEM ED Counter
263 Mwe 169632D 0 3 014948/16000 0 EEM ED Interface
264 Mwe 169632D 0 3 014944/16000 0 EEM ED IOSWD
265 Mwe 169632D 0 3 014948/16000 0 EEM ED None
266 Mwe 169632D 0 3 014944/16000 0 EEM ED OIR
267 Mwe 16A7DEF 0 3 014964/16000 0 EEM ED RF
268 Mwe 169632D 0 3 014960/16000 0 EEM ED SNMP
269 Mwe 16B38F8 0 2 014992/16000 0 EEM ED SNMP Noti
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
270 Mwe 169632D 48 302447 014492/16000 0 EEM ED Timer
271 Mwe 169632D 0 3 014944/16000 0 EEM ED Test
272 Mwe 169632D 0 3 014944/16000 0 EEM ED Config
273 Mwe 169632D 0 3 014948/16000 0 EEM ED Env
274 Mwe 169632D 0 3 014948/16000 0 EM ED GOLD
275 Mwe 18E682E 26 10728 223144/24000 0 Syslog
276 Mwe 169632D 0 3 014948/16000 0 EEM ED RPC
277 Mwe 169632D 0 3 014944/16000 0 EEM ED Ipsla
278 Mwe 2C0ED90 0 1 011788/12000 0 IP SLAs Ethernet
279 Mwe 2D131C7 0 2 014992/16000 0 EEM ED Nf
280 Mwe 19A867C 0 2 023244/24000 0 MRIB Process
281 Hwe A08FA2 0 1 022976/24000 0 tHUB
282 Mwe 49459BA 0 2 013788/14000 0 Key Proc
283 Mwe 2FC4E1 0 1 0 5356/6000 0 Async write proc
284 Mwe 2FC4E1 0 1 0 5356/6000 0 Async write proc
285 Mwe 2FC4E1 0 1 0 5360/6000 0 Async write proc
286 Mwe 2FC4E1 0 1 0 5360/6000 0 Async write proc
287 Msi 2F5673 4592 201434 22 5684/6000 0 DFS flush period
289 Mwe 28B8AA5 0 3361 010916/12000 0 SSH Event handle
290 Mwe 1B041FF 5023288 23778261 21110300/12000 0 IP NAT Ager
291 Mwe 1B66E87 0 1 011808/12000 0 IP NAT WLAN
292 Mwe 1F3F128 7617202 1129008332 611652/12000 0 IP VFR proc
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
293 Lwe 17057B9 36 2010 1711224/12000 0 IP Flow Top Talk
294 Mwe 18E3766 53580 2992728 1721060/24000 0 IP SNMP
295 Lwe 2293029 31426 2685860 1122788/24000 0 PDU DISPATCHER
296 Lwe 2292C47 130586 2685802 4821716/24000 0 SNMP ENGINE
297 Mwe 18E6100 0 2 023252/24000 0 IP SNMPV6
298 Lwe 12576D0 0 1 023584/24000 0 SNMP ConfCopyPro
299 Mwe 228E817 0 2 023308/24000 0 SNMP Traps
300 Mwe ED3BFE 0 1 011560/12000 0 cpf_process_tpQThis is the CPU chart from last night until now:
Please help! Are we under attack?
ASKER
Hi, No the number of VPN server is the same.
Strangely, I saw the IP addresses when issued command "show ip flow top-talker" and "show ip cache flow" but when issued "show ip nat translation | grep IP_address", it did not show anything. I intent to get the ports which those traffic connect to and block them by ACL.
Last night, I configured rate-limit in outbound interface, the CPU reduces to 20-25% but I know it is just mitigate and did not solve problem, we still got strange IPs connect to. Is it okay to issue this command on it:
rate-limit input 25600000 6400000 12800000 conform-action transmit exceed-action drop
Out maximum rate is 64 Mb/s
Please find the result below (I removed rate-limit command before issue these):
Strangely, I saw the IP addresses when issued command "show ip flow top-talker" and "show ip cache flow" but when issued "show ip nat translation | grep IP_address", it did not show anything. I intent to get the ports which those traffic connect to and block them by ACL.
Last night, I configured rate-limit in outbound interface, the CPU reduces to 20-25% but I know it is just mitigate and did not solve problem, we still got strange IPs connect to. Is it okay to issue this command on it:
rate-limit input 25600000 6400000 12800000 conform-action transmit exceed-action drop
Out maximum rate is 64 Mb/s
Please find the result below (I removed rate-limit command before issue these):
R1-ILL#sh process cpu sorted | e 0.00%.*0.00%.*0.00%
CPU utilization for five seconds: 22%/21%; one minute: 20%; five minutes: 21%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
13 3999968 202544 19748 0.39% 0.04% 0.00% 0 Licensing Auto U
68 48927496 12157339 4024 0.27% 0.34% 0.35% 0 Per-Second Jobs
3 108 593 182 0.03% 0.03% 0.02% 578 SSH Process
126 494788548 4060655352 0 0.03% 0.10% 0.11% 0 IP Input
6 11257852 1767157 6370 0.00% 0.11% 0.11% 0 Check heaps
18 3251600 12152323 267 0.00% 0.02% 0.00% 0 Environmental moR1-LL#show ip interface | e [0-9][0-9]\.[0-9][0-9]
GigabitEthernet0/0 is up, line protocol is up
Address determined by non-volatile memory
MTU is 1500 bytes
Helper address is not set
Directed broadcast forwarding is disabled
Outgoing access list is SECURITY-OUT
Inbound access list is SECURITY-IN
Proxy ARP is enabled
Local Proxy ARP is disabled
Security level is default
Split horizon is enabled
ICMP redirects are always sent
ICMP unreachables are always sent
ICMP mask replies are never sent
IP fast switching is enabled
IP fast switching on the same interface is disabled
IP Flow switching is disabled
IP CEF switching is enabled
IP CEF switching turbo vector
IP multicast fast switching is enabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast, CEF
Router Discovery is disabled
IP output packet accounting is disabled
IP access violation accounting is disabled
TCP/IP header compression is disabled
RTP/IP header compression is disabled
Policy routing is disabled
Network address translation is enabled, interface in domain outside
BGP Policy Mapping is disabled
Input features: Stateful Inspection, Ingress-NetFlow, Virtual Fragment Reassembly, Access List, IPSec input classification, Virtual Fragment Reassembly After IPSec Decryption, NAT Outside, MCI Check
Output features: CCE Output Classification, Post-routing NAT Outside, Stateful Inspection, IPSec output classification, IPsec or interface ACL checked on pre-encrypted cleartext packets, Post-Ingress-NetFlow, IPSec: to crypto engine, Post-encryption output features
WCCP Redirect outbound is disabled
WCCP Redirect inbound is disabled
WCCP Redirect exclude is disabled
GigabitEthernet0/1 is administratively down, line protocol is down
Internet protocol processing disabled
GigabitEthernet0/2 is up, line protocol is up
Address determined by non-volatile memory
MTU is 1500 bytes
Helper address is not set
Directed broadcast forwarding is disabled
Outgoing access list is not set
Inbound access list is not set
Proxy ARP is enabled
Local Proxy ARP is disabled
Security level is default
Split horizon is enabled
ICMP redirects are always sent
ICMP unreachables are always sent
ICMP mask replies are never sent
IP fast switching is enabled
IP fast switching on the same interface is disabled
IP Flow switching is disabled
IP CEF switching is enabled
IP CEF switching turbo vector
IP multicast fast switching is enabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast, CEF
Router Discovery is disabled
IP output packet accounting is disabled
IP access violation accounting is disabled
TCP/IP header compression is disabled
RTP/IP header compression is disabled
Policy routing is enabled, using route map test2
Network address translation is enabled, interface in domain inside
BGP Policy Mapping is disabled
Input features: Stateful Inspection, Virtual Fragment Reassembly, Virtual Fragment Reassembly After IPSec Decryption, Policy Routing, MCI Check
Output features: NAT Inside, Stateful Inspection, Post-Ingress-NetFlow
WCCP Redirect outbound is disabled
WCCP Redirect inbound is disabled
WCCP Redirect exclude is disabled
GigabitEthernet0/3 is up, line protocol is up
Address determined by setup command
MTU is 1500 bytes
Helper address is not set
Directed broadcast forwarding is disabled
Outgoing access list is SECURITY-OUT
Inbound access list is SECURITY-IN
Proxy ARP is enabled
Local Proxy ARP is disabled
Security level is default
Split horizon is enabled
ICMP redirects are always sent
ICMP unreachables are always sent
ICMP mask replies are never sent
IP fast switching is enabled
IP fast switching on the same interface is disabled
IP Flow switching is disabled
IP CEF switching is enabled
IP CEF switching turbo vector
IP multicast fast switching is enabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast, CEF
Router Discovery is disabled
IP output packet accounting is disabled
IP access violation accounting is disabled
TCP/IP header compression is disabled
RTP/IP header compression is disabled
Policy routing is disabled
Network address translation is enabled, interface in domain outside
BGP Policy Mapping is disabled
Input features: Stateful Inspection, Virtual Fragment Reassembly, Access List, Virtual Fragment Reassembly After IPSec Decryption, NAT Outside, MCI Check
Output features: CCE Output Classification, Post-routing NAT Outside, Stateful Inspection, IPsec or interface ACL checked on pre-encrypted cleartext packets, Post-Ingress-NetFlow
WCCP Redirect outbound is disabled
WCCP Redirect inbound is disabled
WCCP Redirect exclude is disabled
GigabitEthernet1/0 is up, line protocol is up
Address determined by non-volatile memory
MTU is 1500 bytes
Helper address is not set
Directed broadcast forwarding is disabled
Outgoing access list is not set
Inbound access list is not set
Proxy ARP is enabled
Local Proxy ARP is disabled
Security level is default
Split horizon is enabled
ICMP redirects are always sent
ICMP unreachables are always sent
ICMP mask replies are never sent
IP fast switching is enabled
IP fast switching on the same interface is disabled
IP Flow switching is disabled
IP CEF switching is enabled
IP CEF switching turbo vector
IP multicast fast switching is enabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast, CEF
Router Discovery is disabled
IP output packet accounting is disabled
IP access violation accounting is disabled
TCP/IP header compression is disabled
RTP/IP header compression is disabled
Policy routing is disabled
Network address translation is disabled
BGP Policy Mapping is disabled
Input features: MCI Check
Output features: Post-Ingress-NetFlow
WCCP Redirect outbound is disabled
WCCP Redirect inbound is disabled
WCCP Redirect exclude is disabled
NVI0 is up, line protocol is up
MTU is 1514 bytes
Helper address is not set
Directed broadcast forwarding is disabled
Outgoing access list is not set
Inbound access list is not set
Proxy ARP is enabled
Local Proxy ARP is disabled
Security level is default
Split horizon is enabled
ICMP redirects are always sent
ICMP unreachables are always sent
ICMP mask replies are never sent
IP fast switching is disabled
IP fast switching on the same interface is disabled
IP Flow switching is disabled
IP CEF switching is disabled
IP Null turbo vector
IP Null turbo vector
IP multicast fast switching is enabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast, CEF
Router Discovery is disabled
IP output packet accounting is disabled
IP access violation accounting is disabled
TCP/IP header compression is disabled
RTP/IP header compression is disabled
Policy routing is disabled
Network address translation is disabled
BGP Policy Mapping is disabled
Input features: MCI Check
Output features: Post-routing NAT NVI Output, Post-Ingress-NetFlow
WCCP Redirect outbound is disabled
WCCP Redirect inbound is disabled
WCCP Redirect exclude is disabled
It could be some bug related to Licensing Auto Update Process - like CSCuj27424 I can't really find something useful regarding "Per-Second Jobs".
Which IOS version is currently running?
sh ver | i IOS
Try to issue
show interface | e [0-9][0-9]\.[0-9][0-9]
and check device logs
Since NVI is in use according to above output instead of show ip nat translation | grep IP_address I guess grep if grep command is supported on 3925 try to use
show ip nat nvi translation
But, since rate limiting helps it could be attack, increased traffic rate or still could be some bug.
Which IOS version is currently running?
sh ver | i IOS
Try to issue
show interface | e [0-9][0-9]\.[0-9][0-9]
and check device logs
Since NVI is in use according to above output instead of show ip nat translation | grep IP_address I guess grep if grep command is supported on 3925 try to use
show ip nat nvi translation
But, since rate limiting helps it could be attack, increased traffic rate or still could be some bug.
ASKER
Hi, I mean include, not grep (Linux-in my mind :-)
nothing show up with this
here is our version:
Cisco IOS Software, C3900e Software (C3900e-UNIVERSALK9-M), Version 15.1(1)T1, RELEASE SOFTWARE (fc2)
and the output:
and the logging
Should I increase or decrease the rate-limit?
nothing show up with this
R1-LL#show ip nat nvi translation
Pro Source global Source local Destin local Destin globalhere is our version:
Cisco IOS Software, C3900e Software (C3900e-UNIVERSALK9-M), Version 15.1(1)T1, RELEASE SOFTWARE (fc2)
and the output:
R1-LL
#show interface | e [0-9][0-9]\.[0-9][0-9]
GigabitEthernet0/0 is up, line protocol is up
Hardware is iGbE, address is 1cdf.0fdd.0680 (bia 1cdf.0fdd.0680)
Description: "ket noi 13 Netnam"
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 29/255, rxload 68/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 100Mbps, media type is RJ45
output flow-control is XON, input flow-control is XON
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/238/4104955 (size/max/drops/flushes); Total output drops: 23 97420
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 26780000 bits/sec, 9262 packets/sec
5 minute output rate 11693000 bits/sec, 8513 packets/sec
2721983398 packets input, 3162887304 bytes, 866 no buffer
Received 1355928 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 1206067 multicast, 3 pause input
2801480864 packets output, 1894980946 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
657693 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
2 lost carrier, 0 no carrier, 25400 pause output
0 output buffer failures, 0 output buffers swapped out
GigabitEthernet0/1 is administratively down, line protocol is down
Hardware is iGbE, address is 1cdf.0fdd.0681 (bia 1cdf.0fdd.0681)
Description: "Connect to R2"
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Auto Duplex, Auto Speed, media type is unknown media type
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
1 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
GigabitEthernet0/2 is up, line protocol is up
Hardware is iGbE, address is 1cdf.0fdd.0682 (bia 1cdf.0fdd.0682)
Description: Connect to SW1-3750
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 5/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1Gbps, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/1398653 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 6910000 bits/sec, 2035 packets/sec
5 minute output rate 20215000 bits/sec, 2501 packets/sec
1354166308 packets input, 1467838415 bytes, 113 no buffer
Received 204455 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
39799 input errors, 0 CRC, 0 frame, 39799 overrun, 0 ignored
0 watchdog, 202901 multicast, 0 pause input
352354030 packets output, 2155461732 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
1 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
GigabitEthernet0/3 is up, line protocol is up
Hardware is iGbE, address is 1cdf.0fdd.0683 (bia 1cdf.0fdd.0683)
Description: LLVTN-VNPT
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 100Mbps, media type is RJ45
output flow-control is XON, input flow-control is XON
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:01, output 00:00:03, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/2298 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 3000 bits/sec, 4 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
52099082 packets input, 1613442077 bytes, 0 no buffer
Received 4708431 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
10165446 packets output, 108217256 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
6 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
GigabitEthernet1/0 is up, line protocol is up
Hardware is PSE2, address is 1cdf.0fdd.0688 (bia 1cdf.0fdd.0688)
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1Gbps, media type is internal
output flow-control is XON, input flow-control is XON
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:23, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/404 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
10332755 packets input, 1202288632 bytes, 0 no buffer
Received 8710220 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
587945 packets output, 107532035 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
405715 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
NVI0 is up, line protocol is up
Hardware is NVI
MTU 1514 bytes, BW 56 Kbit/sec, DLY 5000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation UNKNOWN, loopback not set
Keepalive set (10 sec)
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped outand the logging
*Apr 12 00:08:04.776: SSH2 1: Invalid modulus length
*Apr 12 00:10:28.576: SSH2 0: Invalid modulus length
*Apr 12 00:13:53.182: SSH2 1: Invalid modulus length
*Apr 12 00:14:08.956: SSH2 0: Invalid modulus length
*Apr 12 00:16:37.228: SSH2 0: Invalid modulus length
*Apr 12 00:28:09.816: SSH2 0: Invalid modulus length
*Apr 12 00:30:05.890: %IP-3-LOOPPAK: Looping packet detected and dropped -
src=210.86.225.14, dst=192.168.77.66, hl=20, tl=56, prot=1, sport=0, dport=0
in=GigabitEthernet0/0, nexthop=172.16.2.27, out=GigabitEthernet0/2
options=none -Process= "IP Input", ipl= 0, pid= 126 -Traceback= 0x19214B0z 0x192115Fz 0x192111Dz 0x1920C94z 0x1920588z 0x191EE13z 0x190657Az 0x1906187z 0x19082B4z 0x1907A7Bz 0x190777Dz 0x190758Cz
*Apr 12 00:30:51.700: SSH2 1: Invalid modulus length
*Apr 12 00:33:19.576: %IP-3-LOOPPAK: Looping packet detected and dropped -
src=210.86.225.14, dst=192.168.77.44, hl=20, tl=56, prot=1, sport=0, dport=0
in=GigabitEthernet0/0, nexthop=172.16.2.27, out=GigabitEthernet0/2
options=none -Process= "IP Input", ipl= 0, pid= 126 -Traceback= 0x19214B0z 0x192115Fz 0x192111Dz 0x1920C94z 0x1920588z 0x191EE13z 0x190657Az 0x1906187z 0x19082B4z 0x1907A7Bz 0x190777Dz 0x190758Cz
*Apr 12 00:39:44.940: SSH2 0: Invalid modulus length
*Apr 12 00:39:45.506: SSH2 0: no matching cipher found: client aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,arcfour256,arcfour128 server a
*Apr 12 00:46:18.000: SSH2 1: Invalid modulus length
*Apr 12 00:48:44.958: SSH2 1: Invalid modulus lengthShould I increase or decrease the rate-limit?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ACL especially long ones can have that effect, but, I guess in that case there was configuration change and after that CPU can increase (I asked were some changes applied) . If ACL was previously configured and router's CPU utilization was low, I would still guess that ACL itself may not be the only reason higher CPU utilization.
ASKER
problem was solved after recreated the ACL
Is anything reconfigured or something other (e.g more VPN users) happen on device at the time when CPU utilization started to be increased?
Please issue commands and paste output:
show processes cpu sorted | e 0.00%.*0.00%.*0.00%
show ip interface | e [0-9][0-9]\.[0-9][0-9]