Cisco 2960-S stack high CPU
Posted on 2011-03-09
Company has been experiencing poor performance and while checking switches I came across the following:
CPU UTILIZATION ANALYSIS
INFO: Total CPU Utilization is comprised of process and interrupt percentages.
Total CPU Utilization: 89%
Process Utilization: 62%
Interrupt Utilization: 27%
These values are found on the first line of the output:
CPU utilization for five seconds: x%/y%; one minute: a%; five minutes: b%
Total CPU Utilization: x%
Process Utilization: (x - y)%
Interrupt Utilization: y%
Process Utilization is the difference between the Total and Interrupt; x minus
y. The one and five minute utilizations are exponentially decayed averages (rather
than an arithmetic average), therefore recent values have more influence on the
WARNING: Total CPU Utilization is 89% for the past 5 seconds, which is high
This can cause the following symptoms:
- Input queue drops
- Slow performance
- Slow response in Telnet or unable to Telnet to the router
- Slow response on the console
- Slow or no response to ping
- Router doesn't send routing updates
The following processes are causing excessive CPU usage:
PID CPU Time Process
132 15.19 Hulc LED Process
297 35.28 Virtual Exec
INFO: The 'virtual EXEC process' handles virtual type terminal (vty) lines, such as Telnet sessions on the router.
INFO: Issuing commands that generate long output, such as 'show tech-support', can also increase the amount of CPU resources used by the virtual EXEC process.
INFO: Processor memory utilization is 41.80657%.
INFO: Processor memory or main memory stores the running configuration and routing
tables. The Cisco IOS software executes from main memory.
INFO: The amount of processor memory required by the router is affected by the
Cisco IOS version used, the size of the network and by the access list configurations.
Ensure that an optimal IOS version has been chosen.
WARNING: The following processes are currently holding more than 1 MB of memory
with 'Freed' memory less that 'Holding' memory:
'Stack Mgr Notifi' (Holding 1892896 bytes)
This is considered to be high. High memory usage can be normal, or can indicate
a memory leak. To determine this, it is important to see if memory usage changes
INFO: A memory leak occurs when a process requests or allocates memory and then
forgets to free (de-allocate) the memory when it is finished with that task.
As a result, the memory block is reserved until the router is reloaded. Over
time, more and more memory blocks are allocated by that process until there is
no free memory available. At this point, the router will issue the 'SYS-2-MALLOCFAIL'
TRY THIS: Analyze the 'show process memory' output for this router over a period
of time (for example, every few hours or days depending on whether you have a
fast or slow leak).
Check to see if memory utilization for the affected process(es) continues to increase
and the amount of freed memory remains the same. The rate at which free memory
disappears depends on how often the event occurs that leads to the leak. A memory
leak is a complex condition sometimes requiring an IOS upgrade to correct. If
the above is in fact occurring, and you are uncertain about how to proceed, use
the Service Request Tool to contact the Cisco TAC for further assistance.
NOTE: It is normal for some processes, such as Border Gateway Protocol (BGP) or
Open Shortest Path First (OSPF), to use more than one megabyte of memory; this
does not mean they are leaking.
WARNING: The process '*Dead*' is holding more than 1 MB of memory.
This is considered to be high, and indicates a possible memory leak. The 'Dead'
process is not a real process. It only accounts for the memory allocated under
the context of another process which is terminated.
TRY THIS: If a memory leak is detected, and the 'Dead' process seems to be consuming
the memory, issue the "show memory dead" command from your device, and analyze
which process consumes more memory. To do so, look at the "What" section of
the output. This problem can also be caused due to a Cisco IOS Bug. Review the
Bug Tool Kit for potential bugs and upgrade the device with the latest version
of Cisco IOS software. If the problem persists, open a service request with Cisco
at Service Request Tool.
INFO: The top 3 processes that are holding less than 1 MB of memory are:
'hulc running con' is holding 578860 bytes
'HL2MCM' is holding 295164 bytes
'HL2MCM' is holding 295012 bytes
IOS is c2960s-universalk9-mz.122-53.SE1
I used BUG toolkit and did not see any relation
I have made sure no debugs are running
INFO: This device has 131072 K available for main memory
INFO: This device has a total of 128 MB of RAM installed.
Not sure what to do about this. Please advise.