Solved

Cisco 2960-S stack high CPU

Posted on 2011-03-09
9
5,368 Views
Last Modified: 2012-05-11
Hello

Company has been experiencing poor performance and while checking switches I came across the following:

CPU UTILIZATION ANALYSIS
------------------------
  INFO: Total CPU Utilization is comprised of process and interrupt percentages.
    Total CPU Utilization: 89%
    Process Utilization: 62%
    Interrupt Utilization: 27%
  These values are found on the first line of the output:
   CPU utilization for five seconds: x%/y%; one minute: a%; five minutes: b%
    Total CPU Utilization: x%
    Process Utilization: (x - y)%
    Interrupt Utilization: y%
  Process Utilization is the difference between the Total and Interrupt; x minus
  y. The one and five minute utilizations are exponentially decayed averages (rather
  than an arithmetic average), therefore recent values have more influence on the
  calculated average.
 
  WARNING: Total CPU Utilization is 89% for the past 5 seconds, which is high
  (>80%).
  This can cause the following symptoms:
    - Input queue drops
    - Slow performance
    - Slow response in Telnet or unable to Telnet to the router
    - Slow response on the console
    - Slow or no response to ping
    - Router doesn't send routing updates
  The following processes are causing excessive CPU usage:
    PID      CPU Time      Process
   132      15.19            Hulc LED Process
   297      35.28            Virtual Exec
  INFO: The 'virtual EXEC process' handles virtual type terminal (vty) lines,   such as Telnet sessions on the router.
  INFO: Issuing commands that generate long output, such as 'show tech-support',  can also increase the amount of CPU resources used by the virtual EXEC process.
 
MEMORY ANALYSIS
---------------
  INFO: Processor memory utilization is 41.80657%.
  INFO: Processor memory or main memory stores the running configuration and routing
  tables. The Cisco IOS software executes from main memory.
  INFO: The amount of processor memory required by the router is affected by the
  Cisco IOS version used, the size of the network and by the access list configurations.
  Ensure that an optimal IOS version has been chosen.
 
  WARNING: The following processes are currently holding more than 1 MB of memory
  with 'Freed' memory less that 'Holding' memory:
    'Stack Mgr Notifi' (Holding 1892896 bytes)
  This is considered to be high. High memory usage can be normal, or can indicate
  a memory leak. To determine this, it is important to see if memory usage changes
  over time.
  INFO: A memory leak occurs when a process requests or allocates memory and then
  forgets to free (de-allocate) the memory when it is finished with that task.
  As a result, the memory block is reserved until the router is reloaded. Over
  time, more and more memory blocks are allocated by that process until there is
  no free memory available. At this point, the router will issue the 'SYS-2-MALLOCFAIL'
  error message.
  TRY THIS: Analyze the 'show process memory' output for this router over a period
  of time (for example, every few hours or days depending on whether you have a
  fast or slow leak).
  Check to see if memory utilization for the affected process(es) continues to increase
  and the amount of freed memory remains the same. The rate at which free memory
  disappears depends on how often the event occurs that leads to the leak. A memory
  leak is a complex condition sometimes requiring an IOS upgrade to correct. If
  the above is in fact occurring, and you are uncertain about how to proceed, use
  the Service Request Tool to contact the Cisco TAC for further assistance.
  NOTE: It is normal for some processes, such as Border Gateway Protocol (BGP) or
  Open Shortest Path First (OSPF), to use more than one megabyte of memory; this
  does not mean they are leaking.
 
  WARNING: The process '*Dead*' is holding more than 1 MB of memory.
  This is considered to be high, and indicates a possible memory leak. The 'Dead'
  process is not a real process. It only accounts for the memory allocated under
  the context of another process which is terminated.
  TRY THIS: If a memory leak is detected, and the 'Dead' process seems to be consuming
  the memory, issue the "show memory dead" command from your device, and analyze
  which process consumes more memory. To do so, look at the "What" section of
  the output. This problem can also be caused due to a Cisco IOS Bug. Review the
  Bug Tool Kit for potential bugs and upgrade the device with the latest version
  of Cisco IOS software. If the problem persists, open a service request with Cisco
  at Service Request Tool.
 
  INFO: The top 3 processes that are holding less than 1 MB of memory are:
    'hulc running con' is holding 578860 bytes
    'HL2MCM' is holding 295164 bytes
    'HL2MCM' is holding 295012 bytes

IOS is c2960s-universalk9-mz.122-53.SE1

I used BUG toolkit and did not see any relation
I have made sure no debugs are running

INFO: This device has 131072 K available for main memory
INFO: This device has a total of 128 MB of RAM installed.

Not sure what to do about this. Please advise.

Thanks
0
Comment
Question by:scoad5
  • 5
  • 4
9 Comments
 
LVL 3

Expert Comment

by:mrmozaffari
ID: 35088960
Do You have Redundant Links ?

Please Send the result of following commands:
(All of them)


show controllers cpu-interface
show processes cpu sorted 5sec
show platform port-asic stats drop
show ip traffic
show platform ip unicast statistics
show platform ip unicast counts

0
 
LVL 3

Expert Comment

by:mrmozaffari
ID: 35089203
It is Also Maybe something else ,Like a bug ,It is CSCth24278 which it has been fixed in 12.2(58)SE ,Maybe you need to change the IOS.
0
 

Author Comment

by:scoad5
ID: 35096854

The following commands are not available. See attached
show platform ip unicast statistics
show platform ip unicast counts

Also, the bug mentioned seems to state that if you telnet/console in, the proc % drops. Not sure that is the case.



Thanks for your expertise and assitance.




SRS-DIST-2960-XprtsXchng.log
0
 
LVL 3

Expert Comment

by:mrmozaffari
ID: 35098581
There is some error with Spanning tree protocol and layer 2 protocol in your network ,You didnt answer me "Do you have Redundant link in your network ?".

Do you have a network diagram of this network ?
You Hulc process is also high ,is your 2960 POE support ?
The normal Hulc process should be between 0 - 2% and for high 5% or 8% ,the main cause of this problem is Physical link flapping ,You can check for up down status for your ports.
What devices are connected to Port (5,8,10,20,21) ?
0
New My Cloud Pro Series - organize everything!

With space to keep virtually everything, the My Cloud Pro Series offers your team the network storage to edit, save and share production files from anywhere with an internet connection. Compatible with both Mac and PC, you're able to protect your content regardless of OS.

 

Author Comment

by:scoad5
ID: 35099391
No diagram, that is what I am trying to map out.

The 2960 stack connects to Core stack via etherchannel with (3) interfaces) comprising the channel
All 3, on the 2960 side have:
(dot1q not available on 2960)
switchport trunk allowed vlan 1-6,8 (There is no Vlan 4, 6) (vlan 3 is FW-inside)
 switchport mode trunk
 srr-queue bandwidth share 10 10 60 20
 queue-set 2
 priority-queue out
 mls qos trust cos
 macro description cisco-switch
 auto qos voip trust
 spanning-tree link-type point-to-point
 channel-group 1 mode on
end

interface Port-channel1
 switchport trunk allowed vlan 1-6,8
 switchport mode trunk
end


SRS-DIST-STACK#sh spanning-tree int Gig 2/0/48

Vlan                Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
VLAN0001            Root FWD 3         128.224  P2p
VLAN0002            Root FWD 3         128.224  P2p
VLAN0003            Desg FWD 3         128.224  P2p
VLAN0005            Root FWD 3         128.224  P2p
VLAN0008            Root FWD 3         128.224  P2p
SRS-DIST-STACK#sh spanning-tree int Gig 3/0/48

Vlan                Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
VLAN0001            Root FWD 3         128.224  P2p
VLAN0002            Root FWD 3         128.224  P2p
VLAN0003            Desg FWD 3         128.224  P2p
VLAN0005            Root FWD 3         128.224  P2p
VLAN0008            Root FWD 3         128.224  P2p
SRS-DIST-STACK#sh spanning-tree int Gig 4/0/48

Vlan                Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
VLAN0001            Root FWD 3         128.224  P2p
VLAN0002            Root FWD 3         128.224  P2p
VLAN0003            Desg FWD 3         128.224  P2p
VLAN0005            Root FWD 3         128.224  P2p
VLAN0008            Root FWD 3         128.224  P2p
SRS-DIST-STACK#



on Core side

switchport trunk encapsulation dot1q (dot1q not available on 2960)
 switchport trunk allowed vlan 1,2,5,8 ((There is no Vlan 4, 6) (vlan 3 is FW-inside)
 switchport mode trunk
 load-interval 60
 srr-queue bandwidth share 10 10 60 20
 srr-queue bandwidth shape 10 0 0 0
 queue-set 2
 mls qos trust dscp
 auto qos voip trust
 macro description cisco-switch
 channel-group 1 mode on
 spanning-tree link-type point-to-point
end

interface Port-channel1
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 1,2,5,8
 switchport mode trunk

SRS-CORE-STACK#sh spanning-tree int GigabitEthernet1/0/49

Vlan                Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
VLAN0001            Desg FWD 3         128.488  P2p
VLAN0002            Desg FWD 3         128.488  P2p
VLAN0005            Desg FWD 3         128.488  P2p
VLAN0008            Desg FWD 3         128.488  P2p
SRS-CORE-STACK#sh spanning-tree int GigabitEthernet1/0/51

Vlan                Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
VLAN0001            Desg FWD 3         128.488  P2p
VLAN0002            Desg FWD 3         128.488  P2p
VLAN0005            Desg FWD 3         128.488  P2p
VLAN0008            Desg FWD 3         128.488  P2p
SRS-CORE-STACK#sh spanning-tree int GigabitEthernet3/0/25

Vlan                Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
VLAN0001            Desg FWD 3         128.488  P2p
VLAN0002            Desg FWD 3         128.488  P2p
VLAN0005            Desg FWD 3         128.488  P2p
VLAN0008            Desg FWD 3         128.488  P2p
SRS-CORE-STACK#
0
 
LVL 3

Expert Comment

by:mrmozaffari
ID: 35107514
Did you check for Physical link flapping ?
Does your switch have POE ports ?
0
 

Author Comment

by:scoad5
ID: 35131081
The switches do not have POE, they are all WS-C2960S-48TS-L.

Ther are alot of logg entries for interfaces going up/down, however, there are no errors, resets etc to explain why they are going up/dwn.
0
 
LVL 3

Accepted Solution

by:
mrmozaffari earned 500 total points
ID: 35132217
So as i said your hulc process is to high and it is the reason for high cpu utilization ,you have to check your physical connectivity ,Duplex ,speed and check the hosts and devices which connected to switch.
0
 

Author Closing Comment

by:scoad5
ID: 35198326
Thank you for your assistance.
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

This is an article about my experiences with remote access to my clients (so that I may serve them) and eventually to my home office system via Radmin Remote Control. I have been using remote access for over 10 years and have been improving my metho…
If you're not part of the solution, you're part of the problem.   Tips on how to secure IoT devices, even the dumbest ones, so they can't be used as part of a DDoS botnet.  Use PRTG Network Monitor as one of the building blocks, to detect unusual…
Viewers will learn how to connect to a wireless network using the network security key. They will also learn how to access the IP address and DNS server for connections that must be done manually. After setting up a router, find the network security…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now