• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 4525
  • Last Modified:

Cisco 2960-S Switch Stack Constant High CPU

I have 3x Cisco 2960-S Switches stacked. The switch stack shows constant high CPU of 80%. Even on weekends when there is no-one in the office CPU usage never drops below 80%. Users complain intermittently about “slow Internet access”

I have attached the output of some show commands. I’m not really sure what the problem is?
show controllers cpu-interface shows a high number of invalid STP frames?
Is there something wrong with spanning tree or a loop in the network?
I would appreciate some expert helping finding the cause of this problem.

Cisco-2960-S-Switch-Stack-Consta.xls
0
dee_nz
Asked:
dee_nz
  • 4
  • 3
  • 2
  • +1
3 Solutions
 
yelbaglfCommented:
Have you made any configuration changes or any physical connection changes?  Could you paste the results of the following:

show spanning-tree summary
show version


Here's an STP troubleshooting doc that might be helpful...
http://www.cisco.com/en/US/tech/tk389/tk621/technologies_tech_note09186a0080136673.shtml

Also, you might consider removing any uplinks, etc. and rebooting the switches during a maintenance window.  Then take some baseline measurements before plugging everything back in.  Do the baselines here look better?  What about after plugging everything back in and generating some traffic flow?  Uploading these results for us to review would be helpful as well.

0
 
eeRootCommented:
If another switch on the network is incorrectly configured, this switch may end up receiving a lot of STP packets that it has to process.  What spanning-tree protocol are the switches on you network running?  Can you check this switch's config, and the configs of any directly attached switches as well?
0
 
Craig BeckCommented:
This is apparently 'cosmetic' according to Cisco.

There is supposedly a bug related to the CPU showing high until you TELNET to the switch, but I can confirm that this definitely does not appear to be cosmetic, as it affects every single stack of 2960's that I have ever deployed, and it does affect performance in some cases.

Cisco claimed that it would be fixed in 12.2(55), but it was not....



CSCth24278 Bug Details
High CPU when no Console/VTY activity
Symptoms:

Catalyst 2960S switch may report elevated CPU utilization (e.g., 50%) under
normal conditions.

Conditions:

ALL the following conditions MUST match.

- This issue is observed in 2960S even without any configuration and connection.
- This issue occurs when the telnet/console session is idle. When a telnet
/console session to the switch is established, the CPU utilization falls to
normal baseline levels. As long as a telnet/console session remains active, the
CPU utilization remains normal.
- This issue is usually observed by "show process cpu history"

for e.g.,

2960S_A>en
2960S_A# show process cpu history

44444444445555555555555554444444444555555555555555
8888888899999888884444433333000009999988888000002222222222
100
90
80
70
60
50 **************************************************
40 **************************************************
30 **************************************************
20 **************************************************
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)

The above graph indicates the CPU utilization was ~50% and went down to 8%
when console session up.

Workaround:

None. This issue is cosmetic.

Further Problem Description:

On this platform, CPU utilization software accounting is performed incorrectly
resulting in misleading levels. This is true ONLY when the CPU is idle, there
is no console/VTY activity, or there are no packets sent to CPU.

When console/VTY activity is present or packets are sent to CPU, the CPU
utilization software accounting is correct.

There should NOT be any performance impact due to this bug.

This issue may be seen in Catalyst 2360 platforms also.

The bug fix will be in 12.2(58)SE1 and 12.2(55)SE3.
0
Get expert help—faster!

Need expert help—fast? Use the Help Bell for personalized assistance getting answers to your important questions.

 
dee_nzAuthor Commented:
yelbaglf
I have added the outputs of the show show spanning-tree summary & show version commands and re-attached the xls
There are about 3 or 4 other un-managed switches that are connected to the stack.
But patching is a bit "untidy" so I will have to re-patch everything before I can see whats connected where and unplug them to test if this is causing the problem.
I will have to wait for an outage when I can do this - maybe in a weeks time...
So I can just unplug all the other switches and issue the reload command on the stack master to reboot the stack?
Then see if the high CPU problem is still there?

eeRoot
I have also attached the running config in the spreadsheet - can you please have a look and tell me if you can see anything wrong?

craigbeck
Thanks for pointing that out! When I have a telnet session open to the stack master switch CPU usage does drop from 80% to 50%
Disconnect the telnet session and CPU usage goes up to 80% again, 50% CPU still seems pretty high though?
Is it worth doing an IOS update to see if this is fixed?
Cisco-2960-S-Switch-Stack-Show-O.xls
0
 
yelbaglfCommented:
Your IOS version is on the affected list.  I would perform an IOS upgrade, and then determine new baseline cpu metrics.  If the issue continues afterwards, then proceed with the troubleshooting steps provided.
0
 
eeRootCommented:
Based on the charts from the "show processes cpu history" tab on the spreadsheet, it look Like the CPU usage dropped when you logged onto the switch.  Have you been able to observe the high CPU utilization in real time while logged into the switch?
0
 
dee_nzAuthor Commented:
OK, that sounds like good advice ;)
So the process for IOS update on the 2960-S stack is the same as 3750s?

http://www.cisco.com/en/US/products/hw/switches/ps5023/products_configuration_example09186a00804799d7.shtml

Using the Automatic (.tar image) method should be the easiest way to update all x3 2960-S switches in the stack? Or can I use Cisco Device Manager or Cisco Network Assistant to update the IOS on the stacked switches?
0
 
Craig BeckCommented:
You are correct, the automatic method is easiest.

I've tried every version of IOS available for these switches since 12.2(55) and the issue is still apparent.  I've also removed redundant stacking cables to rule out STP but no luck.  I've applied known-good configs from 3750s to compare, but nothing.

There seems to be no way to fix these, from what I've seen!
0
 
dee_nzAuthor Commented:
Good news!
Have updated the switch stack from 122-53.SE2 to 12.2(58)SE2 and CPU usage now looks more "normal". Before the update CPU usage was constantly high 80% after the update it is around 20-30%.
Will monitor the stack closely for another week to check it is really fixed and if our intermittent "slow network" problems are also fixed....
0
 
yelbaglfCommented:
That's awesome!
0
 
dee_nzAuthor Commented:
Thanks for all your help. CPU usage on the switch stack is now around 20%.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

  • 4
  • 3
  • 2
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now