Solved

Cisco 2960-S Switch Stack Constant High CPU

Posted on 2011-09-14
11
4,083 Views
Last Modified: 2012-05-12
I have 3x Cisco 2960-S Switches stacked. The switch stack shows constant high CPU of 80%. Even on weekends when there is no-one in the office CPU usage never drops below 80%. Users complain intermittently about “slow Internet access”

I have attached the output of some show commands. I’m not really sure what the problem is?
show controllers cpu-interface shows a high number of invalid STP frames?
Is there something wrong with spanning tree or a loop in the network?
I would appreciate some expert helping finding the cause of this problem.

Cisco-2960-S-Switch-Stack-Consta.xls
0
Comment
Question by:dee_nz
  • 4
  • 3
  • 2
  • +1
11 Comments
 
LVL 11

Expert Comment

by:yelbaglf
ID: 36540294
Have you made any configuration changes or any physical connection changes?  Could you paste the results of the following:

show spanning-tree summary
show version


Here's an STP troubleshooting doc that might be helpful...
http://www.cisco.com/en/US/tech/tk389/tk621/technologies_tech_note09186a0080136673.shtml

Also, you might consider removing any uplinks, etc. and rebooting the switches during a maintenance window.  Then take some baseline measurements before plugging everything back in.  Do the baselines here look better?  What about after plugging everything back in and generating some traffic flow?  Uploading these results for us to review would be helpful as well.

0
 
LVL 22

Expert Comment

by:eeRoot
ID: 36547308
If another switch on the network is incorrectly configured, this switch may end up receiving a lot of STP packets that it has to process.  What spanning-tree protocol are the switches on you network running?  Can you check this switch's config, and the configs of any directly attached switches as well?
0
 
LVL 46

Accepted Solution

by:
Craig Beck earned 400 total points
ID: 36550420
This is apparently 'cosmetic' according to Cisco.

There is supposedly a bug related to the CPU showing high until you TELNET to the switch, but I can confirm that this definitely does not appear to be cosmetic, as it affects every single stack of 2960's that I have ever deployed, and it does affect performance in some cases.

Cisco claimed that it would be fixed in 12.2(55), but it was not....



CSCth24278 Bug Details
High CPU when no Console/VTY activity
Symptoms:

Catalyst 2960S switch may report elevated CPU utilization (e.g., 50%) under
normal conditions.

Conditions:

ALL the following conditions MUST match.

- This issue is observed in 2960S even without any configuration and connection.
- This issue occurs when the telnet/console session is idle. When a telnet
/console session to the switch is established, the CPU utilization falls to
normal baseline levels. As long as a telnet/console session remains active, the
CPU utilization remains normal.
- This issue is usually observed by "show process cpu history"

for e.g.,

2960S_A>en
2960S_A# show process cpu history

44444444445555555555555554444444444555555555555555
8888888899999888884444433333000009999988888000002222222222
100
90
80
70
60
50 **************************************************
40 **************************************************
30 **************************************************
20 **************************************************
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)

The above graph indicates the CPU utilization was ~50% and went down to 8%
when console session up.

Workaround:

None. This issue is cosmetic.

Further Problem Description:

On this platform, CPU utilization software accounting is performed incorrectly
resulting in misleading levels. This is true ONLY when the CPU is idle, there
is no console/VTY activity, or there are no packets sent to CPU.

When console/VTY activity is present or packets are sent to CPU, the CPU
utilization software accounting is correct.

There should NOT be any performance impact due to this bug.

This issue may be seen in Catalyst 2360 platforms also.

The bug fix will be in 12.2(58)SE1 and 12.2(55)SE3.
0
Connect further...control easier

With the ATEN CE624, you can now enjoy a high-quality visual experience powered by HDBaseT technology and the convenience of a single Cat6 cable to transmit uncompressed video with zero latency and multi-streaming for dual-view applications where remote access is required.

 
LVL 4

Author Comment

by:dee_nz
ID: 36571101
yelbaglf
I have added the outputs of the show show spanning-tree summary & show version commands and re-attached the xls
There are about 3 or 4 other un-managed switches that are connected to the stack.
But patching is a bit "untidy" so I will have to re-patch everything before I can see whats connected where and unplug them to test if this is causing the problem.
I will have to wait for an outage when I can do this - maybe in a weeks time...
So I can just unplug all the other switches and issue the reload command on the stack master to reboot the stack?
Then see if the high CPU problem is still there?

eeRoot
I have also attached the running config in the spreadsheet - can you please have a look and tell me if you can see anything wrong?

craigbeck
Thanks for pointing that out! When I have a telnet session open to the stack master switch CPU usage does drop from 80% to 50%
Disconnect the telnet session and CPU usage goes up to 80% again, 50% CPU still seems pretty high though?
Is it worth doing an IOS update to see if this is fixed?
Cisco-2960-S-Switch-Stack-Show-O.xls
0
 
LVL 11

Assisted Solution

by:yelbaglf
yelbaglf earned 100 total points
ID: 36571166
Your IOS version is on the affected list.  I would perform an IOS upgrade, and then determine new baseline cpu metrics.  If the issue continues afterwards, then proceed with the troubleshooting steps provided.
0
 
LVL 22

Expert Comment

by:eeRoot
ID: 36571362
Based on the charts from the "show processes cpu history" tab on the spreadsheet, it look Like the CPU usage dropped when you logged onto the switch.  Have you been able to observe the high CPU utilization in real time while logged into the switch?
0
 
LVL 4

Author Comment

by:dee_nz
ID: 36571737
OK, that sounds like good advice ;)
So the process for IOS update on the 2960-S stack is the same as 3750s?

http://www.cisco.com/en/US/products/hw/switches/ps5023/products_configuration_example09186a00804799d7.shtml

Using the Automatic (.tar image) method should be the easiest way to update all x3 2960-S switches in the stack? Or can I use Cisco Device Manager or Cisco Network Assistant to update the IOS on the stacked switches?
0
 
LVL 46

Expert Comment

by:Craig Beck
ID: 36572502
You are correct, the automatic method is easiest.

I've tried every version of IOS available for these switches since 12.2(55) and the issue is still apparent.  I've also removed redundant stacking cables to rule out STP but no luck.  I've applied known-good configs from 3750s to compare, but nothing.

There seems to be no way to fix these, from what I've seen!
0
 
LVL 4

Assisted Solution

by:dee_nz
dee_nz earned 0 total points
ID: 36895045
Good news!
Have updated the switch stack from 122-53.SE2 to 12.2(58)SE2 and CPU usage now looks more "normal". Before the update CPU usage was constantly high 80% after the update it is around 20-30%.
Will monitor the stack closely for another week to check it is really fixed and if our intermittent "slow network" problems are also fixed....
0
 
LVL 11

Expert Comment

by:yelbaglf
ID: 36895332
That's awesome!
0
 
LVL 4

Author Closing Comment

by:dee_nz
ID: 36972745
Thanks for all your help. CPU usage on the switch stack is now around 20%.
0

Featured Post

Announcing the Most Valuable Experts of 2016

MVEs are more concerned with the satisfaction of those they help than with the considerable points they can earn. They are the types of people you feel privileged to call colleagues. Join us in honoring this amazing group of Experts.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

The worst thing when starting a new job is when the previous Network Administrator left behind no documentation. How do you get into the devices? If you've been in this situation or just accidently mistyped your password, this article will hopefully…
This tutorial will go through the steps required to write a script that will back up the configuration settings of a HP-ProCurve switch. You will need to get the following things to follow this tutorial: Telnet Scripting Tool e.g. TST10.exe …
The Email Laundry PDF encryption service allows companies to send confidential encrypted  emails to anybody. The PDF document can also contain attachments that are embedded in the encrypted PDF. The password is randomly generated by The Email Laundr…

726 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question