Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

Cisco 2960-S Switch Stack Constant High CPU

Posted on 2011-09-14
11
Medium Priority
?
4,279 Views
Last Modified: 2012-05-12
I have 3x Cisco 2960-S Switches stacked. The switch stack shows constant high CPU of 80%. Even on weekends when there is no-one in the office CPU usage never drops below 80%. Users complain intermittently about “slow Internet access”

I have attached the output of some show commands. I’m not really sure what the problem is?
show controllers cpu-interface shows a high number of invalid STP frames?
Is there something wrong with spanning tree or a loop in the network?
I would appreciate some expert helping finding the cause of this problem.

Cisco-2960-S-Switch-Stack-Consta.xls
0
Comment
Question by:dee_nz
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 3
  • 2
  • +1
11 Comments
 
LVL 11

Expert Comment

by:yelbaglf
ID: 36540294
Have you made any configuration changes or any physical connection changes?  Could you paste the results of the following:

show spanning-tree summary
show version


Here's an STP troubleshooting doc that might be helpful...
http://www.cisco.com/en/US/tech/tk389/tk621/technologies_tech_note09186a0080136673.shtml

Also, you might consider removing any uplinks, etc. and rebooting the switches during a maintenance window.  Then take some baseline measurements before plugging everything back in.  Do the baselines here look better?  What about after plugging everything back in and generating some traffic flow?  Uploading these results for us to review would be helpful as well.

0
 
LVL 22

Expert Comment

by:eeRoot
ID: 36547308
If another switch on the network is incorrectly configured, this switch may end up receiving a lot of STP packets that it has to process.  What spanning-tree protocol are the switches on you network running?  Can you check this switch's config, and the configs of any directly attached switches as well?
0
 
LVL 47

Accepted Solution

by:
Craig Beck earned 1600 total points
ID: 36550420
This is apparently 'cosmetic' according to Cisco.

There is supposedly a bug related to the CPU showing high until you TELNET to the switch, but I can confirm that this definitely does not appear to be cosmetic, as it affects every single stack of 2960's that I have ever deployed, and it does affect performance in some cases.

Cisco claimed that it would be fixed in 12.2(55), but it was not....



CSCth24278 Bug Details
High CPU when no Console/VTY activity
Symptoms:

Catalyst 2960S switch may report elevated CPU utilization (e.g., 50%) under
normal conditions.

Conditions:

ALL the following conditions MUST match.

- This issue is observed in 2960S even without any configuration and connection.
- This issue occurs when the telnet/console session is idle. When a telnet
/console session to the switch is established, the CPU utilization falls to
normal baseline levels. As long as a telnet/console session remains active, the
CPU utilization remains normal.
- This issue is usually observed by "show process cpu history"

for e.g.,

2960S_A>en
2960S_A# show process cpu history

44444444445555555555555554444444444555555555555555
8888888899999888884444433333000009999988888000002222222222
100
90
80
70
60
50 **************************************************
40 **************************************************
30 **************************************************
20 **************************************************
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)

The above graph indicates the CPU utilization was ~50% and went down to 8%
when console session up.

Workaround:

None. This issue is cosmetic.

Further Problem Description:

On this platform, CPU utilization software accounting is performed incorrectly
resulting in misleading levels. This is true ONLY when the CPU is idle, there
is no console/VTY activity, or there are no packets sent to CPU.

When console/VTY activity is present or packets are sent to CPU, the CPU
utilization software accounting is correct.

There should NOT be any performance impact due to this bug.

This issue may be seen in Catalyst 2360 platforms also.

The bug fix will be in 12.2(58)SE1 and 12.2(55)SE3.
0
Ask an Anonymous Question!

Don't feel intimidated by what you don't know. Ask your question anonymously. It's easy! Learn more and upgrade.

 
LVL 4

Author Comment

by:dee_nz
ID: 36571101
yelbaglf
I have added the outputs of the show show spanning-tree summary & show version commands and re-attached the xls
There are about 3 or 4 other un-managed switches that are connected to the stack.
But patching is a bit "untidy" so I will have to re-patch everything before I can see whats connected where and unplug them to test if this is causing the problem.
I will have to wait for an outage when I can do this - maybe in a weeks time...
So I can just unplug all the other switches and issue the reload command on the stack master to reboot the stack?
Then see if the high CPU problem is still there?

eeRoot
I have also attached the running config in the spreadsheet - can you please have a look and tell me if you can see anything wrong?

craigbeck
Thanks for pointing that out! When I have a telnet session open to the stack master switch CPU usage does drop from 80% to 50%
Disconnect the telnet session and CPU usage goes up to 80% again, 50% CPU still seems pretty high though?
Is it worth doing an IOS update to see if this is fixed?
Cisco-2960-S-Switch-Stack-Show-O.xls
0
 
LVL 11

Assisted Solution

by:yelbaglf
yelbaglf earned 400 total points
ID: 36571166
Your IOS version is on the affected list.  I would perform an IOS upgrade, and then determine new baseline cpu metrics.  If the issue continues afterwards, then proceed with the troubleshooting steps provided.
0
 
LVL 22

Expert Comment

by:eeRoot
ID: 36571362
Based on the charts from the "show processes cpu history" tab on the spreadsheet, it look Like the CPU usage dropped when you logged onto the switch.  Have you been able to observe the high CPU utilization in real time while logged into the switch?
0
 
LVL 4

Author Comment

by:dee_nz
ID: 36571737
OK, that sounds like good advice ;)
So the process for IOS update on the 2960-S stack is the same as 3750s?

http://www.cisco.com/en/US/products/hw/switches/ps5023/products_configuration_example09186a00804799d7.shtml

Using the Automatic (.tar image) method should be the easiest way to update all x3 2960-S switches in the stack? Or can I use Cisco Device Manager or Cisco Network Assistant to update the IOS on the stacked switches?
0
 
LVL 47

Expert Comment

by:Craig Beck
ID: 36572502
You are correct, the automatic method is easiest.

I've tried every version of IOS available for these switches since 12.2(55) and the issue is still apparent.  I've also removed redundant stacking cables to rule out STP but no luck.  I've applied known-good configs from 3750s to compare, but nothing.

There seems to be no way to fix these, from what I've seen!
0
 
LVL 4

Assisted Solution

by:dee_nz
dee_nz earned 0 total points
ID: 36895045
Good news!
Have updated the switch stack from 122-53.SE2 to 12.2(58)SE2 and CPU usage now looks more "normal". Before the update CPU usage was constantly high 80% after the update it is around 20-30%.
Will monitor the stack closely for another week to check it is really fixed and if our intermittent "slow network" problems are also fixed....
0
 
LVL 11

Expert Comment

by:yelbaglf
ID: 36895332
That's awesome!
0
 
LVL 4

Author Closing Comment

by:dee_nz
ID: 36972745
Thanks for all your help. CPU usage on the switch stack is now around 20%.
0

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The worst thing when starting a new job is when the previous Network Administrator left behind no documentation. How do you get into the devices? If you've been in this situation or just accidently mistyped your password, this article will hopefully…
I see many questions here on Experts Exchange regarding switch port configurations and trunks. This article is meant for beginners in the subject to help to get basic knowledge about Virtual Local Area Network (VLAN (http://en.wikipedia.org/wiki/Vir…
In this video, Percona Solution Engineer Rick Golba discuss how (and why) you implement high availability in a database environment. To discuss how Percona Consulting can help with your design and architecture needs for your database and infrastr…
In a question here at Experts Exchange (https://www.experts-exchange.com/questions/29062564/Adobe-acrobat-reader-DC.html), a member asked how to create a signature in Adobe Acrobat Reader DC (the free Reader product, not the paid, full Acrobat produ…

636 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question