Solved

Cisco 2960-S Switch Stack Constant High CPU

Posted on 2011-09-14
11
4,017 Views
Last Modified: 2012-05-12
I have 3x Cisco 2960-S Switches stacked. The switch stack shows constant high CPU of 80%. Even on weekends when there is no-one in the office CPU usage never drops below 80%. Users complain intermittently about “slow Internet access”

I have attached the output of some show commands. I’m not really sure what the problem is?
show controllers cpu-interface shows a high number of invalid STP frames?
Is there something wrong with spanning tree or a loop in the network?
I would appreciate some expert helping finding the cause of this problem.

Cisco-2960-S-Switch-Stack-Consta.xls
0
Comment
Question by:dee_nz
  • 4
  • 3
  • 2
  • +1
11 Comments
 
LVL 11

Expert Comment

by:yelbaglf
ID: 36540294
Have you made any configuration changes or any physical connection changes?  Could you paste the results of the following:

show spanning-tree summary
show version


Here's an STP troubleshooting doc that might be helpful...
http://www.cisco.com/en/US/tech/tk389/tk621/technologies_tech_note09186a0080136673.shtml

Also, you might consider removing any uplinks, etc. and rebooting the switches during a maintenance window.  Then take some baseline measurements before plugging everything back in.  Do the baselines here look better?  What about after plugging everything back in and generating some traffic flow?  Uploading these results for us to review would be helpful as well.

0
 
LVL 22

Expert Comment

by:eeRoot
ID: 36547308
If another switch on the network is incorrectly configured, this switch may end up receiving a lot of STP packets that it has to process.  What spanning-tree protocol are the switches on you network running?  Can you check this switch's config, and the configs of any directly attached switches as well?
0
 
LVL 45

Accepted Solution

by:
Craig Beck earned 400 total points
ID: 36550420
This is apparently 'cosmetic' according to Cisco.

There is supposedly a bug related to the CPU showing high until you TELNET to the switch, but I can confirm that this definitely does not appear to be cosmetic, as it affects every single stack of 2960's that I have ever deployed, and it does affect performance in some cases.

Cisco claimed that it would be fixed in 12.2(55), but it was not....



CSCth24278 Bug Details
High CPU when no Console/VTY activity
Symptoms:

Catalyst 2960S switch may report elevated CPU utilization (e.g., 50%) under
normal conditions.

Conditions:

ALL the following conditions MUST match.

- This issue is observed in 2960S even without any configuration and connection.
- This issue occurs when the telnet/console session is idle. When a telnet
/console session to the switch is established, the CPU utilization falls to
normal baseline levels. As long as a telnet/console session remains active, the
CPU utilization remains normal.
- This issue is usually observed by "show process cpu history"

for e.g.,

2960S_A>en
2960S_A# show process cpu history

44444444445555555555555554444444444555555555555555
8888888899999888884444433333000009999988888000002222222222
100
90
80
70
60
50 **************************************************
40 **************************************************
30 **************************************************
20 **************************************************
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)

The above graph indicates the CPU utilization was ~50% and went down to 8%
when console session up.

Workaround:

None. This issue is cosmetic.

Further Problem Description:

On this platform, CPU utilization software accounting is performed incorrectly
resulting in misleading levels. This is true ONLY when the CPU is idle, there
is no console/VTY activity, or there are no packets sent to CPU.

When console/VTY activity is present or packets are sent to CPU, the CPU
utilization software accounting is correct.

There should NOT be any performance impact due to this bug.

This issue may be seen in Catalyst 2360 platforms also.

The bug fix will be in 12.2(58)SE1 and 12.2(55)SE3.
0
PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

 
LVL 4

Author Comment

by:dee_nz
ID: 36571101
yelbaglf
I have added the outputs of the show show spanning-tree summary & show version commands and re-attached the xls
There are about 3 or 4 other un-managed switches that are connected to the stack.
But patching is a bit "untidy" so I will have to re-patch everything before I can see whats connected where and unplug them to test if this is causing the problem.
I will have to wait for an outage when I can do this - maybe in a weeks time...
So I can just unplug all the other switches and issue the reload command on the stack master to reboot the stack?
Then see if the high CPU problem is still there?

eeRoot
I have also attached the running config in the spreadsheet - can you please have a look and tell me if you can see anything wrong?

craigbeck
Thanks for pointing that out! When I have a telnet session open to the stack master switch CPU usage does drop from 80% to 50%
Disconnect the telnet session and CPU usage goes up to 80% again, 50% CPU still seems pretty high though?
Is it worth doing an IOS update to see if this is fixed?
Cisco-2960-S-Switch-Stack-Show-O.xls
0
 
LVL 11

Assisted Solution

by:yelbaglf
yelbaglf earned 100 total points
ID: 36571166
Your IOS version is on the affected list.  I would perform an IOS upgrade, and then determine new baseline cpu metrics.  If the issue continues afterwards, then proceed with the troubleshooting steps provided.
0
 
LVL 22

Expert Comment

by:eeRoot
ID: 36571362
Based on the charts from the "show processes cpu history" tab on the spreadsheet, it look Like the CPU usage dropped when you logged onto the switch.  Have you been able to observe the high CPU utilization in real time while logged into the switch?
0
 
LVL 4

Author Comment

by:dee_nz
ID: 36571737
OK, that sounds like good advice ;)
So the process for IOS update on the 2960-S stack is the same as 3750s?

http://www.cisco.com/en/US/products/hw/switches/ps5023/products_configuration_example09186a00804799d7.shtml

Using the Automatic (.tar image) method should be the easiest way to update all x3 2960-S switches in the stack? Or can I use Cisco Device Manager or Cisco Network Assistant to update the IOS on the stacked switches?
0
 
LVL 45

Expert Comment

by:Craig Beck
ID: 36572502
You are correct, the automatic method is easiest.

I've tried every version of IOS available for these switches since 12.2(55) and the issue is still apparent.  I've also removed redundant stacking cables to rule out STP but no luck.  I've applied known-good configs from 3750s to compare, but nothing.

There seems to be no way to fix these, from what I've seen!
0
 
LVL 4

Assisted Solution

by:dee_nz
dee_nz earned 0 total points
ID: 36895045
Good news!
Have updated the switch stack from 122-53.SE2 to 12.2(58)SE2 and CPU usage now looks more "normal". Before the update CPU usage was constantly high 80% after the update it is around 20-30%.
Will monitor the stack closely for another week to check it is really fixed and if our intermittent "slow network" problems are also fixed....
0
 
LVL 11

Expert Comment

by:yelbaglf
ID: 36895332
That's awesome!
0
 
LVL 4

Author Closing Comment

by:dee_nz
ID: 36972745
Thanks for all your help. CPU usage on the switch stack is now around 20%.
0

Featured Post

Courses: Start Training Online With Pros, Today

Brush up on the basics or master the advanced techniques required to earn essential industry certifications, with Courses. Enroll in a course and start learning today. Training topics range from Android App Dev to the Xen Virtualization Platform.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
How to enable wireless guest network in Cisco Wireless AP 11 60
Cisco VSS or VCP on GNS3 or IOU 3 113
Advertise subnet not directly attached 6 31
Etherchannel balancing 10 21
The worst thing when starting a new job is when the previous Network Administrator left behind no documentation. How do you get into the devices? If you've been in this situation or just accidently mistyped your password, this article will hopefully…
This tutorial will go through the steps required to write a script that will back up the configuration settings of a HP-ProCurve switch. You will need to get the following things to follow this tutorial: Telnet Scripting Tool e.g. TST10.exe …
This Micro Tutorial hows how you can integrate  Mac OSX to a Windows Active Directory Domain. Apple has made it easy to allow users to bind their macs to a windows domain with relative ease. The following video show how to bind OSX Mavericks to …
Although Jacob Bernoulli (1654-1705) has been credited as the creator of "Binomial Distribution Table", Gottfried Leibniz (1646-1716) did his dissertation on the subject in 1666; Leibniz you may recall is the co-inventor of "Calculus" and beat Isaac…

776 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question