Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people, just like you, are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
Solved

Cisco 2960-S Switch Stack Constant High CPU

Posted on 2011-09-14
11
4,044 Views
Last Modified: 2012-05-12
I have 3x Cisco 2960-S Switches stacked. The switch stack shows constant high CPU of 80%. Even on weekends when there is no-one in the office CPU usage never drops below 80%. Users complain intermittently about “slow Internet access”

I have attached the output of some show commands. I’m not really sure what the problem is?
show controllers cpu-interface shows a high number of invalid STP frames?
Is there something wrong with spanning tree or a loop in the network?
I would appreciate some expert helping finding the cause of this problem.

Cisco-2960-S-Switch-Stack-Consta.xls
0
Comment
Question by:dee_nz
  • 4
  • 3
  • 2
  • +1
11 Comments
 
LVL 11

Expert Comment

by:yelbaglf
ID: 36540294
Have you made any configuration changes or any physical connection changes?  Could you paste the results of the following:

show spanning-tree summary
show version


Here's an STP troubleshooting doc that might be helpful...
http://www.cisco.com/en/US/tech/tk389/tk621/technologies_tech_note09186a0080136673.shtml

Also, you might consider removing any uplinks, etc. and rebooting the switches during a maintenance window.  Then take some baseline measurements before plugging everything back in.  Do the baselines here look better?  What about after plugging everything back in and generating some traffic flow?  Uploading these results for us to review would be helpful as well.

0
 
LVL 22

Expert Comment

by:eeRoot
ID: 36547308
If another switch on the network is incorrectly configured, this switch may end up receiving a lot of STP packets that it has to process.  What spanning-tree protocol are the switches on you network running?  Can you check this switch's config, and the configs of any directly attached switches as well?
0
 
LVL 45

Accepted Solution

by:
Craig Beck earned 400 total points
ID: 36550420
This is apparently 'cosmetic' according to Cisco.

There is supposedly a bug related to the CPU showing high until you TELNET to the switch, but I can confirm that this definitely does not appear to be cosmetic, as it affects every single stack of 2960's that I have ever deployed, and it does affect performance in some cases.

Cisco claimed that it would be fixed in 12.2(55), but it was not....



CSCth24278 Bug Details
High CPU when no Console/VTY activity
Symptoms:

Catalyst 2960S switch may report elevated CPU utilization (e.g., 50%) under
normal conditions.

Conditions:

ALL the following conditions MUST match.

- This issue is observed in 2960S even without any configuration and connection.
- This issue occurs when the telnet/console session is idle. When a telnet
/console session to the switch is established, the CPU utilization falls to
normal baseline levels. As long as a telnet/console session remains active, the
CPU utilization remains normal.
- This issue is usually observed by "show process cpu history"

for e.g.,

2960S_A>en
2960S_A# show process cpu history

44444444445555555555555554444444444555555555555555
8888888899999888884444433333000009999988888000002222222222
100
90
80
70
60
50 **************************************************
40 **************************************************
30 **************************************************
20 **************************************************
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)

The above graph indicates the CPU utilization was ~50% and went down to 8%
when console session up.

Workaround:

None. This issue is cosmetic.

Further Problem Description:

On this platform, CPU utilization software accounting is performed incorrectly
resulting in misleading levels. This is true ONLY when the CPU is idle, there
is no console/VTY activity, or there are no packets sent to CPU.

When console/VTY activity is present or packets are sent to CPU, the CPU
utilization software accounting is correct.

There should NOT be any performance impact due to this bug.

This issue may be seen in Catalyst 2360 platforms also.

The bug fix will be in 12.2(58)SE1 and 12.2(55)SE3.
0
Easy, flexible multimedia distribution & control

Coming soon!  Ideal for large-scale A/V applications, ATEN's VM3200 Modular Matrix Switch is an all-in-one solution that simplifies video wall integration. Easily customize display layouts to see what you want, how you want it in 4k.

 
LVL 4

Author Comment

by:dee_nz
ID: 36571101
yelbaglf
I have added the outputs of the show show spanning-tree summary & show version commands and re-attached the xls
There are about 3 or 4 other un-managed switches that are connected to the stack.
But patching is a bit "untidy" so I will have to re-patch everything before I can see whats connected where and unplug them to test if this is causing the problem.
I will have to wait for an outage when I can do this - maybe in a weeks time...
So I can just unplug all the other switches and issue the reload command on the stack master to reboot the stack?
Then see if the high CPU problem is still there?

eeRoot
I have also attached the running config in the spreadsheet - can you please have a look and tell me if you can see anything wrong?

craigbeck
Thanks for pointing that out! When I have a telnet session open to the stack master switch CPU usage does drop from 80% to 50%
Disconnect the telnet session and CPU usage goes up to 80% again, 50% CPU still seems pretty high though?
Is it worth doing an IOS update to see if this is fixed?
Cisco-2960-S-Switch-Stack-Show-O.xls
0
 
LVL 11

Assisted Solution

by:yelbaglf
yelbaglf earned 100 total points
ID: 36571166
Your IOS version is on the affected list.  I would perform an IOS upgrade, and then determine new baseline cpu metrics.  If the issue continues afterwards, then proceed with the troubleshooting steps provided.
0
 
LVL 22

Expert Comment

by:eeRoot
ID: 36571362
Based on the charts from the "show processes cpu history" tab on the spreadsheet, it look Like the CPU usage dropped when you logged onto the switch.  Have you been able to observe the high CPU utilization in real time while logged into the switch?
0
 
LVL 4

Author Comment

by:dee_nz
ID: 36571737
OK, that sounds like good advice ;)
So the process for IOS update on the 2960-S stack is the same as 3750s?

http://www.cisco.com/en/US/products/hw/switches/ps5023/products_configuration_example09186a00804799d7.shtml

Using the Automatic (.tar image) method should be the easiest way to update all x3 2960-S switches in the stack? Or can I use Cisco Device Manager or Cisco Network Assistant to update the IOS on the stacked switches?
0
 
LVL 45

Expert Comment

by:Craig Beck
ID: 36572502
You are correct, the automatic method is easiest.

I've tried every version of IOS available for these switches since 12.2(55) and the issue is still apparent.  I've also removed redundant stacking cables to rule out STP but no luck.  I've applied known-good configs from 3750s to compare, but nothing.

There seems to be no way to fix these, from what I've seen!
0
 
LVL 4

Assisted Solution

by:dee_nz
dee_nz earned 0 total points
ID: 36895045
Good news!
Have updated the switch stack from 122-53.SE2 to 12.2(58)SE2 and CPU usage now looks more "normal". Before the update CPU usage was constantly high 80% after the update it is around 20-30%.
Will monitor the stack closely for another week to check it is really fixed and if our intermittent "slow network" problems are also fixed....
0
 
LVL 11

Expert Comment

by:yelbaglf
ID: 36895332
That's awesome!
0
 
LVL 4

Author Closing Comment

by:dee_nz
ID: 36972745
Thanks for all your help. CPU usage on the switch stack is now around 20%.
0

Featured Post

PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Multicast on 3750x cisco router 1 45
SSH logs Cisco switch 4 95
Stack Switches in IOU  web V22 6 116
configure ASA Vlan Interface 14 70
I see many questions here on Experts Exchange regarding switch port configurations and trunks. This article is meant for beginners in the subject to help to get basic knowledge about Virtual Local Area Network (VLAN (http://en.wikipedia.org/wiki/Vir…
I eventually solved a perplexing problem setting up telnet for a new switch.  I installed a new Cisco WS-03560X-24P switch connected to an existing Cisco 4506 running a WS-X4013-10GE Sup II-Plus. After configuring vlans and trunking,  I could no…
Although Jacob Bernoulli (1654-1705) has been credited as the creator of "Binomial Distribution Table", Gottfried Leibniz (1646-1716) did his dissertation on the subject in 1666; Leibniz you may recall is the co-inventor of "Calculus" and beat Isaac…
The Email Laundry PDF encryption service allows companies to send confidential encrypted  emails to anybody. The PDF document can also contain attachments that are embedded in the encrypted PDF. The password is randomly generated by The Email Laundr…

809 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question