Cisco 48 port 3750 stack, randomly rebooting

Posted on 2016-08-01
Medium Priority
Last Modified: 2016-09-18
We have recently converted 2 x Cisco 48 port 3750's into a mirrored stack configuration for fail over purposes,and for the last 2 weeks it has performed without issue.

This week however and hopefully someone has had the same issue and be able to help

The 3750 stack, 1 switch occasionally/randomly reboots with no particular pattern to it and as far as we can prove and trace there are no power issues

Help appreciated.
Question by:deanwilsons

Expert Comment

by:Ian Arakel
ID: 41738748
Hi There,

Were there any logs generated for the same?

Author Comment

ID: 41738842
Hi Ian

Thanks for looking at this, see below for SH Stack info    

2  -rwx        4911   Aug 2 2016 09:35:57 +00:00  private-config.text
    4  -rwx       32894   Aug 2 2016 09:35:57 +00:00  config.text
    5  drwx         192   Aug 2 2016 05:28:36 +00:00  crashinfo_ext
  510  drwx         192   Mar 1 1993 00:00:03 +00:00  crashinfo
    6  drwx         192   Mar 1 1993 00:19:56 +00:00  c3750-ipbasek9-mz.122-37.SE
  514  -rwx        1048   Aug 1 2016 10:21:52 +00:00  multiple-fs

32514048 bytes total (20894208 bytes free)
swdata01#sh stack
Minimum process stacks:
Free/Size   Name
5104/6000   hulc_flash init
3616/6000   SaveCrashBuffer
5744/6000   CDP BLOB
8572/9000   IP Background
5664/6000   vqpc_shim_create_addr_tbl
5644/6000   SPAN Subsystem
9212/12000  Init
5640/6000   IPC Zone Manager
5716/6000   RTTY Slave Registration Process
5244/6000   MDFS LC IPC Init Process
4788/6000   IPC Split Init
8736/9000   cdp init process
5248/6000   HUDLD fiber table process
5496/6000   bulk config process
4680/6000   HULC CrashBuffer
2596/3000   Rom Random Update Process
5420/6000   HVLAN Mapped Vlan Id Update Process
9280/12000  cfg mgr new master process
5732/6000   hpm bulk_vlan_state process
5728/6000   HCMP redirect process
5644/6000   HCMP new mac process
5352/6000   XDR LC PRIO Background
11244/12000  XDR LC Background
5388/6000   CEF LC Stats
5404/6000   CEF LC IPC Background
4528/6000   HDF Stack-Mac
5708/6000   MDFS Reload
5712/6000   HL3U ARP Refresh Proc
35180/36000  TCP Command
7984/12000  Virtual Exec
10276/12000  SSH Process
5132/6000   RTTY Client Registration
5156/6000   IPC LC Port Opener

Interrupt level stacks:
Level    Called Unused/Size  Name
  4   704335706   7804/9000  NETWORK INTERFACE CHIP
  5           0   9000/9000  SUPERVISOR EXCEPTIONS
  6    16761758   8924/9000  NS16550 VECTOR

System was restarted by address error at PC 0x124BC90, address 0x0
C3750 Software (C3750-IPBASEK9-M), Version 12.2(37)SE, RELEASE SOFTWARE (fc2)
Compiled Thu 10-May-07 16:31 by antonino (current version)
Image text-base: 0x00003000, data-base: 0x012C0000

Stack trace from system failure:
FP: 0x3482418, RA: 0x124BC90
FP: 0x3482428, RA: 0x124BC8C
FP: 0x3482458, RA: 0x1240478
FP: 0x3482480, RA: 0xBF4AA8
FP: 0x34824A0, RA: 0x12435D0
FP: 0x34824A8, RA: 0x9ED1B8
FP: 0x0, RA: 0x9E45A4
******* Information of Last System Crash **********
Using flash:/crashinfo/crashinfo_3.
Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version 12.2(37)SE, RELEASE SOFTWARE (fc2)
Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Thu 10-May-07 16:31 by antonino

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

SRR0 = 0x010DB624  SRR1 = 0x00029230  SRR2 = 0x0067D714  SRR3 = 0x00021000
ESR = 0x00000000  DEAR = 0x00000000  TSR = 0x8C000000  DBSR = 0x01000000

CPU Register Context:
Vector = 0x00002000  PC = 0x0124BC90  MSR = 0x00029230  CR = 0x33000005
LR = 0x0124BC8C  CTR = 0x00000000  XER = 0xE0000062
R0 = 0x0124BC8C  R1 = 0x03482418  R2 = 0x00000000  R3 = 0x03B57DD0
R4 = 0x03482406  R5 = 0x00000000  R6 = 0x00029230  R7 = 0xBEEFCAFE
R8 = 0x00000000  R9 = 0x03482405  R10 = 0x0346CF86  R11 = 0x0346CF85
R12 = 0x04DD32BC  R13 = 0x00110000  R14 = 0x012432B0  R15 = 0x00000000
R16 = 0x00000000  R17 = 0x00000000  R18 = 0x00000000  R19 = 0x00000000
R20 = 0x00000000  R21 = 0x00000000  R22 = 0x00000000  R23 = 0x00000000
R24 = 0x00000000  R25 = 0x00000000  R26 = 0x00000000  R27 = 0x00000000
R28 = 0x03B573D4  R29 = 0x03B57DD0  R30 = 0x00000000  R31 = 0x03B57DD0

Stack trace:
PC = 0x0124BC90, SP = 0x03482418
Frame 00: SP = 0x03482428    PC = 0x0124BC8C
Frame 01: SP = 0x03482458    PC = 0x01240478
Frame 02: SP = 0x03482480    PC = 0x00BF4AA8
Frame 03: SP = 0x034824A0    PC = 0x012435D0
Frame 04: SP = 0x034824A8    PC = 0x009ED1B8
Frame 05: SP = 0x00000000    PC = 0x009E45A4

Expert Comment

ID: 41739127
If the switch in under TAC I would recommend opening a case right away.
Fill in the form and get your FREE NFR key NOW!

Veeam is happy to provide a FREE NFR server license to certified engineers, trainers, and bloggers.  It allows for the non‑production use of Veeam Agent for Microsoft Windows. This license is valid for five workstations and two servers.


Author Comment

ID: 41739187
Unfortunately its not under TAC.

Is there a way to determine which switch is causing the issue?

Seeing as the whole stack reboots, would it be wise to split the stack and mirror the ports for the hosts, at least that way if we lose 1 switch, we keep the remaining alive to pass traffic.

LVL 20

Expert Comment

ID: 41739202
Granted, it would be a blind guess'n'check without TAC assistance, but you could always try upgrading the code to the latest available. I would go with the code level that has a "*" (a star icon) next to it as that indicates a well-tested stable release.

Accepted Solution

deanwilsons earned 0 total points
ID: 41740389
We have removed what we believe is the faulty stack member, and so far we have had no further issues.

looks like we have a faulty switch on our hands.

not sure we can get a later IOS either without a cisco contract.

Assisted Solution

by:Ian Arakel
Ian Arakel earned 2000 total points
ID: 41740391
This indicates that it is a hardware issue.

Author Closing Comment

ID: 41803561
No real solution, as once rebuilt the stack was fine.

Featured Post

Vote for the Most Valuable Expert

It’s time to recognize experts that go above and beyond with helpful solutions and engagement on site. Choose from the top experts in the Hall of Fame or on the right rail of your favorite topic page. Look for the blue “Nominate” button on their profile to vote.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

On Feb. 28, Amazon’s Simple Storage Service (S3) went down after an employee issued the wrong command during a debugging exercise. Among those affected were big names like Netflix, Spotify and Expedia.
Considering cloud tradeoffs and determining the right mix for your organization.
Both in life and business – not all partnerships are created equal. As the demand for cloud services increases, so do the number of self-proclaimed cloud partners. Asking the right questions up front in the partnership, will enable both parties …
Both in life and business – not all partnerships are created equal. Spend 30 short minutes with us to learn:   • Key questions to ask when considering a partnership to accelerate your business into the cloud • Pitfalls and mistakes other partners…
Suggested Courses

862 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question