deanwilsons
asked on
Cisco 48 port 3750 stack, randomly rebooting
We have recently converted 2 x Cisco 48 port 3750's into a mirrored stack configuration for fail over purposes,and for the last 2 weeks it has performed without issue.
This week however and hopefully someone has had the same issue and be able to help
The 3750 stack, 1 switch occasionally/randomly reboots with no particular pattern to it and as far as we can prove and trace there are no power issues
Help appreciated.
This week however and hopefully someone has had the same issue and be able to help
The 3750 stack, 1 switch occasionally/randomly reboots with no particular pattern to it and as far as we can prove and trace there are no power issues
Help appreciated.
ASKER
Hi Ian
Thanks for looking at this, see below for SH Stack info
2 -rwx 4911 Aug 2 2016 09:35:57 +00:00 private-config.text
4 -rwx 32894 Aug 2 2016 09:35:57 +00:00 config.text
5 drwx 192 Aug 2 2016 05:28:36 +00:00 crashinfo_ext
510 drwx 192 Mar 1 1993 00:00:03 +00:00 crashinfo
6 drwx 192 Mar 1 1993 00:19:56 +00:00 c3750-ipbasek9-mz.122-37.S E
514 -rwx 1048 Aug 1 2016 10:21:52 +00:00 multiple-fs
32514048 bytes total (20894208 bytes free)
swdata01#sh stack
Minimum process stacks:
Free/Size Name
5104/6000 hulc_flash init
3616/6000 SaveCrashBuffer
5744/6000 CDP BLOB
8572/9000 IP Background
5664/6000 vqpc_shim_create_addr_tbl
5644/6000 SPAN Subsystem
5400/6000 LICENSE AGENT DEFAULT
9212/12000 Init
5640/6000 IPC Zone Manager
5716/6000 RTTY Slave Registration Process
5244/6000 MDFS LC IPC Init Process
4788/6000 IPC Split Init
8736/9000 cdp init process
5248/6000 HUDLD fiber table process
5496/6000 RADIUS INITCONFIG
5496/6000 bulk config process
4680/6000 HULC CrashBuffer
2596/3000 Rom Random Update Process
5420/6000 HVLAN Mapped Vlan Id Update Process
9280/12000 cfg mgr new master process
5732/6000 hpm bulk_vlan_state process
5728/6000 HCMP redirect process
5644/6000 HCMP new mac process
5352/6000 XDR LC PRIO Background
11244/12000 XDR LC Background
5388/6000 CEF LC Stats
5404/6000 CEF LC IPC Background
4528/6000 HDF Stack-Mac
5708/6000 MDFS Reload
5712/6000 HL3U ARP Refresh Proc
35180/36000 TCP Command
7984/12000 Virtual Exec
10276/12000 SSH Process
5132/6000 RTTY Client Registration
5156/6000 IPC LC Port Opener
Interrupt level stacks:
Level Called Unused/Size Name
4 704335706 7804/9000 NETWORK INTERFACE CHIP
5 0 9000/9000 SUPERVISOR EXCEPTIONS
6 16761758 8924/9000 NS16550 VECTOR
System was restarted by address error at PC 0x124BC90, address 0x0
C3750 Software (C3750-IPBASEK9-M), Version 12.2(37)SE, RELEASE SOFTWARE (fc2)
Compiled Thu 10-May-07 16:31 by antonino (current version)
Image text-base: 0x00003000, data-base: 0x012C0000
Stack trace from system failure:
FP: 0x3482418, RA: 0x124BC90
FP: 0x3482428, RA: 0x124BC8C
FP: 0x3482458, RA: 0x1240478
FP: 0x3482480, RA: 0xBF4AA8
FP: 0x34824A0, RA: 0x12435D0
FP: 0x34824A8, RA: 0x9ED1B8
FP: 0x0, RA: 0x9E45A4
************************** ********** ********** *****
******* Information of Last System Crash **********
************************** ********** ********** *****
Using flash:/crashinfo/crashinfo _3.
Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version 12.2(37)SE, RELEASE SOFTWARE (fc2)
Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Thu 10-May-07 16:31 by antonino
Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!
SRR0 = 0x010DB624 SRR1 = 0x00029230 SRR2 = 0x0067D714 SRR3 = 0x00021000
ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x8C000000 DBSR = 0x01000000
CPU Register Context:
Vector = 0x00002000 PC = 0x0124BC90 MSR = 0x00029230 CR = 0x33000005
LR = 0x0124BC8C CTR = 0x00000000 XER = 0xE0000062
R0 = 0x0124BC8C R1 = 0x03482418 R2 = 0x00000000 R3 = 0x03B57DD0
R4 = 0x03482406 R5 = 0x00000000 R6 = 0x00029230 R7 = 0xBEEFCAFE
R8 = 0x00000000 R9 = 0x03482405 R10 = 0x0346CF86 R11 = 0x0346CF85
R12 = 0x04DD32BC R13 = 0x00110000 R14 = 0x012432B0 R15 = 0x00000000
R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000
R20 = 0x00000000 R21 = 0x00000000 R22 = 0x00000000 R23 = 0x00000000
R24 = 0x00000000 R25 = 0x00000000 R26 = 0x00000000 R27 = 0x00000000
R28 = 0x03B573D4 R29 = 0x03B57DD0 R30 = 0x00000000 R31 = 0x03B57DD0
Stack trace:
PC = 0x0124BC90, SP = 0x03482418
Frame 00: SP = 0x03482428 PC = 0x0124BC8C
Frame 01: SP = 0x03482458 PC = 0x01240478
Frame 02: SP = 0x03482480 PC = 0x00BF4AA8
Frame 03: SP = 0x034824A0 PC = 0x012435D0
Frame 04: SP = 0x034824A8 PC = 0x009ED1B8
Frame 05: SP = 0x00000000 PC = 0x009E45A4
Thanks for looking at this, see below for SH Stack info
2 -rwx 4911 Aug 2 2016 09:35:57 +00:00 private-config.text
4 -rwx 32894 Aug 2 2016 09:35:57 +00:00 config.text
5 drwx 192 Aug 2 2016 05:28:36 +00:00 crashinfo_ext
510 drwx 192 Mar 1 1993 00:00:03 +00:00 crashinfo
6 drwx 192 Mar 1 1993 00:19:56 +00:00 c3750-ipbasek9-mz.122-37.S
514 -rwx 1048 Aug 1 2016 10:21:52 +00:00 multiple-fs
32514048 bytes total (20894208 bytes free)
swdata01#sh stack
Minimum process stacks:
Free/Size Name
5104/6000 hulc_flash init
3616/6000 SaveCrashBuffer
5744/6000 CDP BLOB
8572/9000 IP Background
5664/6000 vqpc_shim_create_addr_tbl
5644/6000 SPAN Subsystem
5400/6000 LICENSE AGENT DEFAULT
9212/12000 Init
5640/6000 IPC Zone Manager
5716/6000 RTTY Slave Registration Process
5244/6000 MDFS LC IPC Init Process
4788/6000 IPC Split Init
8736/9000 cdp init process
5248/6000 HUDLD fiber table process
5496/6000 RADIUS INITCONFIG
5496/6000 bulk config process
4680/6000 HULC CrashBuffer
2596/3000 Rom Random Update Process
5420/6000 HVLAN Mapped Vlan Id Update Process
9280/12000 cfg mgr new master process
5732/6000 hpm bulk_vlan_state process
5728/6000 HCMP redirect process
5644/6000 HCMP new mac process
5352/6000 XDR LC PRIO Background
11244/12000 XDR LC Background
5388/6000 CEF LC Stats
5404/6000 CEF LC IPC Background
4528/6000 HDF Stack-Mac
5708/6000 MDFS Reload
5712/6000 HL3U ARP Refresh Proc
35180/36000 TCP Command
7984/12000 Virtual Exec
10276/12000 SSH Process
5132/6000 RTTY Client Registration
5156/6000 IPC LC Port Opener
Interrupt level stacks:
Level Called Unused/Size Name
4 704335706 7804/9000 NETWORK INTERFACE CHIP
5 0 9000/9000 SUPERVISOR EXCEPTIONS
6 16761758 8924/9000 NS16550 VECTOR
System was restarted by address error at PC 0x124BC90, address 0x0
C3750 Software (C3750-IPBASEK9-M), Version 12.2(37)SE, RELEASE SOFTWARE (fc2)
Compiled Thu 10-May-07 16:31 by antonino (current version)
Image text-base: 0x00003000, data-base: 0x012C0000
Stack trace from system failure:
FP: 0x3482418, RA: 0x124BC90
FP: 0x3482428, RA: 0x124BC8C
FP: 0x3482458, RA: 0x1240478
FP: 0x3482480, RA: 0xBF4AA8
FP: 0x34824A0, RA: 0x12435D0
FP: 0x34824A8, RA: 0x9ED1B8
FP: 0x0, RA: 0x9E45A4
**************************
******* Information of Last System Crash **********
**************************
Using flash:/crashinfo/crashinfo
Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version 12.2(37)SE, RELEASE SOFTWARE (fc2)
Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Thu 10-May-07 16:31 by antonino
Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!
SRR0 = 0x010DB624 SRR1 = 0x00029230 SRR2 = 0x0067D714 SRR3 = 0x00021000
ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x8C000000 DBSR = 0x01000000
CPU Register Context:
Vector = 0x00002000 PC = 0x0124BC90 MSR = 0x00029230 CR = 0x33000005
LR = 0x0124BC8C CTR = 0x00000000 XER = 0xE0000062
R0 = 0x0124BC8C R1 = 0x03482418 R2 = 0x00000000 R3 = 0x03B57DD0
R4 = 0x03482406 R5 = 0x00000000 R6 = 0x00029230 R7 = 0xBEEFCAFE
R8 = 0x00000000 R9 = 0x03482405 R10 = 0x0346CF86 R11 = 0x0346CF85
R12 = 0x04DD32BC R13 = 0x00110000 R14 = 0x012432B0 R15 = 0x00000000
R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000
R20 = 0x00000000 R21 = 0x00000000 R22 = 0x00000000 R23 = 0x00000000
R24 = 0x00000000 R25 = 0x00000000 R26 = 0x00000000 R27 = 0x00000000
R28 = 0x03B573D4 R29 = 0x03B57DD0 R30 = 0x00000000 R31 = 0x03B57DD0
Stack trace:
PC = 0x0124BC90, SP = 0x03482418
Frame 00: SP = 0x03482428 PC = 0x0124BC8C
Frame 01: SP = 0x03482458 PC = 0x01240478
Frame 02: SP = 0x03482480 PC = 0x00BF4AA8
Frame 03: SP = 0x034824A0 PC = 0x012435D0
Frame 04: SP = 0x034824A8 PC = 0x009ED1B8
Frame 05: SP = 0x00000000 PC = 0x009E45A4
If the switch in under TAC I would recommend opening a case right away.
ASKER
Unfortunately its not under TAC.
Is there a way to determine which switch is causing the issue?
Seeing as the whole stack reboots, would it be wise to split the stack and mirror the ports for the hosts, at least that way if we lose 1 switch, we keep the remaining alive to pass traffic.
thanks
Is there a way to determine which switch is causing the issue?
Seeing as the whole stack reboots, would it be wise to split the stack and mirror the ports for the hosts, at least that way if we lose 1 switch, we keep the remaining alive to pass traffic.
thanks
Granted, it would be a blind guess'n'check without TAC assistance, but you could always try upgrading the code to the latest available. I would go with the code level that has a "*" (a star icon) next to it as that indicates a well-tested stable release.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
No real solution, as once rebuilt the stack was fine.
Were there any logs generated for the same?