Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
?
Solved

Cisco switches rebooting problem

Posted on 2011-08-11
24
Medium Priority
?
5,515 Views
Last Modified: 2012-08-13
A stack of 4 switches keep rebooting every hour.  
We replaced the UPS that worked for a while. It started again so we then removed the UPS and plugged straight to outlet they were fine for a while.  

We then notice they were rebooting every 23 hours but couldn't figure out why.  Now they are rebooting every hour.  I think they are failing but before I say yes for sure I thought I would run it by the experts first incase we are missing something.

 I will reward the points only if I find  a solution other than replacing the switches.
0
Comment
Question by:MzFire
23 Comments
 
LVL 4

Accepted Solution

by:
mrbrain646 earned 500 total points
ID: 36353053
plug in the console cable to the switches and reboot. This will tell you if the post fails.
It will let know if the problem is hardware.

was the switches recently upgraded with new firmware?

0
 
LVL 51

Expert Comment

by:Keith Alabaster
ID: 36353056
Don't really give a toss about your points but let's see if we can clear this up.
First, confirm that you have all switches at the same level of IOS and updated accordingly? An issue in this area is well documented for some of the 35xx and 29xx series - what are you using?
How are the switches connected - via the interconnects or just simple wired port-to-port leads?
0
 
LVL 22

Expert Comment

by:eeRoot
ID: 36353084
Are the switches sending notices to a syslog server and having their CPU/Memory monitored by any monitoring tools?  Any errors in the logs?  What is the model # and firmware version of these switches?
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 22

Assisted Solution

by:Matt V
Matt V earned 500 total points
ID: 36353091
Have you tried turning on logging to a syslog server to see if you can trap what is going on?

I could see 1 or two switches being bad, but a stack of 4 behaving identically is very odd.

Have you changed the config or upgraded the IOS on them lately?
0
 
LVL 22

Expert Comment

by:eeRoot
ID: 36353122
Can you post a sanitized config from the switch stack master?
0
 

Author Comment

by:MzFire
ID: 36353169
okay let me gather some information  and get back to ya
0
 
LVL 22

Assisted Solution

by:eeRoot
eeRoot earned 500 total points
ID: 36353203
Can you post the results of these 2 commands:

sh env stack
sh env all
0
 

Author Comment

by:MzFire
ID: 36353307
Sorry the stack is 3 switches not 4

I don't know if this will help you any.  As far as know we have not done any updates also checking on syslog

Cisco Internetwork Operating System Software
IOS (tm) C3750 Software (C3750-I5-M), Version 12.1(14)EA1a, RELEASE SOFTWARE (fc                                                                                                                                1)
Copyright (c) 1986-2003 by cisco Systems, Inc.
Compiled Tue 02-Sep-03 04:39 by antonino
Image text-base: 0x00003000, data-base: 0x008F14A0

ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.1(14r)EA1a, RELEASE SOFTWA                                                                                                                                RE (fc1)

uptime is 42 minutes
System returned to ROM by power-on
System restarted at 14:58:39 EDT Thu Aug 11 2011
System image file is "flash:c3750-i5-mz.121.14-EA1a/c3750-i5-mz.121.14-EA1a.bin"

cisco WS-C3750-48TS (PowerPC405) processor (revision E0) with 118774K/12288K byt                                                                                                                                es of memory.
Processor board ID CAT0808R0TX
Last reset from power-on
Bridging software.
2 Virtual Ethernet/IEEE 802.3  interface(s)
96 FastEthernet/IEEE 802.3 interface(s)
32 Gigabit Ethernet/IEEE 802.3 interface(s)
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address      
Motherboard assembly number     : 73-7056-12
Power supply part number        : 341-0028-01
Motherboard serial number       : CAT08070W5J
Power supply serial number      : LIT080300U0
Model revision number           : E0
Motherboard revision number     : A0
Model number                    : WS-C3750-48TS-E
System serial number            :
SFP Module assembly part number : 73-7757-02
SFP Module revision Number      : A0
SFP Module serial number        : CAT080600NQ


Switch   Ports  Model              SW Version              SW Image
------   -----  -----              ----------              ----------
*    1   52     WS-C3750-48TS      12.1(14)EA1a            C3750-I5-M
     2   52     WS-C3750-48TS      12.1(14)EA1a            C3750-I5-M
     3   24     WS-C3750G-24T      12.1(14)EA1a            C3750-I5-M


Switch 02
---------
Switch Uptime                   : 42 minutes
Base ethernet MAC Address       :
Motherboard assembly number     : 73-7056-12
Power supply part number        : 341-0028-01
Motherboard serial number       : CAT08070VSD
Power supply serial number      : LIT080300YP
Model revision number           : E0
Motherboard revision number     : A0
Model number                    : WS-C3750-48TS-E
System serial number            : CAT0808R0TV
SFP Module assembly part number : 73-7757-02
SFP Module revision number      : A0
SFP Module serial number        : CAT080603ZE

Switch 03
---------
Switch Uptime                   : 43 minutes
Base ethernet MAC Address      
Motherboard assembly number     : 73-8046-05
Power supply part number        : 341-0048-01
Motherboard serial number       : CAT080203MW
Power supply serial number      : DTH07520EV4
Model revision number           : B0
Motherboard revision number     : A0
Model number                    : WS-C3750G-24T-E
System serial number            : CAT0803Z21D

Configuration register is 0xF
0
 

Author Comment

by:MzFire
ID: 36353505
switch#show switch stack-ports
  Switch #    Port 1       Port 2
  --------    ------       ------
    1           Ok          Down
    2           Ok           Ok
    3          Down          Ok
0
 

Author Comment

by:MzFire
ID: 36353517

switch#sh switch neighbors
  Switch #    Port 1       Port 2
  --------    ------       ------
      1        None          None
      2        None          None
      3        None          None

switch#sh switch stack-ports
  Switch #    Port 1       Port 2
  --------    ------       ------
    1           Ok          Down
    2           Ok           Ok
    3          Down          Ok
0
 
LVL 79

Assisted Solution

by:lrmoore
lrmoore earned 500 total points
ID: 36354824
2 things jump out at me
12.1(14)EA1a - very old version of IOS. Recommend upgrading
The very latest IOS is 122-55.SE3
Anything prior to 12.2 is considered a deferred release and not available to download even.

>System returned to ROM by power-on
This looks  like an environmental power issue if the power is lost
0
 

Author Comment

by:MzFire
ID: 36358774
here u go eeroot

switch#sh env stack
SWITCH: 1
FAN is OK
TEMPERATURE is OK
POWER is OK
RPS is NOT PRESENT
SWITCH: 2
FAN is OK
TEMPERATURE is OK
POWER is OK
RPS is NOT PRESENT
SWITCH: 3
FAN is OK
TEMPERATURE is OK
POWER is OK
RPS is NOT PRESENT

switch#sh env all
FAN is OK
TEMPERATURE is OK
POWER is OK
RPS is NOT PRESENT
0
 

Author Comment

by:MzFire
ID: 36358790
lrmoore
We are going to check the power in that room I will let you know
0
 

Author Comment

by:MzFire
ID: 36358911
0
 

Author Comment

by:MzFire
ID: 36358915
I attached a clean config file take a look
0
 
LVL 47

Expert Comment

by:Craig Beck
ID: 36370712
I would agree with lrmoore - it definitely looks like the power went off.

You could always try connecting a laptop to the console so you can see what happens live as the switch loses its power.  That might give you some clues?
0
 

Author Comment

by:MzFire
ID: 36372677
I tried that Friday and the switch didn't go down....lol Then Saturday every hour again....Sunday a few times....Today a few times at 1:50 this morning and nothing since.  
0
 
LVL 22

Expert Comment

by:eeRoot
ID: 36372779
Again, any errors in the logs, syslog, or SNMP monitoring tools?  It's likely either bad power or a bad power supply in the 3750.  You need to move the stack master's power cord to a different circuit or take it out of the stack completely and test it independently.
0
 

Author Comment

by:MzFire
ID: 36425121
thanks everyone for there input.  eeroot no they syslog didn't give me anything.  The new switches are here and will be put in place tomorrow.  Very expensive solution but it will be interesting to see happens next.  

Thanks again
0
 
LVL 34

Expert Comment

by:Istvan Kalmar
ID: 37265036
Hi,

I advise to monitor dhe cpu and the interfaces via SNMP, is there any broadcast or multicast storm?

http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SXF/native/configuration/guide/storm.html
0
 

Author Comment

by:MzFire
ID: 37461815
I am unsure how to close this question. None of the offered solutions worked, we ended up replacing the stack.
0
 
LVL 35

Expert Comment

by:Ernie Beek
ID: 37461985
If replacing the stack did it, then your last answer would be the solution.
But if you want to reward people for their help anyway, you can always split points.
0
 

Author Closing Comment

by:MzFire
ID: 37462151
Thanks for the help.
0

Featured Post

Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

There are two basic ways to configure a static route for Cisco IOS devices. I've written this article to highlight a case study comparing the configuration of a static route using the next-hop IP and the configuration of a static route using an outg…
Arrow Electronics was searching for a KVM  (Keyboard/Video/Mouse) switch that could display on one single monitor the current status of all units being tested on the rack.
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

564 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question