Cisco switches rebooting problem

A stack of 4 switches keep rebooting every hour.  
We replaced the UPS that worked for a while. It started again so we then removed the UPS and plugged straight to outlet they were fine for a while.  

We then notice they were rebooting every 23 hours but couldn't figure out why.  Now they are rebooting every hour.  I think they are failing but before I say yes for sure I thought I would run it by the experts first incase we are missing something.

 I will reward the points only if I find  a solution other than replacing the switches.
MzFireAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

mrbrain646Commented:
plug in the console cable to the switches and reboot. This will tell you if the post fails.
It will let know if the problem is hardware.

was the switches recently upgraded with new firmware?

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Keith AlabasterEnterprise ArchitectCommented:
Don't really give a toss about your points but let's see if we can clear this up.
First, confirm that you have all switches at the same level of IOS and updated accordingly? An issue in this area is well documented for some of the 35xx and 29xx series - what are you using?
How are the switches connected - via the interconnects or just simple wired port-to-port leads?
0
eeRootCommented:
Are the switches sending notices to a syslog server and having their CPU/Memory monitored by any monitoring tools?  Any errors in the logs?  What is the model # and firmware version of these switches?
0
Introducing the "443 Security Simplified" Podcast

This new podcast puts you inside the minds of leading white-hat hackers and security researchers. Hosts Marc Laliberte and Corey Nachreiner turn complex security concepts into easily understood and actionable insights on the latest cyber security headlines and trends.

Matt VCommented:
Have you tried turning on logging to a syslog server to see if you can trap what is going on?

I could see 1 or two switches being bad, but a stack of 4 behaving identically is very odd.

Have you changed the config or upgraded the IOS on them lately?
0
eeRootCommented:
Can you post a sanitized config from the switch stack master?
0
MzFireAuthor Commented:
okay let me gather some information  and get back to ya
0
eeRootCommented:
Can you post the results of these 2 commands:

sh env stack
sh env all
0
MzFireAuthor Commented:
Sorry the stack is 3 switches not 4

I don't know if this will help you any.  As far as know we have not done any updates also checking on syslog

Cisco Internetwork Operating System Software
IOS (tm) C3750 Software (C3750-I5-M), Version 12.1(14)EA1a, RELEASE SOFTWARE (fc                                                                                                                                1)
Copyright (c) 1986-2003 by cisco Systems, Inc.
Compiled Tue 02-Sep-03 04:39 by antonino
Image text-base: 0x00003000, data-base: 0x008F14A0

ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.1(14r)EA1a, RELEASE SOFTWA                                                                                                                                RE (fc1)

uptime is 42 minutes
System returned to ROM by power-on
System restarted at 14:58:39 EDT Thu Aug 11 2011
System image file is "flash:c3750-i5-mz.121.14-EA1a/c3750-i5-mz.121.14-EA1a.bin"

cisco WS-C3750-48TS (PowerPC405) processor (revision E0) with 118774K/12288K byt                                                                                                                                es of memory.
Processor board ID CAT0808R0TX
Last reset from power-on
Bridging software.
2 Virtual Ethernet/IEEE 802.3  interface(s)
96 FastEthernet/IEEE 802.3 interface(s)
32 Gigabit Ethernet/IEEE 802.3 interface(s)
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address      
Motherboard assembly number     : 73-7056-12
Power supply part number        : 341-0028-01
Motherboard serial number       : CAT08070W5J
Power supply serial number      : LIT080300U0
Model revision number           : E0
Motherboard revision number     : A0
Model number                    : WS-C3750-48TS-E
System serial number            :
SFP Module assembly part number : 73-7757-02
SFP Module revision Number      : A0
SFP Module serial number        : CAT080600NQ


Switch   Ports  Model              SW Version              SW Image
------   -----  -----              ----------              ----------
*    1   52     WS-C3750-48TS      12.1(14)EA1a            C3750-I5-M
     2   52     WS-C3750-48TS      12.1(14)EA1a            C3750-I5-M
     3   24     WS-C3750G-24T      12.1(14)EA1a            C3750-I5-M


Switch 02
---------
Switch Uptime                   : 42 minutes
Base ethernet MAC Address       :
Motherboard assembly number     : 73-7056-12
Power supply part number        : 341-0028-01
Motherboard serial number       : CAT08070VSD
Power supply serial number      : LIT080300YP
Model revision number           : E0
Motherboard revision number     : A0
Model number                    : WS-C3750-48TS-E
System serial number            : CAT0808R0TV
SFP Module assembly part number : 73-7757-02
SFP Module revision number      : A0
SFP Module serial number        : CAT080603ZE

Switch 03
---------
Switch Uptime                   : 43 minutes
Base ethernet MAC Address      
Motherboard assembly number     : 73-8046-05
Power supply part number        : 341-0048-01
Motherboard serial number       : CAT080203MW
Power supply serial number      : DTH07520EV4
Model revision number           : B0
Motherboard revision number     : A0
Model number                    : WS-C3750G-24T-E
System serial number            : CAT0803Z21D

Configuration register is 0xF
0
MzFireAuthor Commented:
switch#show switch stack-ports
  Switch #    Port 1       Port 2
  --------    ------       ------
    1           Ok          Down
    2           Ok           Ok
    3          Down          Ok
0
MzFireAuthor Commented:

switch#sh switch neighbors
  Switch #    Port 1       Port 2
  --------    ------       ------
      1        None          None
      2        None          None
      3        None          None

switch#sh switch stack-ports
  Switch #    Port 1       Port 2
  --------    ------       ------
    1           Ok          Down
    2           Ok           Ok
    3          Down          Ok
0
lrmooreCommented:
2 things jump out at me
12.1(14)EA1a - very old version of IOS. Recommend upgrading
The very latest IOS is 122-55.SE3
Anything prior to 12.2 is considered a deferred release and not available to download even.

>System returned to ROM by power-on
This looks  like an environmental power issue if the power is lost
0
MzFireAuthor Commented:
here u go eeroot

switch#sh env stack
SWITCH: 1
FAN is OK
TEMPERATURE is OK
POWER is OK
RPS is NOT PRESENT
SWITCH: 2
FAN is OK
TEMPERATURE is OK
POWER is OK
RPS is NOT PRESENT
SWITCH: 3
FAN is OK
TEMPERATURE is OK
POWER is OK
RPS is NOT PRESENT

switch#sh env all
FAN is OK
TEMPERATURE is OK
POWER is OK
RPS is NOT PRESENT
0
MzFireAuthor Commented:
lrmoore
We are going to check the power in that room I will let you know
0
MzFireAuthor Commented:
0
MzFireAuthor Commented:
I attached a clean config file take a look
0
Craig BeckCommented:
I would agree with lrmoore - it definitely looks like the power went off.

You could always try connecting a laptop to the console so you can see what happens live as the switch loses its power.  That might give you some clues?
0
MzFireAuthor Commented:
I tried that Friday and the switch didn't go down....lol Then Saturday every hour again....Sunday a few times....Today a few times at 1:50 this morning and nothing since.  
0
eeRootCommented:
Again, any errors in the logs, syslog, or SNMP monitoring tools?  It's likely either bad power or a bad power supply in the 3750.  You need to move the stack master's power cord to a different circuit or take it out of the stack completely and test it independently.
0
MzFireAuthor Commented:
thanks everyone for there input.  eeroot no they syslog didn't give me anything.  The new switches are here and will be put in place tomorrow.  Very expensive solution but it will be interesting to see happens next.  

Thanks again
0
Istvan KalmarHead of IT Security Division Commented:
Hi,

I advise to monitor dhe cpu and the interfaces via SNMP, is there any broadcast or multicast storm?

http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SXF/native/configuration/guide/storm.html
0
MzFireAuthor Commented:
I am unsure how to close this question. None of the offered solutions worked, we ended up replacing the stack.
0
Ernie BeekExpertCommented:
If replacing the stack did it, then your last answer would be the solution.
But if you want to reward people for their help anyway, you can always split points.
0
MzFireAuthor Commented:
Thanks for the help.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Routers

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.