Cisco 2811 router powers down and up
I have a 2811 router with a T3 module. It was running great until two and a half weeks ago, when it started taking CRC errors on the input side of the T3. Then it really got strange. The router started turning itself off for various lengths of time and turning back on. Thinking the router was going bad, I RMA'd it and the T3 card and replaced it. All was good for a couple of days, then the same thing started up again. I RMA'd the second router. When I went to install the third router, I couldn't get power to it. It was plugged into a power stripon our UPS with a Cisco switch, which did not have any problem. I moved the plug to a different circuit on the UPS and finally got the router working. Again, it was fine for 2-3 days, and then it started the power down/power up thing. I even replaced the power cord, to be safe. Anyone here of something like this? Could a malformed packet across the T3 cause this? The router's T3 comes off a Fluke Network probe. I tried to eliminate this device, but the router wouldn't talk to the T3 without the probe being in-line.
ASKER
Cisco IOS Software, 2800 Software (C2800NM-ADVSECURITYK9-M),
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Wed 18-Aug-10 06:10 by prod_rel_team
ROM: System Bootstrap, Version 12.4(13r)T6, RELEASE SOFTWARE (fc1)
PHI-RTR uptime is 17 hours, 5 minutes
System returned to ROM by power-on
System restarted at 14:36:49 edt Sun Aug 29 2010
System image file is "flash:c2800nm-advsecurity
It was running on an older 12.4 release. I upgraded it with the hope the issue would disappear. No luck. I also have the console cable plugged into the server on-site, with a emulator running, so see if anything shows, but nothing.
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Wed 18-Aug-10 06:10 by prod_rel_team
ROM: System Bootstrap, Version 12.4(13r)T6, RELEASE SOFTWARE (fc1)
PHI-RTR uptime is 17 hours, 5 minutes
System returned to ROM by power-on
System restarted at 14:36:49 edt Sun Aug 29 2010
System image file is "flash:c2800nm-advsecurity
It was running on an older 12.4 release. I upgraded it with the hope the issue would disappear. No luck. I also have the console cable plugged into the server on-site, with a emulator running, so see if anything shows, but nothing.
what is this router plugged into, UPS, direction connection to the wallk socket, shared power strip (ANy other devices connected to the strip)?
Billy
Billy
ASKER
The router was plugged into a power strip off of a UPS that was shared with a 3750 switch and the site's server. They were not experiencing any issues. We first moved the router to a wall outlet, same results. Moved it to a seperate circuit, by itself, on the UPS. still happens. Checked the UPS, it indicated no issues at all. Cisco said it could be a parity issue, but on three different routers, two different versions of the IOS?
>Cisco said it could be a parity issue, but on three different routers, two different versions of the IOS?
yeah, dunno; possible, but not very likely.
What about graphing, do you graph Temp, CPU, memory, fans, etc from this 2811; if not, this might be a good time to fire up an on box, install Cacti on it and graph:
http://www.cacti.net/
Bandwidth on interface
interface errors
CPU
Memory
Temp
Fans
Anything you can graph to get a baseline of what is going before it powers down/up.
Something to think about too; the power supplies in the 2811 could be have an issue with the power coming in to the device (Other devices might night be seeing an issue as they might not be that sensitive to bad/dirty power). At this point you have been through 3 routers and already have replaced the hardware on the devices, so I find it hard to believe it is related to hardware (again, not impossible, just not very likely).
Billy
yeah, dunno; possible, but not very likely.
What about graphing, do you graph Temp, CPU, memory, fans, etc from this 2811; if not, this might be a good time to fire up an on box, install Cacti on it and graph:
http://www.cacti.net/
Bandwidth on interface
interface errors
CPU
Memory
Temp
Fans
Anything you can graph to get a baseline of what is going before it powers down/up.
Something to think about too; the power supplies in the 2811 could be have an issue with the power coming in to the device (Other devices might night be seeing an issue as they might not be that sensitive to bad/dirty power). At this point you have been through 3 routers and already have replaced the hardware on the devices, so I find it hard to believe it is related to hardware (again, not impossible, just not very likely).
Billy
ASKER
Changed IOS to a version Cisco said works better with the DS3. This did not solve the problem. Talked with a tech from another company. He said years ago he ran into a similar problem. His company took a power hit, and it caused a harmonic fluctuation that made devices power down. We also took a power flux(a microwave oven blew up). This morning, the network went down again. No power on the router. We moved the power cable to a wall outlet. Still no power. We then unplugged the switch directly above the router, plugged it back in, and the router recovered. Not knowing the location of the internal power supplies of these devices, I wonder if they are located above/below each other, and the power flux caused the switch to radiate just enough RF to kill the router
>We then unplugged the switch directly above the router, plugged it back in, and the router recovered.
I would get an electrician in to look at power
Billy
I would get an electrician in to look at power
Billy
ASKER
Electrician has cleared the power lines. Maybe the UPS is having the issue. I plan on replacing the switch first, and seeing if the problem continues.
> I plan on replacing the switch first, and seeing if the problem continues.
sorry, little confused, I thought you were having issues with a router
sorry, little confused, I thought you were having issues with a router
ASKER
I am, but it's been replaced twice already. Now, we are trying to eliminate possible causes.
>Now, we are trying to eliminate possible causes.
I would not replace the switch, if the switch is not having any issues then I would focus on the cause elsewhere.
>Maybe the UPS is having the issue
you said that you have bypassed this already and the router is plugged directly into the outlet
Billy
I would not replace the switch, if the switch is not having any issues then I would focus on the cause elsewhere.
>Maybe the UPS is having the issue
you said that you have bypassed this already and the router is plugged directly into the outlet
Billy
ASKER
Ok, status as of now...
I've replaced the UPS, RMA'd the router(again), replaced the switch that plugs into the router. Things are stable except for one item...the T3 circuit will not work unless the Fluke DS3 ASE, model 204, is in-line. Plugging the T3 directly into the router causes a yellow alarm that won't clear.
I've replaced the UPS, RMA'd the router(again), replaced the switch that plugs into the router. Things are stable except for one item...the T3 circuit will not work unless the Fluke DS3 ASE, model 204, is in-line. Plugging the T3 directly into the router causes a yellow alarm that won't clear.
one of the reasons why you do not do bulk changes at once; sounds like you have possibly resolved the router powering down and up, but what fixed it, what introduced the issue that you are seeing now with the T3; I have no clue, you made too many changes. Are you able to plan a BNC loopback on the router T3 module to see if the alarm still exists?
ASKER
Update...Global Crossing sent out a tech with a t-berd to do some more testing. He found that the signal from the router out was dead on target for the DS3, which he said should be 7.5db. However, the signal from the SmartJack to the router was running hot, around 13db. He was going to put in a order to have the local telco adjust the signaling. We have run clean right now since Friday, with the exception of a two minute outage Saturday morning around 3:am.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Billy