• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 2840
  • Last Modified:

Cisco 2821 Crashing (Crash Info log)

Hey guys, I have a Cisco 2821VK9 router that is crashing periodically.  Can you take a look at the crash info log and give me some advice? (See attached)

 crashinfo-20100819-143437-EDT crashinfo-20100819-150027-EDT

Here is my sh ver

Cisco IOS Software, 2800 Software (C2800NM-SPSERVICESK9-M), Version 15.1(1)T, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Mon 22-Mar-10 01:25 by prod_rel_team

ROM: System Bootstrap, Version 12.3(8r)T7, RELEASE SOFTWARE (fc1)

ATLVIP1 uptime is 18 hours, 43 minutes
System returned to ROM by bus error at PC 0x0, address 0x0 at 15:00:28 EDT Thu Aug 19 2010
System restarted at 15:02:00 EDT Thu Aug 19 2010
System image file is "flash:c2800nm-spservicesk9-mz.151-1.T.bin"
Last reload type: Normal Reload

Cisco 2821 (revision 53.51) with 249856K/12288K bytes of memory.
Processor board ID FTX0912A12T
2 Gigabit Ethernet interfaces
144 Serial interfaces
6 Channelized T1/PRI ports
8 Voice FXS interfaces
DRAM configuration is 64 bits wide with parity enabled.
239K bytes of non-volatile configuration memory.
500472K bytes of ATA CompactFlash (Read/Write)
2 Solutions
Istvan KalmarHead of IT Security Division Commented:

your CPU is not enough for this:

%IVR-3-LOW_CPU_RESOURCE : IVR: System experiencing high cpu utilization ([dec]/100). Call (callID=[dec]) is rejected.\n

Explanation    The system does not have enough CPU resources available to accept a new call.

Recommended Action    Ensure that the call setup rate is within the supported capacity of this gateway.

I advise to upgrade the ios
AtlantixglobalAuthor Commented:
I did see that error and thought the same thing.  The IOS is 15.1.1T (ED).  How would an IOS upgrade possibly fix the issue?  Right now the CPU load is low.
Alex BaharCommented:
You're probably hitting an IOS IVR script software issue. Could your IVR script getting into a loop causing high CPU utilisation? Is the IVR script compatible with the version of IOS you're running? If this IVR script has been there before you upgraded to current IOS version, try to downgrade your IOS and see if the problem will disappear. If the IVR script is customised by you, debug your IVR script. There could be thousands (millions?) of reasons crashing your router. IMHO nobody can tell you why it is hapenning, or provide a fix by just looking at the crashdump. Crash is caused by software, you should contact Cisco TAC for support. TAC will forward it to the software developers.They will try to recreate the same issue in their labs using the same IOS release and configurations. Then they will debug it line by line  and try to come up with a resolution and probably give you an IOS image which does have a fix.
Never miss a deadline with monday.com

The revolutionary project management tool is here!   Plan visually with a single glance and make sure your projects get done.

>Possible software fault. Upon reccurence, please collect
   crashinfo, "show tech" and contact Cisco Technical Support.

Pretty clear that this is potential IOS bug...

>6 Channelized T1/PRI ports
On a 2821? That's working it really hard. What else is this router doing besides being a voice gateway? Is it MGCP gateway, or other?

AtlantixglobalAuthor Commented:
I just found out that the following debugs were running prior to the crash.  It's nice to have that info before hand.

ATLVIP1#sh deb
  TCP special event debugging is on
The following ISDN debugs are enabled on all DSLs:
debug isdn error is             ON.
debug isdn q931 is              ON.   (filter is OFF)
  H.225 ASN1 Messages debugging is on
  H.245 ASN1 Messages debugging is on
CCH323 SPI: H245 State Machine tracing is enabled (filter is OFF)
  debug voip ccapi inout is ON (filter is OFF)

I'm pretty sure this is what started hogging CPU time and ultimately caused the router to crash.
Alex BaharCommented:
It is good that you have found those detailed call processing debugs were turned on. Excessive amounts of terminal output can create issues especially if console logging is turned on. Turning off console logging using "no logging console" is a good practice. Logging messages to the console port generates non maskable interrupts, its impact gets worse especially if those messages are debug messages, which are high priority than other log types. Instead you can log to the buffer or to a syslog server or to telnet (terminal monitor) if logging is required.

Featured Post

Never miss a deadline with monday.com

The revolutionary project management tool is here!   Plan visually with a single glance and make sure your projects get done.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now