Question

Severe Static Problems

Asked by: ctrost

We are at the end of the road here and I'm using this site as a last resort.  We had a site complaining that they would receive static on the line when making calls external to the network.  They also mentioned this happened randomly but when it does happen, you can only hear maybe every 3rd word and the call is rendered useless.  Berbee confirmed that it wasn't a configuration or a hardware issue.  We are using a Cisco 1760 Router with a combination of Cisco 7910 and 7912 phones at this site.

I built an exact replica of the router (hardware wise) but upgraded the IOS to a crypto image of 12.4.  The old router was on a basic IP voice image of 12.2.  We went out this morning to do extensive testing.  I had 2 guys making calls while I was monitoring the router.  As we made calls, Serial1/0:23 came up first, then progressively the interfaces came up counting down from :23 down to about :13 or :12.  After that, the calls became static'd and impossible.  In the meanwhile, users were taking calls and making calls internally to the network and those calls (1 or 2 calls) appeared to bring up Serial1/0:0 and :1.  After that, calls would no longer work.  

I checked the router and I was getting this message:

14:04:43: %IVR-3-LOW_CPU_RESOURCE: IVR: System experiencing high cpu utilization (96/100).
     Call (callID=267) is rejected.

14:04:45: %IVR-3-LOW_CPU_RESOURCE: IVR: System experiencing high cpu utilization (96/100).
     Call (callID=268) is rejected.

14:05:08: %IVR-3-LOW_CPU_RESOURCE: IVR: System experiencing high cpu utilization (96/100).
     Call (callID=269) is rejected.

If you're like me, you'd jump quickly to say ok well, the 1760 just can't handle this many PRIs.  However, we have a 1760 at another site with the same amount of PVDM cards (2) and the same amount of DSP resources.  This other site has maxxed out their PRIs on several occasions and they have never experienced static.  I cannot find any major configuration differences between the other site and this one.

Any ideas?  Anything I can check?  I can access the working site and the static'd site both remotely so please let me know if you have any ideas.

This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.

Subscribe now for full access to Experts Exchange and get

Instant Access to this Solution

  • Plus...
  • 30 Day FREE access, no risk, no obligation
  • Collaborate with the world's top tech experts
  • Unlimited access to our exclusive solution database
  • Never be left without tech help again

Subscribe Now

Asked On
2006-09-27 at 07:10:55ID22004576
Tags

cpu

,

cisco

,

high

Topic

Voice Over IP

Participating Experts
1
Points
500
Comments
28

Trusted by hundreds of thousands everyday for fast, accurate and reliable tech support.

  • "The time we save is the biggest benefit of Experts Exchange to Warner Bros. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange." Mike Kapnisakis, Warner Bros.
  • "Our team likes having a resource that is more secure than just using Google and most experts using this service really know their stuff. It's nice to look here first versus using Google." Dayna Sellner, Lockheed Martin
  • "Anytime that I've been stumped with a problem, 9 out of 10 times Experts Exchange has either the accepted solution or an open discussion of the potential solution to the problem." Kenny Red, eBay Inc.

See what Experts Exchange can do for you.

Got a question?

We've got the answer.

Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.

Screenshot of Experts Exchange Knowledgebase

Need individual assistance?

Our experts are ready to help.

If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.

Screenshot of Experts Exchange Knowledgebase

Want to learn from the best?

Read articles from industry experts.

Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.

Screenshot of an Article

Working on a long term project?

Store your work and research.

Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.

Screenshot of Experts Exchange Knowledgebase

Access the answers to your technology questions today.

Subscribe Now

30-day free trial. Register in 60 seconds.

What Makes Experts Exchange Unique?

Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.

Trusted by the world's most respected brands.

image of each brand's logo

Faithfully serving IT professionals since 1996.

Experts Exchange Logo

Try it out and discover for yourself.

Subscribe Now

30-day free trial. Register in 60 seconds.

Related Solutions

  1. Cisco 2611 Crypto IKMP Memory Problem
    Hi, I have two Cisco 2611 VPN Access routers, each with several IPSEC GRE tunnels and servicing dynamic VPN Clients (using Cisco VPN Client 4.0) Two weeks ago both routers locked up, with no telnet access, and attempts to Console in we rejected with "Not enough Mem...
  2. Crypto ACL
    I'm after the answer to something thats been bugging me. I was asked to look at a network today that consisted of two sites a main site and a remote, a DSL link between the sites had an IPSEC VPN running over it terminating on a PIX at each end. The main site had a data subne...
  3. Voice Services  M
    My phones will not register. Please tell me what is wrong with my configuration version 12.4 no service pad service tcp-keepalives-in service tcp-keepalives-out service timestamps debug datetime localtime service timestamps log datetime localtime service pass...
  4. Voice Services  M
    My phones will not register. Please tell me what is wrong with my configuration version 12.4 no service pad service tcp-keepalives-in service tcp-keepalives-out service timestamps debug datetime localtime service timestamps log datetime localtime service pass...
  5. Voice Over IP Cisco Router
    Please I am reposting my configurations for your review and suggestions. I can't seem to register any phone on this. I have done debugs and no luck so far. version 12.4 no service pad service tcp-keepalives-in service tcp-keepalives-out service timestamps debug date...
  6. Cisco CUBE, SIP trunks and voice gateways
    Pretty straight-forward question here, I think: We have a customer that currently has Cisco ISR routers (2821s, specifically) serving as voice gateways for PRI circuits at multiple sites. This customer is converting their voice connectivity to SIP trunks, which were all t...

Free Tech Articles

  1. WARNING: 5 Reasons why you should NEVER fix a computer for free.
    It is in our nature to love the puzzle. We are obsessed. The lot of us. We love puzzles. We love the challenge. We thrive on finding the answer. We hate disarray. It bothers us deep in our soul. W...
  2. SCCM OSD Basic troubleshooting
    SCCM 2007 OSD is a fantastic way to deploy operating systems, however, like most things SCCM issues can sometimes be difficult to resolve due to the sheer volume of logs to sift through and the dispe...
  3. Migrate Small Business Server 2003 to Exchange 2010 and Windows 2008 R2
    This guide is intended to provide step by step instructions on how to migrate from Small Business Server 2003 to Windows 2008 R2 with Exchange 2010. For this migration to work you will need the fo...
  4. Create a Win7 Gadget
    This article shows you how to create a simple "Gadget" -- a sort of mini-application supported by Windows 7 and Vista. Gadgets can be dropped anywhere on the desktop to provide instant information, ...
  5. Outlook continually prompting for username and password
    There have been a lot of questions recently regarding Outlook prompting for a username and password whilst using Exchange 2007. There are a few reasons why this would happen and I will try to cover t...
  6. Backup Exchange 2010 Information Store using Windows Backup
    There seems to be quite a lot of confusion around the ability to backup Exchange 2010 using the built in Windows Backup feature. This stems from the omission of this feature prior to Exchange 2007 s...

Cloud Class Webinars

  1. Avoiding Bugs in Microsoft Access
    Alison Balter takes and in-depth look at avoiding bugs in Access. In this webinar you will learn about using the immediate window to debug your applications, invoking the debugger, using breakpoints to troubleshoot, stepping through code, setting the next statement to execute, ...
  2. Top 10 Best New Features in Visio 2010
    Scott Helmers gives live demonstrations of the top 10 new features in Visio 2010. This webinar will teach you how to create compelling diagrams by adding shapes to the page with a single click, linking the shapes in a diagram to data in Excel (or SQL Server, or SharePoint), ...
  3. IT Consultant Business Secrets Revealed
    Michael Munger, Experts Exchange tech pro and IT consultant, pulls back the curtain on his very successful businesses and answers question on every IT consultant and business owner should know about. He shares secrets on what he did to solve the 5 most common problems in IT, ...
  4. Disaster Recovery and Business Continuity
    Quest CTO, Mike Billon, gives an overview of the steps involved in building a dunamic disaster recovery plan. Through case studies and an examination of software/hardware tooles for monitoring and testing, you'll gain a better understandin of where you are, where you want ...
  5. Organize Your Visio Diagrams with Containers and Lists
    Scott Helmers uses cross functional flowcharts, wireframe diagrams, data graphic legends and seating charts to teach you: how to ustilize all three new structured diagram components in Visio 2010, the best practices for organizeing shapes in previous version of Visio, how to organize ...
  6. How to Us Objects, Properties, Events and Methods in Microsoft Access
    Alison Dalter gives an in-depbth look at objects, properties, events and methods in Microsoft Access. In this webinar you will learn about using the object browser, referring to objects, working with properties and methods, working with object variables, understanding the ...

Join the Community

Give a Little. Get a Lot.

Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.

Join the Community

Answers

 

by: ctrostPosted on 2006-09-27 at 07:16:05ID: 17610452

Here's an output of a SHOW IP INT BRIEF right before the problem started to happen:

Interface                  IP-Address      OK? Method Status                Protocol
FastEthernet0/0            unassigned      YES NVRAM  up                    up      
FastEthernet0/0.1          172.20.1.1      YES NVRAM  up                    up      
FastEthernet0/0.2          172.20.2.1      YES NVRAM  up                    up      
FastEthernet0/0.3          172.20.3.1      YES NVRAM  up                    up      
FastEthernet0/0.4          172.20.4.1      YES NVRAM  up                    up      
FastEthernet0/0.99         172.20.0.1      YES NVRAM  up                    up      
Serial0/0                  172.16.250.30   YES NVRAM  up                    up      
Serial1/0:0                unassigned      YES unset  down                  down    
Serial1/0:1                unassigned      YES unset  down                  down    
Serial1/0:2                unassigned      YES unset  down                  down    
Serial1/0:3                unassigned      YES unset  down                  down    
Serial1/0:4                unassigned      YES unset  down                  down    
Serial1/0:5                unassigned      YES unset  down                  down    
Serial1/0:6                unassigned      YES unset  down                  down    
Serial1/0:7                unassigned      YES unset  down                  down    
Serial1/0:8                unassigned      YES unset  down                  down    
Serial1/0:9                unassigned      YES unset  down                  down    
Serial1/0:10               unassigned      YES unset  down                  down    
Serial1/0:11               unassigned      YES unset  down                  down    
Serial1/0:12               unassigned      YES unset  up                    up      
Serial1/0:13               unassigned      YES unset  up                    up      
Serial1/0:14               unassigned      YES unset  up                    up      
Serial1/0:15               unassigned      YES unset  up                    up      
Serial1/0:16               unassigned      YES unset  up                    up      
Serial1/0:17               unassigned      YES unset  up                    up      
Serial1/0:18               unassigned      YES unset  up                    up      
Serial1/0:19               unassigned      YES unset  up                    up      
Serial1/0:20               unassigned      YES unset  up                    up      
Serial1/0:21               unassigned      YES unset  up                    up      
Serial1/0:22               unassigned      YES unset  up                    up      
Serial1/0:23               unassigned      YES NVRAM  up                 up      
Loopback0                  172.20.254.1   YES NVRAM  up                 up    

 

by: jfradyPosted on 2006-09-27 at 08:19:33ID: 17611062

Do a show cpu command verbose and see which process/processes are using all the CPU cycles.

 

by: ctrostPosted on 2006-09-27 at 08:28:32ID: 17611156

"show cpu" is an unrecognized command.

Again, I'm using a Cisco 1760
System image file is "flash:c1700-ipvoicek9-mz.124-10.bin

 

by: jfradyPosted on 2006-09-27 at 08:38:30ID: 17611234

I guess I should have put the exact command.  The command is "show processes cpu".  See below:

The show processes cpu Command
The show processes cpu command displays information about the active processes in the router and their corresponding CPU utilization statistics. The following is a sample output of the show processes cpu command:

 router#show processes cpu
      CPU utilization for five seconds: 8%/4%; one minute: 6%; five minutes: 5%
       PID Runtime(ms)   Invoked  uSecs    5Sec   1Min   5Min TTY Process
         1         384     32789     11   0.00%  0.00%  0.00%   0 Load Meter
         2        2752      1179   2334   0.73%  1.06%  0.29%   0 Exec
         3      318592      5273  60419   0.00%  0.15%  0.17%   0 Check heaps
         4           4         1   4000   0.00%  0.00%  0.00%   0 Pool Manager
         5        6472      6568    985   0.00%  0.00%  0.00%   0 ARP Input
         6       10892      9461   1151   0.00%  0.00%  0.00%   0 IP Input
         7       67388     53244   1265   0.16%  0.04%  0.02%   0 CDP Protocol
         8      145520    166455    874   0.40%  0.29%  0.29%   0 IP Background
         9        3356      1568   2140   0.08%  0.00%  0.00%   0 BOOTP Server
        10          32      5469      5   0.00%  0.00%  0.00%   0 Net Background
        11       42256    163623    258   0.16%  0.02%  0.00%   0 Per-Second Jobs
        12      189936    163623   1160   0.00%  0.04%  0.05%   0 Net Periodic
        13        3248      6351    511   0.00%  0.00%  0.00%   0 Net Input
        14         168     32790      5   0.00%  0.00%  0.00%   0 Compute load avgs
        15      152408      2731  55806   0.98%  0.12%  0.07%   0 Per-minute Jobs The following table lists and describes the fields in the show processes cpu output:

Field
 Description
 
CPU utilization for five seconds
 CPU utilization for the last five seconds. The first number indicates the total, the second number indicates the percent of CPU time spent at the interrupt level.
 
one minute
 CPU utilization for the last minute
 
five minutes
 CPU utilization for the last five minutes
 
PID
 The process ID
 
Runtime (ms)
 CPU time the process has used, expressed in milliseconds
 
Invoked
 The number of times the process has been invoked
 
uSecs
 Microseconds of CPU time for each process invocation
 
5Sec
 CPU utilization by task in the last five seconds
 
1Min
 CPU utilization by task in the last minute
 
5Min
 CPU utilization by task in the last five minutes
 
TTY
 Terminal that controls the process
 
Process
 Name of the process. For more information, refer to The Processes section of this document.
 

 

by: ctrostPosted on 2006-09-27 at 08:48:32ID: 17611320

I don't see anything out of the ordinary.  i wish i would've known this command at the time of being on site when I got the error message (as we used an increased amount of PRI channels for testing)

SAFECO-1760-01#show processes cpu
CPU utilization for five seconds: 40%/38%; one minute: 32%; five minutes: 28%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
   1          88        41       2146  0.00%  0.00%  0.00%   0 Chunk Manager    
   2       11646     12920        901  0.00%  0.00%  0.00%   0 Load Meter      
   3       32772     64357        509  0.00%  0.01%  0.00%   0 Skinny Msg Serve
   4           0         1          0  0.00%  0.00%  0.00%   0 EDDRI_MAIN      
   5      169410     11466      14774  0.00%  0.17%  0.17%   0 Check heaps      
   6          88        99        888  0.00%  0.00%  0.00%   0 Pool Manager    
   7           0         2          0  0.00%  0.00%  0.00%   0 Timers          
   8           0         1          0  0.00%  0.00%  0.00%   0 Crash writer    
   9        7175      7440        964  0.00%  0.00%  0.00%   0 ARP Input        
  10           0         2          0  0.00%  0.00%  0.00%   0 ATM Idle Timer  
  11           8        60        133  0.00%  0.00%  0.00%   0 AAA high-capacit
  12           0         1          0  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT
  13           0         1          0  0.00%  0.00%  0.00%   0 Policy Manager  
  14         105       345        304  0.00%  0.00%  0.00%   0 DDR Timers      
  15          12         2       6000  0.00%  0.00%  0.00%   0 Entity MIB API  
  16          76        36       2111  0.00%  0.00%  0.00%   0 EEM ED Syslog    
  17        2271     19334        117  0.00%  0.00%  0.00%   0 HC Counter Timer
  18           0         2          0  0.00%  0.00%  0.00%   0 Serial Backgroun
  19           0         1          0  0.00%  0.00%  0.00%   0 RO Notify Timers
  20           4         2       2000  0.00%  0.00%  0.00%   0 SMART            
  21        2660     64551         41  0.00%  0.00%  0.00%   0 GraphIt          
  22           0         2          0  0.00%  0.00%  0.00%   0 Dialer event    
  23           0         1          0  0.00%  0.00%  0.00%   0 SERIAL A'detect  
  24           0         2          0  0.00%  0.00%  0.00%   0 XML Proxy Client
  25           0         1          0  0.00%  0.00%  0.00%   0 Critical Bkgnd  
  26        6140     25736        238  0.00%  0.00%  0.00%   0 Net Background  
  27           8         5       1600  0.00%  0.00%  0.00%   0 IDB Work        
  28          56        27       2074  0.00%  0.00%  0.00%   0 Logger          
  29        3946     64360         61  0.00%  0.00%  0.00%   0 TTY Background  
  30        9864     64574        152  0.00%  0.00%  0.00%   0 Per-Second Jobs  
  31          49       539         90  0.00%  0.00%  0.00%   0 DHCPD Timer      
  32           0         1          0  0.00%  0.00%  0.00%   0 AggMgr Process  
  33           0         1          0  0.00%  0.00%  0.00%   0 dev_device_inser
  34           0         1          0  0.00%  0.00%  0.00%   0 dev_device_remov
  35           0         2          0  0.00%  0.00%  0.00%   0 ESWPPM          
  36        2788     10570        263  0.00%  0.00%  0.00%   0 Net Input        
  37       27284     12928       2110  0.08%  0.02%  0.01%   0 Compute load avg
  38       70121      1092      64213  0.00%  0.03%  0.05%   0 Per-minute Jobs  
  39           0         1          0  0.00%  0.00%  0.00%   0 AAL2CPS TIMER_CU
  40           0         3          0  0.00%  0.00%  0.00%   0 Service-module a
  41       12199    257981         47  0.00%  0.00%  0.00%   0 e1t1 Framer back
  42          88        17       5176  0.00%  0.00%  0.00%   0 IPM DSPRM MAIN  
  43           4         2       2000  0.00%  0.00%  0.00%   0 DTP Protocol    
  44          16       130        123  0.00%  0.00%  0.00%   0 AAA Dictionary R
  45          36        38        947  0.00%  0.00%  0.00%   0 AAA Server      
  46           0         1          0  0.00%  0.00%  0.00%   0 AAA ACCT Proc    
  47           0         1          0  0.00%  0.00%  0.00%   0 ACCT Periodic Pr
  48        3287      9689        339  0.00%  0.00%  0.00%   0 CDP Protocol    
  49       54568     62588        871  0.32%  0.11%  0.07%   0 IP Input        
  50           0         1          0  0.00%  0.00%  0.00%   0 ICMP event handl
  51           0         3          0  0.00%  0.00%  0.00%   0 PPP Hooks        
  52        9332      3606       2587  0.00%  0.04%  0.01%   0 AFW_application_
  53           0         1          0  0.00%  0.00%  0.00%   0 SSS Manager      
  54         149      8613         17  0.00%  0.00%  0.00%   0 SSS Test Client  
  55           0         1          0  0.00%  0.00%  0.00%   0 SSS Feature Mana
  56       13177    251987         52  0.00%  0.00%  0.00%   0 SSS Feature Time
  57           0         1          0  0.00%  0.00%  0.00%   0 VPDN call manage
  58           0         1          0  0.00%  0.00%  0.00%   0 L2X Socket proce
  59           0         1          0  0.00%  0.00%  0.00%   0 L2X SSS manager  
  60           0         2          0  0.00%  0.00%  0.00%   0 L2TP mgmt daemon
  61           0         1          0  0.00%  0.00%  0.00%   0 X.25 Encaps Mana
  62           4         2       2000  0.00%  0.00%  0.00%   0 PPP IP Route    
  63           0         2          0  0.00%  0.00%  0.00%   0 PPP IPCP        
  64        1237      2166        571  0.00%  0.00%  0.00%   0 IP Background    
  65         197      1087        181  0.00%  0.00%  0.00%   0 IP RIB Update    
  66       33507    113190        296  0.08%  0.02%  0.00%   0 CEF process      
  67        5124     96046         53  0.00%  0.00%  0.00%   0 Socket Timers    
  68           0         2          0  0.00%  0.00%  0.00%   0 SNMP Timers      
  69        3103      2285       1357  0.00%  0.00%  0.00%   0 TCP Timer        
  70          32        12       2666  0.00%  0.00%  0.00%   0 TCP Protocols    
  71           0         1          0  0.00%  0.00%  0.00%   0 COPS            
  72       39192    129460        302  0.00%  0.04%  0.00%   0 DHCPD Receive    
  73           0         2          0  0.00%  0.00%  0.00%   0 Dialer Forwarder
  74        1787      1078       1657  0.00%  0.00%  0.00%   0 Adj Manager      
  75           8       217         36  0.00%  0.00%  0.00%   0 HTTP CORE        
  76           0         1          0  0.00%  0.00%  0.00%   0 IP Traceroute    
  77          80      1076         74  0.00%  0.00%  0.00%   0 IP Cache Ager    
  78           0         1          0  0.00%  0.00%  0.00%   0 RARP Input      
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  79           0         1          0  0.00%  0.00%  0.00%   0 PAD InCall      
  80           0         2          0  0.00%  0.00%  0.00%   0 X.25 Background  
  81           0         2          0  0.00%  0.00%  0.00%   0 PPP Bind        
  82           0         2          0  0.00%  0.00%  0.00%   0 PPP SSS          
  83        4718     64369         73  0.00%  0.00%  0.00%   0 RUDPV1 Main Proc
  84         228      2586         88  0.00%  0.00%  0.00%   0 CRM_CALL_UPDATE_
  85           0         2          0  0.00%  0.00%  0.00%   0 ENABLE AAA      
  86           0         1          0  0.00%  0.00%  0.00%   0 EM Background Pr
  87           0         1          0  0.00%  0.00%  0.00%   0 Key chain liveke
  88           0         2          0  0.00%  0.00%  0.00%   0 LINE AAA        
  89          61        26       2346  0.00%  0.00%  0.00%   0 LOCAL AAA        
  90           4         2       2000  0.00%  0.00%  0.00%   0 TPLUS            
  91           0         2          0  0.00%  0.00%  0.00%   0 VSP_MGR          
  92           0         1          0  0.00%  0.00%  0.00%   0 encrypt proc    
  93         352       118       2983  0.00%  0.00%  0.00%   0 CCVPM_HTSP      
  94           0         2          0  0.00%  0.00%  0.00%   0 VPM_MWI_BACKGROU
  95           0         1          0  0.00%  0.00%  0.00%   0 CCVPM_R2        
  96           4        72         55  0.00%  0.00%  0.00%   0 FB/KS Log HouseK
  97           0         2          0  0.00%  0.00%  0.00%   0 EPHONE MWI BG Pr
  98           0         1          0  0.00%  0.00%  0.00%   0 CCSWVOICE        
  99       27574      1624      16979  3.11%  0.32%  0.54%   6 SSH Process      
 100           0         1          0  0.00%  0.00%  0.00%   0 http client proc
 101          16      1083         14  0.00%  0.00%  0.00%   0 DHCPD Database  
 102           0         1          0  0.00%  0.00%  0.00%   0 QOS_MODULE_MAIN  
 103           0         1          0  0.00%  0.00%  0.00%   0 RPMS_PROC_MAIN  
 104           0         1          0  0.00%  0.00%  0.00%   0 VoIP AAA        
 105           0         1          0  0.00%  0.00%  0.00%   0 crypto engine pr
 106          20         4       5000  0.00%  0.00%  0.00%   0 Crypto CA        
 107           0         1          0  0.00%  0.00%  0.00%   0 Crypto PKI-CRL  
 108           0         1          0  0.00%  0.00%  0.00%   0 Crypto SSL      
 109           0         1          0  0.00%  0.00%  0.00%   0 PM Callback      
 110        1107       704       1572  0.00%  0.00%  0.00%   0 AAA SEND STOP EV
 111           0         3          0  0.00%  0.00%  0.00%   0 EEM ED CLI      
 112           0         2          0  0.00%  0.00%  0.00%   0 EEM ED Counter  
 113           0         2          0  0.00%  0.00%  0.00%   0 EEM ED Interface
 114           0         3          0  0.00%  0.00%  0.00%   0 EEM ED IOSWD    
 115           0         2          0  0.00%  0.00%  0.00%   0 EEM ED Memory-th
 116           0         2          0  0.00%  0.00%  0.00%   0 EEM ED None      
 117           0         2          0  0.00%  0.00%  0.00%   0 EM ED OIR        
 118           0         2          0  0.00%  0.00%  0.00%   0 EEM ED SNMP      
 119         124      1079        114  0.00%  0.00%  0.00%   0 EEM ED Timer    
 120        1249     13723         91  0.00%  0.00%  0.00%   0 EEM Server      
 121           0         1          0  0.00%  0.00%  0.00%   0 Syslog Traps    
 122        2352     64220         36  0.00%  0.00%  0.00%   0 trunk conditioni
 123           0         1          0  0.00%  0.00%  0.00%   0 trunk conditioni
 124          84       213        394  0.00%  0.00%  0.00%   0 VLAN Manager    
 125           4         2       2000  0.00%  0.00%  0.00%   0 EEM Policy Direc
 126          48        17       2823  0.00%  0.00%  0.00%   0 Syslog          
 127           0         1          0  0.00%  0.00%  0.00%   0 VPDN Scal        
 129         829      2500        331  0.00%  0.00%  0.00%   0 Resource Monitor
 130        1702      1424       1195  0.00%  0.00%  0.00%   0 CC-API_VCM      
 131      122212     29434       4152  0.08%  0.07%  0.02%   0 DSMP            
 132       10415      4720       2206  0.00%  0.02%  0.00%   0 VTSP            
 133       56988     21799       2614  0.00%  0.03%  0.00%   0 VOIP_RTCP        
 134        3522      1901       1852  0.00%  0.00%  0.00%   0 TSP              
 135           0         1          0  0.00%  0.00%  0.00%   0 lib_off_app      
 136           8         2       4000  0.00%  0.00%  0.00%   0 Voice Player    
 137           0         1          0  0.00%  0.00%  0.00%   0 Media Record    
 138           0         1          0  0.00%  0.00%  0.00%   0 Resource Measure
 139          28         6       4666  0.00%  0.00%  0.00%   0 Session Applicat
 140       44918     10416       4312  0.00%  0.13%  0.04%   0 CCH323_CT        
 141           0         1          0  0.00%  0.00%  0.00%   0 CCH323_DNS      
 142          32         2      16000  0.00%  0.00%  0.00%   0 CCSIP_SPI_CONTRO
 143           0         1          0  0.00%  0.00%  0.00%   0 CCSIP_DNS        
 144           8         1       8000  0.00%  0.00%  0.00%   0 CCSIP_UDP_SOCKET
 145           4         1       4000  0.00%  0.00%  0.00%   0 CCSIP_TCP_SOCKET
 146          72      1077         66  0.00%  0.00%  0.00%   0 RTPSPI          
 147           8         2       4000  0.00%  0.00%  0.00%   0 CCFRF11_CT      
 148           8         2       4000  0.00%  0.00%  0.00%   0 CCAAL2_CT        
 149          48        37       1297  0.00%  0.00%  0.00%   0 SSH Event handle
 150       24119      2128      11334  0.00%  0.05%  0.01%   0 ISDN            
 151           0         2          0  0.00%  0.00%  0.00%   0 ISDN Timer      
 152          40       223        179  0.00%  0.00%  0.00%   0 ISDNMIB Backgrou
 153         356       634        561  0.00%  0.00%  0.00%   0 CallMIB Backgrou
 154        2443      8945        273  0.00%  0.00%  0.00%   0 ISDN From Driver
 155       41367     68184        606  0.08%  0.08%  0.03%   0 IGMP Input      
 156       38633    640945         60  0.00%  0.00%  0.00%   0 Mwheel Process  
 157       33760     86042        392  0.00%  0.03%  0.00%   0 PIM Process      
 158         256      2239        114  0.00%  0.00%  0.00%   0 CEF Scanner      
 159        1530     21513         71  0.00%  0.00%  0.00%   0 IPHC Admin      
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
 161        8975     10045        893  0.00%  0.01%  0.00%   0 IP SNMP          
 162        6335      5024       1260  0.00%  0.01%  0.00%   0 PDU DISPATCHER  
 163       50133      5076       9876  0.00%  0.10%  0.02%   0 SNMP ENGINE      
 164           0         1          0  0.00%  0.00%  0.00%   0 SNMP ConfCopyPro
 165           0         1          0  0.00%  0.00%  0.00%   0 SNMP Traps      
 166       75536   1988969         37  0.08%  0.00%  0.00%   0 RADIUS          
 167       10218     64391        158  0.08%  0.01%  0.00%   0 Skinny Socket Se
 168           0         1          0  0.00%  0.00%  0.00%   0 C1700_RING_LIMIT
 169       15282     15209       1004  0.00%  0.03%  0.00%   0 ISDN L2 Process  
 170         132       861        153  0.00%  0.00%  0.00%   0 ISDN L3 Timer Pr
 171        1763      8945        197  0.00%  0.01%  0.00%   0 ISDN L2D SRQ Pro
 172           0         1          0  0.00%  0.00%  0.00%   0 faxmsp          
 173           0         1          0  0.00%  0.00%  0.00%   0 DocMSP          
 174           8        19        421  0.00%  0.00%  0.00%   0 fax timers      
 175        5664    128655         44  0.00%  0.00%  0.00%   0 esmtp client soc
 176           0         2          0  0.00%  0.00%  0.00%   0 esmtp server soc
 177           0         1          0  0.00%  0.00%  0.00%   0 Fax Onramp-VFC  
 178      111319       291     382539  0.00%  0.00%  0.00%   0 crypto sw pk pro
 179           0        12          0  0.00%  0.00%  0.00%   0 IP-EIGRP Router  
 180        8239     27910        295  0.00%  0.00%  0.00%   0 IP-EIGRP: PDM    
 181       34136    125059        272  0.00%  0.01%  0.00%   0 IP-EIGRP: HELLO

 

by: jfradyPosted on 2006-09-27 at 08:57:44ID: 17611417

I agree nothing here jumps out.  Really need to run this command while issue is occuring.  

Any other details you can provide about the configuration?  You mention another site - are there VPN tunnels with encryption or anything like that?  Possibly any routing loops or duplicate IP's?  Have you sniffed the network while this is occuring?  Is data throughput seemingly OK?  Have you tried isolating the PVDM cards?

 

by: ctrostPosted on 2006-09-27 at 09:01:20ID: 17611447

When I first turned the router up, i got a message about a duplicate IP...something about a server and a ping.  I didn't document it because everything was working fine.   I haven't done any network testing while the problem occurred because at the time, I didn't really know what to test.  Throughput SEEMS ok....    The PVDM cards are both new.  2 completely different PVDM cards in a completely different router produce the same exact problem so I think we can rule that one out.

I sure wish I could reproduce that dup. IP message, it doesnt show up in "show log" nor does anything else of relevance.

 

by: jfradyPosted on 2006-09-27 at 09:11:08ID: 17611540

Interesting.  I agree re: the PVDM cards.  I meant to clarify that you had tried the alternate router and cards.

Are the phones being addressed via DHCP?  What about other devices on the network?  Are there multiple VLAN's?

Unfortunately the problem may have to be recreated to troubleshoot it more.  Unless we can stab enough to find it.  Of course there are still a thousand possibilities but a duplicate IP being handed out periodically could certainly be one possible answer.

 

by: ctrostPosted on 2006-09-27 at 09:16:29ID: 17611590

That dup. IP message has not appeared since boot.  I have terminal monitor turned on remotely so I can hopefully see messages that would normally only appear when consoled into the device and nothing is showing up.

Yes, DHCP is being done on the router to 4 switches with multiple devices (such as PCs, phones) plugged into the switchports.  And yes, as you can see in the show interfaces above, there are subinterfaces which would indicate multiple vlans.

 

by: jfradyPosted on 2006-09-27 at 09:35:08ID: 17611729

Yes I see that now that I scroll all the way back up.  Forgive me if I ask a question that seems obvious to you.  You have been working with it for some time.  I have for maybe 5 minutes.  Please keep that in mind.

So, 5 Class C subnets internal?  4 (assumably) layer 2 switches?  Even if they are high density switches that only leaves less than 200 usable ports.  I must assume then that there are broadcast containment or security issues you are dealing with?

Also - as you know - the 1760 is an edge router.  Not very good at routing between multiple potentially 100Mb/sec segments.  If you have a lot of unicast routed traffic I imagine the 1760 would start choking around 7-8Mb/sec.  Not too mention if you have significant multicast.  20 calls is the absolute max allowed on that platform.

 

by: ctrostPosted on 2006-09-27 at 09:43:08ID: 17611795

You are correct.

1 x 1760 Router
4 x 2924XL (24 port) Switches.

I'm new to the company and new to this setup as well.  This problem existed before they brought me on (I'm their new Cisco guy).  However, I don't know diddly about VoIP other than what i've learned in the few weeks I've been here.  I haven't yet done much in the way of monitoring bandwidth nor knowing whats good and whats bad.

How can I check this site (and the working site with the same hardware setup) to see if they are within limits of bandwidth.  I must say, though, that after what we've already checked and things I'm continuing to monitor, it seems as though the CPU Utilization problem isn't a fluke.  I just made a change to the router and did a "wr" to write it to memory, and it rejected 2 calls because of CPU utilization.

Also... my boss told me the PRIs have been maxed out multiple times at another site, and they have 23 PRIs just like this site.  Does that mean they had 23 calls simultaneously?

 

by: jfradyPosted on 2006-09-27 at 10:51:42ID: 17612364

Understood.  I've been in VoIP for a number of years but know little about the Cisco.  I'm one of those anti-Cisco people I guess you could say.  I do know IOS somewhat though as we all have to.

With 23 channels they actually have 1 PRI circuit.  A PRI is 23 B channels and 1 D channel for signalling and call setup.  If the entire PRI is used then indeed they have 23 simultaneous calls.  

On Ciscos website there is a calculator you can use to determine the max number of calls.  That is where I got the 20 from.  In my experience a software based router like the 1760 will not be able to do more than 50-70K packets per second at most depending on packet size.  Generally equates to 6-8 Mb/sec.

If you tried a different router with different cards though and had the same result there must be some other traffic causing the high utilization.

You can monitor and/or capture traffic at both sites using Ethereal (or Wireshark).  If you search the net you can find where to download these.  I would also recommend watching CPU stats on both the working and problem site.  You can also use MRTG, SolarWinds or any other SNMP or monitoring platform to monitor the routers over longer periods of time.  MRTG (multi router traffic grapher) is free.

 

by: ctrostPosted on 2006-09-28 at 05:36:05ID: 17618568

Something on this damn router is running the CPU resources very high.  I'm monitoring with an SNMP Agent this router, and another router with the same hardware at another site that has no problems.  Right now, this router's CPU utilization is around 50% (although the processes listed don't add up to 50, nor do they add up to 10%).  The other router's CPU util. is running stable at 15%.  Obviously with each call at this problem site, the CPU Util % rises....and when it gets high enough, it starts rejecting calls.

 

by: jfradyPosted on 2006-09-28 at 05:43:53ID: 17618635

Thats very interesting.

Could you check the interface stats on the Ethernet interface?  Maybe even clear counters on both routers and then see if the stats are similar?  This would rule out potential differences in traffic levels.  Also, a capture right in front of the router would be most telling if there are significant events occuring on the LAN that is causing utilization to spike.

In most cases I would immediately replace this hardware.  In this case though the weirdness is compunded by the fact that you have already tried replacing all the hardware!

 

by: ctrostPosted on 2006-09-28 at 05:49:47ID: 17618674

Yeah...I really doubt I'll be able to convince my boss to let me replace the router again, since we just did it and we have the same problem.

FastEthernet0/0 is up, line protocol is up
  Hardware is PQUICC_FEC, address is 000b.be90.2724 (bia 000b.be90.2724)
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 2/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, 100BaseTX/FX
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 18:52:02
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 635000 bits/sec, 929 packets/sec
  5 minute output rate 838000 bits/sec, 837 packets/sec
     10737381 packets input, 1605811727 bytes
     Received 31054 broadcasts, 0 runts, 0 giants, 0 throttles
     4 input errors, 0 CRC, 0 frame, 4 overrun, 0 ignored
     0 watchdog
     0 input packets with dribble condition detected
     9724343 packets output, 1637557284 bytes, 6 underruns
     6 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out

 

by: jfradyPosted on 2006-09-28 at 06:09:29ID: 17618838

Running right under 1Mbit/sec.  Was CPU utilization around 50% when this stat was captured?

There is a whole road to go down to troubleshoot high CPU utilization.  Some of the things to look at would be the software interrupts, processes that may be causing it, switching paths, memory, arp, etc.

There are some docs on Ciscos site detailing troubleshooting steps.  Here is a link to some if you would like to look through them:
http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a00800a70f2.shtml
http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800a6f3a.shtml
http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800a7306.shtml

Overall the problem is defined.  High CPU.  Issue is we need to find why.  If you can I would highly recommend sniffing in front of the router.  

 

by: ctrostPosted on 2006-09-28 at 06:14:21ID: 17618870

Looking through the documents now.  Thank you.

 

by: ctrostPosted on 2006-09-28 at 12:01:22ID: 17621930

I couldn't find anything that helped in any of the documentation, but there were some good ideas in there.  I'm going to the site today to downgrade the IOS to 12.4(3c) per Berbee's instruction.  They claim there may be bugs in my existing IOS that are causing the problems I'm having.  I'm also putting an NTAP in front of the router to sniff a few things.  I will monitor it tomorrow remotely through the sniffer and we'll see what happens.  I doubt changing IOS's is going to fix the problem but I've seen crazier things happen.

 

by: jfradyPosted on 2006-09-28 at 12:11:36ID: 17622028

Well, I've seen crazy things too but 2 different routers with 2 different IOS loads?  You never know.

I'm really interested to see what happens.  Good deal having an nTap.  Can see full duplex.  Unfortunately I'm a half duplex hub sort of sniffer.

Good luck!!!!

 

by: ctrostPosted on 2006-09-29 at 06:51:47ID: 17627240

As far as we can tell, nothing has changed by changing IOS versions to 12.4(3c).  We called the site with 6 external phones and the CPU load quickly rose above 80%.  However, we called with 10-12 phones internal to the network and although the CPU usage spiked (as it always does), it seemed to hold stable around 35-40%.  I believe its time to go in another direction.   Also Note:  our other site with a similar setup (which uses IPX in addition to everything else) is using less than half of the resources that this site is.

 

by: jfradyPosted on 2006-09-29 at 06:58:47ID: 17627297

When you say internal to the network:  Do you mean all station to station on the Internal LAN's or across to another site?  If internal that really makes sense.  The router would just be setting up things but voice traffic is unicast from phone to phone.   Going out to the real world the voice traffic has to be transcoded and converted from IP to Analog and then to PRI in this case.

Just to verify - when the router was exchanged the line cards were exchanged as well right?  And not moved to the replacement router?

Any luck on the sniffer capture yet?

 

by: ctrostPosted on 2006-09-29 at 07:04:38ID: 17627346

internal to the network in this case means we called internal extension #s even though they are at a different site.  When we called the public line thats when the problems occur.  And yes, the new router has all completely new hardware in it.

The way our sniffer works, we have to leave a laptop or something in that closet hooked up to one of the Analysis ports on the sniffer in order to actually report whats going on.  As it stands right now, information is being captured, we just can't retrieve it yet.  My boss wanted to see if getting the "right" IOS worked first.

 

by: ctrostPosted on 2006-09-29 at 07:07:02ID: 17627372

Here's the most recent CPU utilization sorted...

SAFECO-1760-01#show proc cpu sorted

CPU utilization for five seconds: 90%/68%; one minute: 71%; five minutes: 63%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
 192      547947    342543       1599  6.21%  2.46%  1.21%   0 SNMP ENGINE      
 190      470870    684698        687  5.39%  1.02%  0.97%   0 IP SNMP          
  51      241578    411613        586  4.90%  1.38%  0.61%   0 IP Input        
 191      176123    342439        514  1.87%  0.69%  0.34%   0 PDU DISPATCHER  
 115        5124       223      22977  1.79%  0.15%  0.19%   6 SSH Process      
 186       12874     76231        168  0.16%  0.08%  0.07%   0 PIM Process      
  77       66239    138423        478  0.16%  0.17%  0.15%   0 DHCPD Receive    
  54       15426     60336        255  0.16%  0.07%  0.08%   0 Skinny Msg Serve
   2       53550     11719       4569  0.08%  0.05%  0.07%   0 Load Meter      
   6         112        52       2153  0.08%  0.00%  0.00%   0 Pool Manager    
 210       15689    113660        138  0.08%  0.08%  0.06%   0 IP-EIGRP: HELLO  
  88       14599    583403         25  0.08%  0.06%  0.05%   0 RBSCP Background
  41        5029    234319         21  0.08%  0.01%  0.00%   0 e1t1 Framer back
  74         882      1329        663  0.08%  0.00%  0.00%   0 TCP Timer        
 195       42352   1815976         23  0.08%  0.12%  0.09%   0 RADIUS          
 168       26961     13172       2046  0.08%  0.12%  0.11%   0 DSMP            
  71       15307    114742        133  0.08%  0.06%  0.07%   0 CEF process      
  26        3149     33369         94  0.08%  0.00%  0.00%   0 Net Background  
 170       11202      9104       1230  0.08%  0.11%  0.09%   0 VOIP_RTCP        
  19           0         1          0  0.00%  0.00%  0.00%   0 RO Notify Timers
  18           0         3          0  0.00%  0.00%  0.00%   0 Serial Backgroun
  20           4         2       2000  0.00%  0.00%  0.00%   0 SMART            
  17         586     14648         40  0.00%  0.00%  0.00%   0 HC Counter Timer
  24           0         2          0  0.00%  0.00%  0.00%   0 XML Proxy Client
  21        1424     58589         24  0.00%  0.00%  0.00%   0 GraphIt          
  16           4        86         46  0.00%  0.00%  0.00%   0 EEM ED Syslog    
  27           8         5       1600  0.00%  0.00%  0.00%   0 IDB Work        
  22           0         2          0  0.00%  0.00%  0.00%   0 Dialer event    
  23           0         1          0  0.00%  0.00%  0.00%   0 SERIAL A'detect  
  30        4426     58607         75  0.00%  0.01%  0.00%   0 Per-Second Jobs  
  31          24       489         49  0.00%  0.00%  0.00%   0 DHCPD Timer      
  15          12         2       6000  0.00%  0.00%  0.00%   0 Entity MIB API  
  33           0         1          0  0.00%  0.00%  0.00%   0 dev_device_inser
  34           0         1          0  0.00%  0.00%  0.00%   0 dev_device_remov
  25           0         1          0  0.00%  0.00%  0.00%   0 Critical Bkgnd  
  28          36        59        610  0.00%  0.00%  0.00%   0 Logger          
  37       13795     11730       1176  0.00%  0.05%  0.07%   0 Compute load avg
  38       57121       990      57697  0.00%  0.04%  0.05%   0 Per-minute Jobs  
  39           0         1          0  0.00%  0.00%  0.00%   0 AAL2CPS TIMER_CU
  40           0         4          0  0.00%  0.00%  0.00%   0 Service-module a
  13           0         1          0  0.00%  0.00%  0.00%   0 Policy Manager  
  29        1632     58459         27  0.00%  0.00%  0.00%   0 TTY Background  
  14          20       167        119  0.00%  0.00%  0.00%   0 DDR Timers      
  44           8         2       4000  0.00%  0.00%  0.00%   0 DTP Protocol    
  45          16        62        258  0.00%  0.00%  0.00%   0 AAA Dictionary R
  46           0        22          0  0.00%  0.00%  0.00%   0 AAA Server      
  47           0         1          0  0.00%  0.00%  0.00%   0 AAA ACCT Proc    
  48           0         1          0  0.00%  0.00%  0.00%   0 ACCT Periodic Pr
  49           0         1          0  0.00%  0.00%  0.00%   0 AC Mgr          
  32           0         1          0  0.00%  0.00%  0.00%   0 AggMgr Process  
  35           0         2          0  0.00%  0.00%  0.00%   0 ESWPPM          
  52           0         1          0  0.00%  0.00%  0.00%   0 ICMP event handl
  53           4         3       1333  0.00%  0.00%  0.00%   0 PPP Hooks        
  10           0         2          0  0.00%  0.00%  0.00%   0 ATM Idle Timer  
  55           0         1          0  0.00%  0.00%  0.00%   0 SSS Manager      
  36        1901      9692        196  0.00%  0.00%  0.00%   0 Net Input        
  57           0         1          0  0.00%  0.00%  0.00%   0 SSS Feature Mana
  42          85        13       6538  0.00%  0.00%  0.00%   0 IPM DSPRM MAIN  
  59           0         1          0  0.00%  0.00%  0.00%   0 VPDN call manage
  60           0         1          0  0.00%  0.00%  0.00%   0 L2X Socket proce
  61           0         1          0  0.00%  0.00%  0.00%   0 L2X SSS manager  
  62           0         2          0  0.00%  0.00%  0.00%   0 L2TP mgmt daemon
  63           0         1          0  0.00%  0.00%  0.00%   0 X.25 Encaps Mana
  64           0         2          0  0.00%  0.00%  0.00%   0 EAPoUDP Process  
  65           0         1          0  0.00%  0.00%  0.00%   0 IPv6 RIB Redistr
  66           0         2          0  0.00%  0.00%  0.00%   0 KRB5 AAA        
  67           0         2          0  0.00%  0.00%  0.00%   0 PPP IP Route    
  68           0         2          0  0.00%  0.00%  0.00%   0 PPP IPCP        
  69         783      1767        443  0.00%  0.00%  0.00%   0 IP Background    
  70         192       997        192  0.00%  0.00%  0.00%   0 IP RIB Update    
  43           0         1          0  0.00%  0.00%  0.00%   0 L2X Data Daemon  
  72        2832     87750         32  0.00%  0.01%  0.00%   0 Socket Timers    
  73           0         4          0  0.00%  0.00%  0.00%   0 SNMP Timers      
  12           0         1          0  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT
  75           8         4       2000  0.00%  0.00%  0.00%   0 TCP Protocols    
  76           4         1       4000  0.00%  0.00%  0.00%   0 COPS            
   9        5112      6739        758  0.00%  0.01%  0.00%   0 ARP Input        
  78           0         2          0  0.00%  0.00%  0.00%   0 Dialer Forwarder
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  79         608       978        621  0.00%  0.00%  0.00%   0 Adj Manager      
  80           8       200         40  0.00%  0.00%  0.00%   0 HTTP CORE        
  81           0         1          0  0.00%  0.00%  0.00%   0 IP Traceroute    
  82          20       977         20  0.00%  0.00%  0.00%   0 IP Cache Ager    
  83           0         1          0  0.00%  0.00%  0.00%   0 RARP Input      
  84           0         1          0  0.00%  0.00%  0.00%   0 PAD InCall      
  85           0         2          0  0.00%  0.00%  0.00%   0 X.25 Background  
  86           0         2          0  0.00%  0.00%  0.00%   0 PPP Bind        
  87           0         2          0  0.00%  0.00%  0.00%   0 PPP SSS          
  50        2239      7877        284  0.00%  0.00%  0.00%   0 CDP Protocol    
  11           4        30        133  0.00%  0.00%  0.00%   0 AAA high-capacit
  56          96      7813         12  0.00%  0.00%  0.00%   0 SSS Test Client  
  91           8       979          8  0.00%  0.00%  0.00%   0 DHCPD Database  
  92           8       196         40  0.00%  0.00%  0.00%   0 Authentication P
  93           0         1          0  0.00%  0.00%  0.00%   0 Auth-proxy AAA B
  94           0         1          0  0.00%  0.00%  0.00%   0 IPS Timer        
   8           0         1          0  0.00%  0.00%  0.00%   0 Crash writer    
   7           0         2          0  0.00%  0.00%  0.00%   0 Timers          
  97           0         2          0  0.00%  0.00%  0.00%   0 URL filter proc  
  98          68      2346         28  0.00%  0.00%  0.00%   0 CRM_CALL_UPDATE_
  99           0         2          0  0.00%  0.00%  0.00%   0 ENABLE AAA      
  58        5303    228855         23  0.00%  0.02%  0.00%   0 SSS Feature Time
 101           0         1          0  0.00%  0.00%  0.00%   0 Key chain liveke
 102           0         2          0  0.00%  0.00%  0.00%   0 LINE AAA        
 103          16        18        888  0.00%  0.00%  0.00%   0 LOCAL AAA        
  89        2080     58466         35  0.00%  0.00%  0.00%   0 RUDPV1 Main Proc
 105           0         2          0  0.00%  0.00%  0.00%   0 VSP_MGR          
 106           0         1          0  0.00%  0.00%  0.00%   0 encrypt proc    
 107           0         3          0  0.00%  0.00%  0.00%   0 Crypto WUI      
 108           0         2          0  0.00%  0.00%  0.00%   0 Crypto Support  
 109         341        76       4486  0.00%  0.00%  0.00%   0 CCVPM_HTSP      
 110           0         2          0  0.00%  0.00%  0.00%   0 VPM_MWI_BACKGROU
 111           0         1          0  0.00%  0.00%  0.00%   0 CCVPM_R2        
 112           0        66          0  0.00%  0.00%  0.00%   0 FB/KS Log HouseK
 113           0         2          0  0.00%  0.00%  0.00%   0 EPHONE MWI BG Pr
 114           0         1          0  0.00%  0.00%  0.00%   0 CCSWVOICE        
   5      139394      9656      14435  0.00%  0.24%  0.21%   0 Check heaps      
 116           0         1          0  0.00%  0.00%  0.00%   0 http client proc
 117       18316      5784       3166  0.00%  0.11%  0.09%   0 CCH323_CT        
 118           0         1          0  0.00%  0.00%  0.00%   0 CCH323_DNS      
 119           8         2       4000  0.00%  0.00%  0.00%   0 CCAAL2_CT        
 120           4         2       2000  0.00%  0.00%  0.00%   0 CCFRF11_CT      
 121          32         2      16000  0.00%  0.00%  0.00%   0 CCSIP_SPI_CONTRO
 122           0         1          0  0.00%  0.00%  0.00%   0 CCSIP_DNS        
 123           4         1       4000  0.00%  0.00%  0.00%   0 CCSIP_UDP_SOCKET
 124           8         1       8000  0.00%  0.00%  0.00%   0 CCSIP_TCP_SOCKET
 125           0         1          0  0.00%  0.00%  0.00%   0 faxmsp          
 126           0         1          0  0.00%  0.00%  0.00%   0 QOS_MODULE_MAIN  
 127           0         1          0  0.00%  0.00%  0.00%   0 RPMS_PROC_MAIN  
 128           0         1          0  0.00%  0.00%  0.00%   0 VoIP AAA        
 129           0         1          0  0.00%  0.00%  0.00%   0 crypto engine pr
 130          20         4       5000  0.00%  0.00%  0.00%   0 Crypto CA        
 131           0         1          0  0.00%  0.00%  0.00%   0 Crypto PKI-CRL  
 132           0         1          0  0.00%  0.00%  0.00%   0 Crypto SSL      
 133           4         6        666  0.00%  0.00%  0.00%   0 Crypto ACL      
 134           0         1          0  0.00%  0.00%  0.00%   0 CRYPTO QoS proce
 135           0         1          0  0.00%  0.00%  0.00%   0 Crypto INT      
 136           4         3       1333  0.00%  0.00%  0.00%   0 Crypto IKMP      
 137         742      2934        252  0.00%  0.00%  0.00%   0 IPSEC key engine
 138           0         1          0  0.00%  0.00%  0.00%   0 IPSEC manual key
 139           0         1          0  0.00%  0.00%  0.00%   0 Crypto PAS Proc  
 140           4         1       4000  0.00%  0.00%  0.00%   0 Crypto Delete Ma
 141           0         1          0  0.00%  0.00%  0.00%   0 Key Proc        
 142           0         1          0  0.00%  0.00%  0.00%   0 PM Callback      
 143           0         1          0  0.00%  0.00%  0.00%   0 DATA Transfer Pr
 144           0         1          0  0.00%  0.00%  0.00%   0 DATA Collector  
 145         427       334       1278  0.00%  0.00%  0.00%   0 AAA SEND STOP EV
 147           0         3          0  0.00%  0.00%  0.00%   0 EEM ED CLI      
 148           0         2          0  0.00%  0.00%  0.00%   0 EEM ED Counter  
 149           0         2          0  0.00%  0.00%  0.00%   0 EEM ED Interface
 150           0         3          0  0.00%  0.00%  0.00%   0 EEM ED IOSWD    
 151           0         2          0  0.00%  0.00%  0.00%   0 EEM ED Memory-th
 152           0         2          0  0.00%  0.00%  0.00%   0 EEM ED None      
 153           0         2          0  0.00%  0.00%  0.00%   0 EM ED OIR        
 154           0         2          0  0.00%  0.00%  0.00%   0 EEM ED SNMP      
 155          16       979         16  0.00%  0.00%  0.00%   0 EEM ED Timer    
 156         477     12505         38  0.00%  0.00%  0.00%   0 EEM Server      
 157           0         1          0  0.00%  0.00%  0.00%   0 Syslog Traps    
 158           0         2          0  0.00%  0.00%  0.00%   0 IpSecMibTopN    
 159        1083     58395         18  0.00%  0.00%  0.00%   0 trunk conditioni
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
 160           0         1          0  0.00%  0.00%  0.00%   0 trunk conditioni
 161          56       194        288  0.00%  0.00%  0.00%   0 VLAN Manager    
 162           0         2          0  0.00%  0.00%  0.00%   0 EEM Policy Direc
 163         145        40       3625  0.00%  0.00%  0.00%   0 Syslog          
 164           0         1          0  0.00%  0.00%  0.00%   0 VPDN Scal        
 166         326      1183        275  0.00%  0.00%  0.00%   0 Resource Monitor
 167         699       933        749  0.00%  0.00%  0.00%   0 CC-API_VCM      
  90           0         1          0  0.00%  0.00%  0.00%   0 Inspect Timer    
 169        4129      2268       1820  0.00%  0.01%  0.00%   0 VTSP            
 100           0         1          0  0.00%  0.00%  0.00%   0 EM Background Pr
 171        1587       906       1751  0.00%  0.00%  0.00%   0 TSP              
 172           0         1          0  0.00%  0.00%  0.00%   0 lib_off_app      
 173           4         2       2000  0.00%  0.00%  0.00%   0 Voice Player    
 174           0         1          0  0.00%  0.00%  0.00%   0 Media Record    
 175           0         1          0  0.00%  0.00%  0.00%   0 Resource Measure
 176           4         6        666  0.00%  0.00%  0.00%   0 Session Applicat
 177          52       977         53  0.00%  0.00%  0.00%   0 RTPSPI          
 178          32        35        914  0.00%  0.00%  0.00%   0 SSH Event handle
 179        9161      1032       8876  0.00%  0.04%  0.03%   0 ISDN            
 180           0         2          0  0.00%  0.00%  0.00%   0 ISDN Timer      
 181          24       117        205  0.00%  0.00%  0.00%   0 ISDNMIB Backgrou
 182         120       298        402  0.00%  0.00%  0.00%   0 CallMIB Backgrou
 183        1569      6182        253  0.00%  0.00%  0.00%   0 ISDN From Driver
 184        7010     62129        112  0.00%  0.02%  0.01%   0 IGMP Input      
 185       16930    583437         29  0.00%  0.07%  0.07%   0 Mwheel Process  
   4           0         1          0  0.00%  0.00%  0.00%   0 EDDRI_MAIN      
 187         128      2031         63  0.00%  0.00%  0.00%   0 CEF Scanner      
 188         557     19523         28  0.00%  0.00%  0.00%   0 IPHC Admin      
   3        6562      1798       3649  0.00%  0.04%  0.01%   0 AFW_application_
  95           4         2       2000  0.00%  0.00%  0.00%   0 SDEE Management  
  96           0         1          0  0.00%  0.00%  0.00%   0 IPv6 Inspect Tim
 193           0         1          0  0.00%  0.00%  0.00%   0 SNMP ConfCopyPro
 194           0         1          0  0.00%  0.00%  0.00%   0 SNMP Traps      
   1          48        31       1548  0.00%  0.00%  0.00%   0 Chunk Manager    
 196        3997     58494         68  0.00%  0.02%  0.00%   0 Skinny Socket Se
 197        1858     58985         31  0.00%  0.00%  0.00%   0 NTP              
 198           0         1          0  0.00%  0.00%  0.00%   0 C1700_RING_LIMIT
 199        7332     11048        663  0.00%  0.02%  0.00%   0 ISDN L2 Process  
 200          80       413        193  0.00%  0.00%  0.00%   0 ISDN L3 Timer Pr
 201         768      6176        124  0.00%  0.00%  0.00%   0 ISDN L2D SRQ Pro
 202           0         1          0  0.00%  0.00%  0.00%   0 DocMSP          
 203           0        18          0  0.00%  0.00%  0.00%   0 fax timers      
 204        2210    116896         18  0.00%  0.00%  0.00%   0 esmtp client soc
 205           0         2          0  0.00%  0.00%  0.00%   0 esmtp server soc
 206           0         1          0  0.00%  0.00%  0.00%   0 Fax Onramp-VFC  
 207       66968       293     228559  0.00%  0.00%  0.01%   0 crypto sw pk pro
 208           8        22        363  0.00%  0.00%  0.00%   0 IP-EIGRP Router  
 209        5547     25401        218  0.00%  0.00%  0.00%   0 IP-EIGRP: PDM    
 104           0         2          0  0.00%  0.00%  0.00%   0 TPLUS  

 

by: ctrostPosted on 2006-09-29 at 07:11:51ID: 17627415

I did just find out that its not a process problem, but an interrupt problem.  I got this information from the Cisco Output Interpreter:
INFO: Total CPU Utilization is comprised of process and interrupt percentages.
  Total CPU Utilization: 90%
  Process Utilization: 22%
  Interrupt Utilization: 68%

WARNING: Interrupt CPU Utilization is 68%, which is very high (>60%)
CPU interrupts are primarily caused by fast switching of traffic. Interrupts are
also generated any time a character is output from the console or auxiliary ports
of a router.

 

by: jfradyPosted on 2006-09-29 at 07:17:10ID: 17627465

I understand completely.  

 

by: ctrostPosted on 2006-09-29 at 07:19:26ID: 17627477

Now I need to figure out some things for interrupts.  Currently looking into this document: http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml

let me know if you have any insight given what you know up to this point

 

by: ctrostPosted on 2006-09-29 at 08:25:41ID: 17628179

Since I've realized that static is being caused by High CPU Utilization due to Interrupts, my main question has been answered.  I am going to close this question and open a TAC request with Cisco and get the issue resolved.  Thanks jfrady for all of your quick responses and feedback.

 

by: jfradyPosted on 2006-09-29 at 08:28:53ID: 17628207

No worries.  I really wish we could have pinpointed it faster.  We were on the trail though!!  Thanks for the points and good luck with TAC!

20120131-EE-VQP-002

3 Ways to Join

30-Day Free Trial

The Experts

98% positive feedback on 31,087 answers since March 2000. angeliii is a Microsoft Most Valuable Professional for his work with MS SQL Server & Develoment.

He has also proven his knowledge of Visual Basic Programming, PHP Scripting and Oracle Databases.

The Experts

97% positive feedback on 10,752 answers since July 2000. lrmoore has more than 18 years experience in the networking industry.

The six-time Mircosoft MVPs specialties include firewalls, virtual private networking, and network management.

Testimonials

"...and excellent source for support... Kind of like having your very own IT dept." Electriciansnet

Testimonials

"I was apprehensive at signing up at first. However... it has already made my life as an IT administrator much easier." JaCrews

Testimonials

"WOW! You guys have great, active, and knowledgeable people on here." moore50

Business Clients

Business Clients

In the Press

"If you’ve got a question... Experts Exchange can supply an answer.”

In the Press

"...an invaluable aid for both IT professionals and those who require tech support."

In the Press

"where IT professionals provide quick answers on just about any topic"

Business Account Plans

Loading Advertisement...