Solved

Cisco Router Losing Wan Connection at Specific Time Intervals

Posted on 2006-06-22
51
828 Views
Last Modified: 2008-01-09
We have a remote site with a 3700 series router that loses their connection to our district office at pretty specific time intervals. They connect to a 3745 series cisco router via an Alltel T1 with Wan CSU-DSU cards. I can reboot our 3745 Router and the connection seems to come back up, but due to the pretty specific time intervals I'm wondering how this is happening. It will come back up on its own somewhere b/w 2h 4m and 2h 20m according to the logs below from the last couple days. The problem seems to me to exist inside the 3745 router, but I have no idea what to check for.

6/20/2006 3:25:44 PM 0 Server1 PING Sensor1 (PING) WARNING - Lost 30000 ms (Code: 11010)
6/20/2006 3:26:15 PM 0 Server1 PING Sensor1 (PING) DOWN - Lost 30000 ms (Code: 11010)
6/20/2006 5:31:24 PM 0 Server1 PING Sensor1 (PING) UP - OK 8 ms - Downtime: 2h 5m
6/20/2006 8:01:53 PM 0 Server1 PING Sensor1 (PING) WARNING - Lost 30000 ms (Code: 11010)
6/20/2006 8:02:24 PM 0 Server1 PING Sensor1 (PING) DOWN - Lost 30000 ms (Code: 11010)
6/20/2006 10:07:23 PM 0 Server1 PING Sensor1 (PING) UP - OK 5 ms - Downtime: 2h 4m
6/21/2006 12:22:50 AM 0 Server1 PING Sensor1 (PING) WARNING - Lost 30000 ms (Code: 11010)
6/21/2006 12:23:21 AM 0 Server1 PING Sensor1 (PING) DOWN - Lost 30000 ms (Code: 11010)
6/21/2006 1:53:01 AM -1 Could not send Admin message: Socket Error # 0  
6/21/2006 2:43:21 AM 0 Server1 PING Sensor1 (PING) UP - OK 7 ms - Downtime: 2h 19m
6/21/2006 4:58:53 AM 0 Server1 PING Sensor1 (PING) WARNING - Lost 30000 ms (Code: 11010)
6/21/2006 4:59:24 AM 0 Server1 PING Sensor1 (PING) DOWN - Lost 30000 ms (Code: 11010)
6/21/2006 7:19:22 AM 0 Server1 PING Sensor1 (PING) UP - OK 5 ms - Downtime: 2h 19m
6/21/2006 9:34:51 AM 0 Server1 PING Sensor1 (PING) WARNING - Lost 30000 ms (Code: 11010)
6/21/2006 9:35:22 AM 0 Server1 PING Sensor1 (PING) DOWN - Lost 30000 ms (Code: 11010)
6/21/2006 11:55:27 AM 0 Server1 PING Sensor1 (PING) UP - OK 7 ms - Downtime: 2h 20m
6/21/2006 2:10:58 PM 0 Server1 PING Sensor1 (PING) WARNING - Lost 30000 ms (Code: 11010)
6/21/2006 2:11:29 PM 0 Server1 PING Sensor1 (PING) DOWN - Lost 30000 ms (Code: 11010)
6/21/2006 4:31:31 PM 0 Server1 PING Sensor1 (PING) UP - OK 8 ms - Downtime: 2h 20m
6/21/2006 6:47:02 PM 0 Server1 PING Sensor1 (PING) WARNING - Lost 30000 ms (Code: 11010)
6/21/2006 6:47:33 PM 0 Server1 PING Sensor1 (PING) DOWN - Lost 30000 ms (Code: 11010)
6/21/2006 8:52:31 PM 0 Server1 PING Sensor1 (PING) UP - OK 9 ms - Downtime: 2h 4m
6/21/2006 11:08:01 PM 0 Server1 PING Sensor1 (PING) WARNING - Lost 30000 ms (Code: 11010)
6/21/2006 11:08:32 PM 0 Server1 PING Sensor1 (PING) DOWN - Lost 30000 ms (Code: 11010)
6/22/2006 1:28:31 AM 0 Server1 PING Sensor1 (PING) UP - OK 8 ms - Downtime: 2h 19m
6/22/2006 1:53:09 AM -1 Could not send Admin message: Socket Error # 0  
6/22/2006 3:44:01 AM 0 Server1 PING Sensor1 (PING) WARNING - Lost 30000 ms (Code: 11010)
6/22/2006 3:44:32 AM 0 Server1 PING Sensor1 (PING) DOWN - Lost 30000 ms (Code: 11010)
6/22/2006 6:04:31 AM 0 Server1 PING Sensor1 (PING) UP - OK 6 ms - Downtime: 2h 19m
6/22/2006 7:20:01 AM 0 Server1 PING Sensor1 (PING) WARNING - Lost 30000 ms (Code: 11010)
6/22/2006 7:20:02 AM 0 Server1 PING Sensor1 (PING) UP - OK 5 ms
6/22/2006 8:20:50 AM 0 Server1 PING Sensor1 (PING) WARNING - Lost 30000 ms (Code: 11010)
6/22/2006 8:21:21 AM 0 Server1 PING Sensor1 (PING) DOWN - Lost 30000 ms (Code: 11010)
6/22/2006 10:41:19 AM 0 Server1 PING Sensor1 (PING) UP - OK 5 ms - Downtime: 2h 19m
6/22/2006 12:56:49 PM 0 Server1 PING Sensor1 (PING) WARNING - Lost 30000 ms (Code: 11010)
6/22/2006 12:57:20 PM 0 Server1 PING Sensor1 (PING) DOWN - Lost 30000 ms (Code: 11010)
0
Comment
Question by:grevels
  • 26
  • 12
  • 7
  • +2
51 Comments
 

Author Comment

by:grevels
ID: 16962207
Oh, and I've had Alltel check the T1 line, replace their cards, and I've replaced the WIC CSU/DSU cards on both routers.
0
 
LVL 12

Expert Comment

by:Scotty_cisco
ID: 16962489
who is providing clocking on the circuit and what are the clocks set to on the router?

Thanks
scott
0
 

Author Comment

by:grevels
ID: 16962635
I'm not sure what you mean by who is providing clocking on the circuit. The circuit is through Alltel. I checked the clocks on the router and the district router was 1 hour behind and the panther router was set for the wrong time back in 1993. I changed both clocks to match the current time. If everything holds correctly, the line would go back down around 330 - 350pm today.
0
 
LVL 12

Expert Comment

by:Scotty_cisco
ID: 16962869
When I am reffering to a clock I am reffering to a circuit clock network timming.

do you have any clock source in the router?
How far apart are these 2 locations?

Thanks
Scott
0
 

Author Comment

by:grevels
ID: 16962934
All references to clock in the running config on our district router are as follows:
no network-clock-participate slot 4
ntp clock-period 17175391
ntp server 65.211.109.1 prefer
ntp server 204.74.68.55
ntp server 208.21.108.186

In the remote site (Panther) there are no references to any ntp servers or a network clock.

The two locations are only a couple miles apart.
0
 
LVL 12

Expert Comment

by:Scotty_cisco
ID: 16962953
what slot is the serial connection in?  post a show run if you can

Thanks
Scott
0
 

Author Comment

by:grevels
ID: 16963092
Serial 0/1, it's the one with Description Panther. Here's the running config.

r3745-district#show run                      
Building configuration...ls                     ml

Current configuration : 2986 by                              
version 12.3            
service timestamps debug datetime msecion Unit (MTU)                        
service timestamps log datetime msec              
  netbios            
service password-encryptionccess list or enable      
!
                       
!
boot-start-marker                
boot system flash flash:c3745-ik9o3s-mz.123-9.bin
  no                      Negate a command or s
boot-end-marker              
!
logging buffered 8092 debugging          
  nrzi-encoding    
                                       
  ntp
!
           
  priority-group          Assign a prior
clock timezone EST -5ace                  
no network-clock-participate slot 4          
  pulse-time          
no aaa new-modelduri            
ip subnet-zero              
!
!

interface FastEthernet0/0                        
 no ip addressty            
 duplex autority group  
 speed auto          
!
interface FastEthernet0/0.1c                   Configu
 encapsulation dot1Q 1 native                            
 ip address 20.51.154.1 255.255.255.0 secondary       serial interface commands              
 ip address 20.51.156.1 255.255.255.0 secondaryodule          Service module configuration sub
 ip address 20.51.157.1 255.255.255.0 secondary                          
  service-policy    
 ip address 20.51.158.1 255.255.255.0 secondary                                        
  shut
 ip address 179.149.30.1 255.255.255.0 secondaryce                                              
 ip address 20.51.155.1 255.                          

interface Serial0/0  sna              
 description
 ip address 20.51.146.25 255.255.255.252                                        
 no fair-queue            

!s
interface FastEthernet0/1ify SNMP interface parame
 no ip address              
 speed 100          
 full-duplex  source    
!
interface Serial0/1
 description Panther                    
 ip address 20.51.146.33 255.255.255.252s                                      
!
interface Serial0/2                 De
 description                                  
 ip address 20.51.146.29 255.255.255.252-shape           Enable Traffic Shaping
!
interface FastEthernet4/0                        
 no ip address              
 duplex auto            
 speed autorface      
!
interface Serial4/0          
       
!
interface FastEthernet4/1  
  trunk-group        
 description PIX Firewall FIRNn a trunk group              
 ip address 202.178.220.1 255.255.255.0            
  tx-ring-limit          
 ip nbar protocol-discovery ring limit                
 speed 100          
 full-duplex            
!

interface Serial4/1      Virtual Priva
 description                                  
 ip address 20.51.146.5 255.255.255.252rface configuration commands          
 encapsulation ppp                  
!
no ip http server5-district(config
no ip http secure-server              
r3745-dis
ip classless#exit      
ip route 0.0.0.0 0.0.0.0 192.168.220.2w int serial0/1                      
ip route 20.51.144.0 255.255.255.0 Serial0/0up                                    
  Har
ip route 20.51.149.0 255.255.255.0 Serial0/2                                      
  Enc
ip route 20.51.150.0 255.255.255.0 Serial4/1                          
  Keepalive set (
ip route 20.51.151.0 255.255.255.0 Serial0/100:00:00, output 00:00:06, output hang never
ip route 20.51.152.0 255.255.255.0 Serial0/0            
  Last clearing of "show interf
ip route 20.51.153.0 255.255.255.0 Serial0/0                        
  Input queue: 0/75
!/
!(
!z
tftp-server flash:c3550-i5q3l2-mz.121-9.EA1c.bin                                                
!
!
 
 transport preferred all
 transport input all
 transport output all
!
ntp clock-period 17175395
ntp server 65.211.109.1 prefer
ntp server 204.74.68.55
ntp server 208.21.108.186
end
0
 
LVL 12

Expert Comment

by:Scotty_cisco
ID: 16963133
ok so does this have an external CSU/DSU and what is the clock on that unit set for internal or network?

Thanks
Scott
0
 

Author Comment

by:grevels
ID: 16963196
It has a Cisco WIC CSU/DSU add-on card plugged into the router. That is interface serial 0/1. A cable runs from there to a card provided by Alltel, then onto the circuit to Panther I guess where there is another card provided by alltel. Then we have a cable plugged into that card running into another Cisco WIC CSU/DSU add-on card plugged into that router. I'm confused what you mean by external CSU/DSU. We don't control anything between the two routers, it is all Alltel and all of the router config is there.

Could you clarify what you mean or need to know, I'm just not sure. I can get around a little bit on Cisco, but that's it, I can't say I know all the terms or where to look for everything.

Thanks.
0
 
LVL 12

Expert Comment

by:Scotty_cisco
ID: 16963240
the serial interface then has a wic-1dsu-1t in both sides.  have you called alltel and asked them if they are providing clock for the circuit?  I would do that if they are then we need to looks somewhere else, if not then we need to at a clock source line.

Thanks
scott
0
 
LVL 12

Expert Comment

by:Scotty_cisco
ID: 16963250
or clock source excuse me and clock ect it is what keeps the circuit in step from end to end.

Thanks
scott
0
 

Author Comment

by:grevels
ID: 16963322
The guy at Alltel said he's fairly certain it's an internal clock, but he's not positive. That mean anything to you?
0
 

Author Comment

by:grevels
ID: 16963345
Also, I might add that we have 5 sites using T1's remotely in the same setup and from what I can tell the configuration is pretty standard across the board.
0
 
LVL 12

Expert Comment

by:Scotty_cisco
ID: 16963374
the clocking is usually provided by the telco but they have to tell you.  What the framing and linecode is but sounds like he is saying that he is providing clock for the network which means that you would not need a clock.  Is this a full T1 or a fractional?

Thanks
Scott
0
 

Author Comment

by:grevels
ID: 16963506
full
0
 

Author Comment

by:grevels
ID: 16963528
I rebooted this router here when they went down last time and it brought them right up. Previously it would bring itself back up in that 2h4m-2h20m range I told you about. They have now been up for 2h32m since I rebooted our district office router.
0
 
LVL 12

Expert Comment

by:Scotty_cisco
ID: 16963547
weird internal clock may have been screwed up .... this sounds like a clock slip issue could be wrong but I doubt it I have seen happen just like this.

Thanks
scott
0
 

Author Comment

by:grevels
ID: 16963589
This has been an ongoing problem for a while. We had gotten it mostly under control, they would only go down probably once a week for a short period of time, but now it's getting bad again. We have rebooted the router in the past to fix it, but the problem just comes back over time. We'll see if changing the clocks on the routers may fix it, I'm fixing to leave work in a few and will be gone until Monday so if you come up with anymore suggestions just put them up here and I'll try them first thing Monday and give you a status report of how it acted over the weekend.

Thanks for your help.
0
 
LVL 12

Expert Comment

by:Scotty_cisco
ID: 16963701
when I am talking clock has nothing to do with the time clock on the router but everything to do with the clock on the circuit.  So  I doubt that changing the date time on the router will help at all.  Have you replaced the cables between the altel dmarc and the routers on both ends?

Thanks
Scott
0
 
LVL 79

Expert Comment

by:lrmoore
ID: 16965081
Assuming that this is a point-to-point T1 circuit, take a look at the results of
show interface serial 0/1
  Look at CRC and frame errors. Do they increase and are they significant?

show service-module serial 0/1
  Look at error counters

On the Interface Panther, try this:
 interface serial 0/1
  service-module t1 clock source internal  

That will change the clocking source from default "line" which you can't see in the config simply because it is default, to it's own internal clock, and then it will become the clocking source for the other end to syncronize the T1 modulation against. Now they will both be timing off the same clock source.


0
 
LVL 6

Expert Comment

by:fullerms
ID: 16969119
You can also try to paste the "Show controller Serial 0/1" output. That usually provides information on your clocking. If you can paste that output from both ends, it can help in isolating the issue. If your link is on a different port, please substitute "0/1" with the actual port number

Hope this helps.
0
 

Author Comment

by:grevels
ID: 16983187
Well, so far since I rebooted our district office router on Thursday, the line hasn't gone down again. This is not necessarily uncommon though, generally a reboot will fix the line for several days, then it slowly gets worse again. I pasted the 3 commands I was asked to run down below. Let me know if any of you still think I need to change the clocking source to internal after viewing the output. Thanks again for all the help.

Here is the show interface serial0/1 command ran today, after the router has been up continuous for 4 days:

Serial0/1 is up, line protocol is up                                    
  Hardware is GT96K with integrated T1 CSU/DSU                                              
  Description:
  Internet address is X.X.X.X/30                                    
  MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation HDLC, loopback not set
  Keepalive set (10 sec)
  Last input 00:00:00, output 00:00:03, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/10/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1158 kilobits/sec
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     1130293 packets input, 1358131404 bytes, 0 no buffer
     Received 37956 broadcasts, 0 runts, 1 giants, 0 throttles
     1 input errors, 1 CRC, 0 frame, 0 overrun, 0 ignored, 1 abort
     762623 packets output, 198675772 bytes, 0 underruns
     0 output errors, 0 collisions, 6 interface resets
     0 output buffer failures, 0 output buffers swapped out
     3 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up



Here is the show service-module serial0/1 command, also ran after 4 days of the line being up:

Module type is T1/fractional
    Hardware revision is 0.52, Software revision is 20030528,
    Image checksum is 0x3E664E, Protocol revision is 0.1
Receiver has no alarms.
Framing is ESF, Line Code is B8ZS, Current clock source is line,
Fraction has 24 timeslots (64 Kbits/sec each), Net bandwidth is 1536 Kbits/sec.
Last module self-test (done at startup): Passed
Last clearing of alarm counters 3d18h
    loss of signal        :    0,
    loss of frame         :    1, last occurred 2d10h
    AIS alarm             :    1, last occurred 2d10h
    Remote alarm          :    1, last occurred 3d18h
    Module access errors  :    0,
Total Data (last 96 15 minute intervals):
    0 Line Code Violations, 0 Path Code Violations
    0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
    0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
Data in current interval (384 seconds elapsed):
    0 Line Code Violations, 0 Path Code Violations
    0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
    0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs



Here is the show controller serial0/1 command, ran after 4 successful days:

Interface Serial0/1                  
Hardware is GT96K with Integrated FT1 CSU/DSU module                                                    
 TX and RX clocks detected.                          
idb at 0x63A2CA98, driver data structure at 0x63A3482C                                                      
wic_info 0x63A34D88                  
Physical Port 4, SCC Num 4                          
MPSC Registers:              
MMCR_L=0x000300C0, MMCR_H=0x00000000, MPCR=0x00000000                                                    
CHR1=0x00FE007E, CHR2=0x00000000, CHR3=0x00000648, CHR4=0x00000000                                                                  
CHR5=0x00000000, CHR6=0x00000000, CHR7=0x00000000, CHR8=0x00000000                                                                  
CHR9=0x00000000, CHR10=0x00003008, CHR11=0x0083F83C                                                  
SDMA Registers:              
SDC=0x00003201, SDCM=0x00000080, SGC=0x00000000                                              
CRDP=0x076D9080, CTDP=0x076D9360, FTDB=0x076D9360                                                
Main Routing Register=0x00FC0FC7 BRG Conf Register=0x00480000                                                            
Rx Clk Routing Register=0x00880080 Tx Clk Routing Register=0x00990090                                                                    
GPP Registers:              
Conf=0x3030000 , Io=0x6060000 , Data=0x7B7B7F7F, Level=0x18180000                                                                
Conf0=0x3000003 , Io0=0x6004008 , Data0=0x3B7F8FD3, Level0=0x18000000                                                                    
TDM FPGA Registers:                  
frr = 0x47, vmcra[0] = 0x4, vmcra[1] = 0x4, vmcra[2] = 0x4                                                          
vmcrb[0] = 0x7, vmcrb[1] = 0x7, vmcrb[2] = 0x7                                              
nmtcr[0] = 0xFE, nmtcr[1] = 0xFE, nmtcr[2] = 0xFE, nmtcr[3] = 0xFE                                                                  
nhr[0] = 0x8C, isr = 0x0, imr = 0x0, dtr = 0x0                                              
1 input aborts on receiving flag sequence                                        
0 throttles, 0 enables                      
0 overruns          
0 transmitter underruns                      
0 transmitter CTS losts                      
1130417 rxintr, 762391 txintr, 0 rxerr, 0 txerr                                              
0 mpsc_rx, 0 mpsc_rxerr, 0 mpsc_rlsc, 0 mpsc_rhnt, 0 mpsc_rfsc                                                              
0 mpsc_rcsc, 0 mpsc_rovr, 0 mpsc_rcdl, 0 mpsc_rckg, 0 mpsc_bper                                                              
0 mpsc_txerr, 0 mpsc_teidl, 0 mpsc_tudr, 0 mpsc_tctsl, 0 mpsc_tckg                                                                  
0 sdma_rx_sf, 1 sdma_rx_mfl, 0 sdma_rx                                    
0 sdma_rx_de, 0 sdma_rx_cdl, 1 sdma_rx_ce, 0 sdma_tx_rl, 0 sdma_tx_ur                                                                    
0 sdma_rx_reserr, 0 sdma_tx_reserr                                  

tx_limited = 1(2), errata19 count1 - 0, count2 - 0                                                  
Receive Ring            
rxr head (1)(0x076D9090), rxr tail (0)(0x076D9080)                                                  
  rmd(76D9080): nbd 76D9090 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76E0148                                                                            
  rmd(76D9090): nbd 76D90A0 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76E0E08                                                                            
  rmd(76D90A0): nbd 76D90B0 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76E1468                                                                            
  rmd(76D90B0): nbd 76D90C0 cmd_sts 80800000 buf_sz 06000000 buf_                                                              
  rmd(76D90C0): nbd 76D90D0 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76E2DE8                                                                            
  rmd(76D90D0): nbd 76D90E0 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76E2788                                                                            
  rmd(76D90E0): nbd 76D90F0 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76E3448                                                                            
  rmd(76D90F0): nbd 76D9100 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76E3AA8                                                                            
  rmd(76D9100): nbd 76D9110 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76E4108                                                                            
  rmd(76D9110): nbd 76D9120 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76E4768                                                                            
  rmd(76D9120): nbd 76D9130 cmd_sts 80800000 buf_sz 0                                                    
  rmd(76D9130): nbd 76D9140 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76E5428                                                                            
  rmd(76D9140): nbd 76D9150 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76E5A88                                                                            
  rmd(76D9150): nbd 76D9160 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76E60E8                                                                            
  rmd(76D9160): nbd 76D9170 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76E6748                                                                            
  rmd(76D9170): nbd 76D9180 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76D9B48                                                                            
  rmd(76D9180): nbd 76D9190 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76DA1A8                                                                            
  rmd(76D9190): nbd 76D91A0 cmd_sts 80800000 buf_sz 0                                                    
  rmd(76D91A0): nbd 76D91B0 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76DAE68                                                                            
  rmd(76D91B0): nbd 76D91C0 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76DB4C8                                                                            
  rmd(76D91C0): nbd 76D91D0 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76E2128                                                                            
  rmd(76D91D0): nbd 76D91E0 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76DBB28                                                                            
  rmd(76D91E0): nbd 76D91F0 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76DC188                                                                            
  rmd(76D91F0): nbd 76D9200 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76DC7E8                                                                            
  rmd(76D9200): nbd 76D9210 cmd_sts 80800000 buf_sz 0                                                    
  rmd(76D9210): nbd 76D9220 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76DD4A8                                                                            
  rmd(76D9220): nbd 76D9230 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76DDB08                                                                            
  rmd(76D9230): nbd 76D9240 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76DE168                                                                            
  rmd(76D9240): nbd 76D9250 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76DE7C8                                                                            
  rmd(76D9250): nbd 76D9260 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76DEE28                                                                            
  rmd(76D9260): nbd 76D9270 cmd_sts 80800000 buf_sz 06000000 buf_ptr 76DF488                                                                            
  rmd(76D9270): nbd 76D9080 cmd_sts 80800000 buf_sz 0                                                    
Transmit Ring            
txr head (11)(0x076D9370), txr tail (11)(0x076D9370)                                                    
  tmd(76D92C0): nbd 76D92D0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77B544A                                                                            
  tmd(76D92D0): nbd 76D92E0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77E2E54                                                                            
  tmd(76D92E0): nbd 76D92F0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 7300FD4                                                                            
  tmd(76D92F0): nbd 76D9300 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 7300994                                                                            
  tmd(76D9300): nbd 76D9310 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77E2D14                                                                            
  tmd(76D9310): nbd 76D9320 cmd_sts 00C30000 byt_cnt ABCDAB                                                        
  tmd(76D9320): nbd 76D9330 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 73014D4                                                                            
  tmd(76D9330): nbd 76D9340 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77E7314                                                                            
  tmd(76D9340): nbd 76D9350 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 779ADCA                                                                            
  tmd(76D9350): nbd 76D9360 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 7301C54                                                                            
  tmd(76D9360): nbd 76D9370 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77AF94A                                                                            
  tmd(76D9370): nbd 76D9380 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 74E408A                                                                            
  tmd(76D9380): nbd 76D9390 cmd_sts 00C30000 by                                              
  tmd(76D9390): nbd 76D93A0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 754790A                                                                            
  tmd(76D93A0): nbd 76D93B0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77E2454                                                                            
  tmd(76D93B0): nbd 76D93C0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 7300714                                                                            
  tmd(76D93C0): nbd 76D93D0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77E2814                                                                            
  tmd(76D93D0): nbd 76D93E0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 7300D54                                                                            
  tmd(76D93E0): nbd 76D93F0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77E1F54                                                                            
  tmd(76D93F0): nbd 76D9400 cmd_sts 00C30000 by                                              
  tmd(76D9400): nbd 76D9410 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77E2BD4                                                                            
  tmd(76D9410): nbd 76D9420 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 74E5A8A                                                                            
  tmd(76D9420): nbd 76D9430 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 755768A                                                                            
  tmd(76D9430): nbd 76D9440 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 74E130A                                                                            
  tmd(76D9440): nbd 76D9450 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 74E4D8A                                                                            
  tmd(76D9450): nbd 76D9460 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 74DF90A                                                                            
  tmd(76D9460): nbd 76D9470 cmd_sts 00C30000 by                                              
  tmd(76D9470): nbd 76D9480 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 74E198A                                                                            
  tmd(76D9480): nbd 76D9490 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 74EFD0A                                                                            
  tmd(76D9490): nbd 76D94A0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 74F5E8A                                                                            
  tmd(76D94A0): nbd 76D94B0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 74EF00A                                                                            
  tmd(76D94B0): nbd 76D94C0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 754CD8A                                                                            
  tmd(76D94C0): nbd 76D94D0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 754DA8A                                                                            
  tmd(76D94D0): nbd 76D94E0 cmd_sts 00C30000 by                                              
  tmd(76D94E0): nbd 76D94F0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 754C70A                                                                            
  tmd(76D94F0): nbd 76D9500 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 74E818A                                                                            
  tmd(76D9500): nbd 76D9510 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 755080A                                                                            
  tmd(76D9510): nbd 76D9520 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 74E198A                                                                            
  tmd(76D9520): nbd 76D9530 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 754658A                                                                            
  tmd(76D9530): nbd 76D9540 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 754DA8A                                                                            
  tmd(76D9540): nbd 76D9550 cmd_sts 00C30000 by                                              
  tmd(76D9550): nbd 76D9560 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 7551B8A                                                                            
  tmd(76D9560): nbd 76D9570 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 74EAF0A                                                                            
  tmd(76D9570): nbd 76D9580 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 754F48A                                                                            
  tmd(76D9580): nbd 76D9590 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 74DF90A                                                                            
  tmd(76D9590): nbd 76D95A0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 74EFD0A                                                                            
  tmd(76D95A0): nbd 76D95B0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 755700A                                                                            
  tmd(76D95B0): nbd 76D95C0 cmd_sts 00C30000 by                                              
  tmd(76D95C0): nbd 76D95D0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 74F0A0A                                                                            
  tmd(76D95D0): nbd 76D95E0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 74E950A                                                                            
  tmd(76D95E0): nbd 76D95F0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 74EBC0A                                                                            
  tmd(76D95F0): nbd 76D9600 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 7300FD4                                                                            
  tmd(76D9600): nbd 76D9610 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77E3214                                                                            
  tmd(76D9610): nbd 76D9620 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77E21D4                                                                            
  tmd(76D9620): nbd 76D9630 cmd_sts 00C30000 by                                              
  tmd(76D9630): nbd 76D9640 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 7301254                                                                            
  tmd(76D9640): nbd 76D9650 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77EA334                                                                            
  tmd(76D9650): nbd 76D9660 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 73019D4                                                                            
  tmd(76D9660): nbd 76D9670 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77E2954                                                                            
  tmd(76D9670): nbd 76D9680 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77E1E14                                                                            
  tmd(76D9680): nbd 76D9690 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 7300E94                                                                            
  tmd(76D9690): nbd 76D96A0 cmd_sts 00C30000 by                                              
  tmd(76D96A0): nbd 76D96B0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77E2094                                                                            
  tmd(76D96B0): nbd 76D96C0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77F1814                                                                            
  tmd(76D96C0): nbd 76D96D0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 7300854                                                                            
  tmd(76D96D0): nbd 76D96E0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 7301C54                                                                            
  tmd(76D96E0): nbd 76D96F0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77E2954                                                                            
  tmd(76D96F0): nbd 76D9700 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77E2594                                                                            
  tmd(76D9700): nbd 76D9710 cmd_sts 00C30000 by                                              
  tmd(76D9710): nbd 76D9720 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77E3494                                                                            
  tmd(76D9720): nbd 76D9730 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77F25D4                                                                            
  tmd(76D9730): nbd 76D9740 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 7300E94                                                                            
  tmd(76D9740): nbd 76D9750 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77E21D4                                                                            
  tmd(76D9750): nbd 76D9760 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77E21D4                                                                            
  tmd(76D9760): nbd 76D9770 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77E3214                                                                            
  tmd(76D9770): nbd 76D9780 cmd_sts 00C30000 by                                              
  tmd(76D9780): nbd 76D9790 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77AEC4A                                                                            
  tmd(76D9790): nbd 76D97A0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 779FBCA                                                                            
  tmd(76D97A0): nbd 76D97B0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77B8ECA                                                                            
  tmd(76D97B0): nbd 76D97C0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77A5D4A                                                                            
  tmd(76D97C0): nbd 76D97D0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77A9E4A                                                                            
  tmd(76D97D0): nbd 76D97E0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 779A0CA                                                                            
  tmd(76D97E0): nbd 76D97F0 cmd_sts 00C30000 by                                              
  tmd(76D97F0): nbd 76D9800 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 7554F8A                                                                            
  tmd(76D9800): nbd 76D9810 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 754520A                                                                            
  tmd(76D9810): nbd 76D9820 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 755768A                                                                            
  tmd(76D9820): nbd 76D9830 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77A5D4A                                                                            
  tmd(76D9830): nbd 76D9840 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77A9E4A                                                                            
  tmd(76D9840): nbd 76D9850 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 779A0CA                                                                            
  tmd(76D9850): nbd 76D9860 cmd_sts 00C30000 by                                              
  tmd(76D9860): nbd 76D9870 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77B884A                                                                            
  tmd(76D9870): nbd 76D9880 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77E2E54                                                                            
  tmd(76D9880): nbd 76D9890 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77AE5CA                                                                            
  tmd(76D9890): nbd 76D98A0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 779FBCA                                                                            
  tmd(76D98A0): nbd 76D98B0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 779DB4A                                                                            
  tmd(76D98B0): nbd 76D98C0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77B2D4A                                                                            
  tmd(76D98C0): nbd 76D98D0 cmd_sts 00C30000 by                                              
  tmd(76D98D0): nbd 76D98E0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77C0A4A                                                                            
  tmd(76D98E0): nbd 76D98F0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77BC2CA                                                                            
  tmd(76D98F0): nbd 76D9900 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77AB84A                                                                            
  tmd(76D9900): nbd 76D9910 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77B544A                                                                            
  tmd(76D9910): nbd 76D9920 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77C244A                                                                            
  tmd(76D9920): nbd 76D9930 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77A70CA                                                                            
  tmd(76D9930): nbd 76D9940 cmd_sts 00C30000 by                                              
  tmd(76D9940): nbd 76D9950 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 779C14A                                                                            
  tmd(76D9950): nbd 76D9960 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77BDCCA                                                                            
  tmd(76D9960): nbd 76D9970 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77B2D4A                                                                            
  tmd(76D9970): nbd 76D9980 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77A8ACA  
  tmd(76D9980): nbd 76D9990 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 779EECA
  tmd(76D9990): nbd 76D99A0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77C4B4A
  tmd(76D99A0): nbd 76D99B0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 779C14A
  tmd(76D99B0): nbd 76D99C0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77E2314
  tmd(76D99C0): nbd 76D99D0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 779F54A
  tmd(76D99D0): nbd 76D99E0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77C1DCA
  tmd(76D99E0): nbd 76D99F0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 7795FCA
  tmd(76D99F0): nbd 76D9A00 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 779A0CA
  tmd(76D9A00): nbd 76D9A10 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77B8ECA
  tmd(76D9A10): nbd 76D9A20 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 779E84A
  tmd(76D9A20): nbd 76D9A30 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 7798D4A
  tmd(76D9A30): nbd 76D9A40 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77F8614
  tmd(76D9A40): nbd 76D9A50 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 7794C4A
  tmd(76D9A50): nbd 76D9A60 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77B74CA
  tmd(76D9A60): nbd 76D9A70 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77BD64A
  tmd(76D9A70): nbd 76D9A80 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77A6A4A
  tmd(76D9A80): nbd 76D9A90 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77AF94A
  tmd(76D9A90): nbd 76D9AA0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 779D4CA
  tmd(76D9AA0): nbd 76D9AB0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77B6E4A
  tmd(76D9AB0): nbd 76D92C0 cmd_sts 00C30000 byt_cnt ABCDABCD buf_ptr 77C0A4A

buffer size 1524
0
 
LVL 79

Expert Comment

by:lrmoore
ID: 16983259
> 1 input errors, 1 CRC, 0 frame, 0 overrun, 0 ignored, 1 abort
This initial error is probably just from restarting the router, but keep an eye on these error counters (from show interface command). If they continue to increase it is a sure sign that you have 1 of 3 things happening:

1) circuit timing issue. One side needs to provide T1 clocking for the other side. Use the command
  service-module t1 clock source internal
 on one side only

2) bad cabling between the DSU and the smart jack at your end
3) bad cabling between the DSU and the smart jack at the remote end

I'm betting that the error counts will increase slowly at first, then much faster after a couple more days.

0
 
LVL 12

Expert Comment

by:Scotty_cisco
ID: 16984629
The timing issue keeps coming about...... Grevels has talked to the telco they are claiming that they are providing clock.  I would still try clocking one side and see if that fixes the issue it just sounds like a typical clock slip issue.

Thanks
Scott
0
 

Author Comment

by:grevels
ID: 16993375
The show interface is still showing the same thing so far and we haven't went down again yet...

1 input errors, 1 CRC, 0 frame, 0 overrun, 0 ignored, 1 abort
0
Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

 

Author Comment

by:grevels
ID: 17052976
Guys, the line still hasn't went down, but the errors did start showing up like you mentioned they may lrmoore. Does this mean the line may go down again soon?? I'm not sure as to what exactly all of these errors mean in the grand scheme of things. I never changed the config to where I made one of the routers do the clocking.

100 input errors, 100 CRC, 24 frame, 12 overrun, 0 ignored, 45 abort

Thanks for all the help so far.
0
 
LVL 12

Expert Comment

by:Scotty_cisco
ID: 17052993
have you tried to replace the cables between the routers and the TELCO demarc's?

Thanks
Scott
0
 
LVL 79

Expert Comment

by:lrmoore
ID: 17057478
If the errors start increasing faster and faster it is doomed to go down again. Did you change the clock source for this line as suggested?
0
 

Author Comment

by:grevels
ID: 17072656
Well, it happened, they went down Saturday night at 11:18pm and stayed down for the 2:18m, which is pretty well in line with the amount of downtime they were having before. Here's what that line of the show interface serial0/1 command shows now.

129342 input errors, 129341 CRC, 55389 frame, 26876 overrun, 0 ignored, 80704 abort.

I never changed the clock source as was suggested, but I am going to do it now. Would it be recommended to change it on our main, central router here at the district office or just change it on the router at the site that is going down??
0
 

Author Comment

by:grevels
ID: 17072661
Scotty, we have replaced all cables and hardware except for the actual routers.
0
 
LVL 12

Expert Comment

by:Scotty_cisco
ID: 17073298
The next step I would take you are already taking with the clock source... then I would make a loopback plug and run through the hardware to test if the there are any bad equipment.

Thanks
scott
0
 

Author Comment

by:grevels
ID: 17074032
I changed the clock source on our Panther router by adding the service-module t1 clock source internal command about an hour ago. I saved the config and rebooted the router. They just went down again, after 2h 16m of uptime, as usual.

Scotty, would you still suspect bad equipment if rebooting the router here at the district office fixes the problem for almost 2 weeks?
0
 
LVL 6

Expert Comment

by:fullerms
ID: 17078287
What do you see in your logs? Any entries which ring a bell? If you haven't done it yet, it would be a good idea to setup a syslog server and collect the logs. If that is not possible, enable "logging buffered 16384 debugging" where 16384 is the amount of buffer memory, and is dependant on your router's specs.

Collect the logs and post the entries which come in at the time the link goes down.
0
 

Author Comment

by:grevels
ID: 17081046
Logging was already set on the router with 8096 of buffer memory allocated. The line is up right now so I will check when it goes down and see if the contents change. Is there another way to view this in more detail?

r3745-district#show logging history
Syslog History Table:1 maximum table entries,
saving level warnings or higher
 83 messages ignored, 10 dropped, 0 recursion drops
 48750 table entries flushed
 SNMP notifications not enabled
   entry number 48751 : FAN-3-FAN_FAILED
    Fan 3 had a rotation error reported.
    timestamp: 162609539

0
 

Author Comment

by:grevels
ID: 17084792
The logs haven't changed. We fixed the fans on the router, fan 3 is coming loose and scraping the metal, thus the rotation problem. I'm still looking for some more ideas though.
0
 

Author Comment

by:grevels
ID: 17093346
Here's the log since we fixed the fans...

Jul 11 21:21:21.738: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1,                                                                              
changed state to up                  
Jul 11 23:38:00.099: %LINK-3-UPDOWN: Interface Serial0/1, changed state to down                                                                              
Jul 11 23:38:01.099: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1,                                                                              
changed state to down                    
Jul 12 01:54:41.484: %LINK-3-UPDOWN: Interface Serial0/1, changed state to up  
Jul 12 01:54:42.484: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1,
changed state to up
Jul 12 04:11:20.778: %LINK-3-UPDOWN: Interface Serial0/1, changed state to down
Jul 12 04:11:21.778: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1,
changed state to down
Jul 12 06:28:02.171: %LINK-3-UPDOWN: Interface Serial0/1, changed state to up
Jul 12 06:28:03.170: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1,
changed state to up
Jul 12 08:44:41.566: %LINK-3-UPDOWN: Interface Serial0/1, changed state to down
Jul 12 08:44:42.566: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1,
changed state to down
Jul 12 11:01:22.931: %LINK-3-UPDOWN: Interface Serial0/1, changed state to up
Jul 12 11:01:23.931: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1,
changed state to up
Jul 12 13:18:02.206: %LINK-3-UPDOWN: Interface Serial0/1, changed state to down
Jul 12 13:18:03.206: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1,
changed state to down
Jul 12 15:34:43.590: %LINK-3-UPDOWN: Interface Serial0/1, changed state to up
Jul 12 15:34:44.589: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1,
changed state to up
Jul 12 17:51:22.792: %LINK-3-UPDOWN: Interface Serial0/1, changed state to down
Jul 12 17:51:23.792: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1,
changed state to down
0
 
LVL 6

Expert Comment

by:fullerms
ID: 17096371
Did you have a chance to observe the CPU / Interface utilization when the link flaps? Netflow give out any clues? You have enabled NBAR and traffic shaping also. Can you disable NBAR and check? NBAR could cause issues in high traffic situations.
0
 

Author Comment

by:grevels
ID: 17099832
Fullerms,
I'm not sure how to check any of the stuff you mentioned. I didn't set the router up and I really can't say I know much more than basic stuff about cisco. If you can give me the commands I can post the info for you though. Where do you see that traffic shaping and NBAR are enabled??

The link that is going up/down has probably the least amount of traffic of any of the remote sites we have. I could rule out traffic on their side causing it because all of their internal stuff is working fine and there is no problem with traffic on our side that seems like it could cause that entire interface to go down without affecting anything else here. It seems everything works fine other than the line just going up/down, there are no noticeable causes in our network as a whole before it goes down or while it is down other than they just can't get to here. Also, the only fix I have found is rebooting our district router, which fixed the problem for 2 weeks last time I did that. That would almost make me rule out a virus, high traffic, or configuration error as well -- but I don't know enough about cisco to say that for sure.

Thanks a lot for your help.

Here's a show interface serial0/1 while the line is down.

Serial0/1 is down, line protocol is down                              
  Hardware is GT96K with integrated T1 CSU/
  Internet address is 10.41.146.33/30
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation HDLC, loopback not set
  Keepalive set (10 sec)
  Last input 01:38:53, output 01:38:58, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 452
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/446 (size/max total/threshold/drops)
     Conversations  0/22/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1158 kilobits/sec
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     8250780 packets input, 3378998601 bytes, 0 no buffer
     Received 187408 broadcasts, 0 runts, 6 giants, 0 throttles
     129354 input errors, 129353 CRC, 55393 frame, 26876 overrun, 0 ignored, 807
07 abort
     7001873 packets output, 4005337381 bytes, 0 underruns
     0 output errors, 0 collisions, 6513 interface resets
     0 output buffer failures, 0 output buffers swapped out
     60 carrier transitions
     DCD=down  DSR=up  DTR=up  RTS=up  CTS=down
0
 
LVL 6

Expert Comment

by:fullerms
ID: 17100425
NBAR is configured on your interface Fa4/1. Can you do a "show tech"  the next time your link goes down? Post the output and we will try to analyze the results. Disable NBAR after getting the "show tech" output.
0
 

Author Comment

by:grevels
ID: 17100482
It'll go down again around 1pm Eastern time so I'll post the info then.

Thanks!
0
 

Author Comment

by:grevels
ID: 17101686
Fullerms,

Shoot me an email to awalker3 at gmail dot com if you don't mind. This thing is incredibly huge and I don't want to post all of it here on the internet for threat of me not taking out some critical information.

Thanks.
0
 
LVL 6

Expert Comment

by:fullerms
ID: 17105165
grevels, a show tech usually removes all password information. You may also remove all IP address information if you feel uncomfortable posting the show tech on the internet.

Experts, is there an FTP site where members can post such diagnostic logs? It may be improper for members to have a one to one communication.

grevels, lets wait to see if there is a ftp option. Otherwise, I will send you an email.
0
 
LVL 79

Expert Comment

by:lrmoore
ID: 17110500
EE Members can post/read stuff here:
http://www.ee-stuff.com/
Log in, go to Expert Area and upload a file related to this specific Q number
0
 

Author Comment

by:grevels
ID: 17121368
Ok, the file is uploaded, so I'll wait for your responses.

Thanks again.
0
 
LVL 6

Accepted Solution

by:
fullerms earned 500 total points
ID: 17122022
Ran the file through the Output injtrepreter, and found a few interesting notes. You may want to go through the information below.


---------------
MEMORY ANALYSIS
---------------
  INFO: Processor memory utilization is 15.00385%.
  INFO: Processor memory or main memory stores the running configuration and routing
  tables. The Cisco IOS software executes from main memory.
  INFO: The amount of processor memory required by the router is affected by the
  Cisco IOS version used, the size of the network and by the access list configurations.
  Ensure that an optimal IOS version has been chosen.
 
  WARNING: The following processes are currently holding more than 1 MB of memory
  with 'Freed' memory less that 'Holding' memory:
    'SCTP Main Proces' (Holding 1054820 bytes)
  This is considered to be high. High memory usage can be normal, or can indicate
  a memory leak. To determine this, it is important to see if memory usage changes
  over time.
  INFO: A memory leak occurs when a process requests or allocates memory and then
  forgets to free (de-allocate) the memory when it is finished with that task.
  As a result, the memory block is reserved until the router is reloaded. Over
  time, more and more memory blocks are allocated by that process until there is
  no free memory available. At this point, the router will issue the 'SYS-2-MALLOCFAIL'
  error message.
  TRY THIS: Analyze the 'show process memory' output for this router over a period
  of time (for example, every few hours or days depending on whether you have a
  fast or slow leak).
  Check to see if memory utilization for the affected process(es) continues to increase
  and the amount of freed memory remains the same. The rate at which free memory
  disappears depends on how often the event occurs that leads to the leak. A memory
  leak is a complex condition sometimes requiring an IOS upgrade to correct. If
  the above is in fact occurring, and you are uncertain about how to proceed, use
  the Service Request Tool to contact the Cisco TAC for further assistance.
  NOTE: It is normal for some processes, such as Border Gateway Protocol (BGP) or
  Open Shortest Path First (OSPF), to use more than one megabyte of memory; this
  does not mean they are leaking.
 
  INFO: The top 3 processes that are holding less than 1 MB of memory are:
    'PPP Events' is holding 204160 bytes
    'EDDRI_MAIN' is holding 90432 bytes
    'CCVPM_HTSP' is holding 89820 bytes
0
 

Author Comment

by:grevels
ID: 17124337
Well, this certainly sounds in-line with our problem, being that reloading the router fixes the problem for a while. Here's what I read on cisco's site:

A memory leak occurs when a process requests or allocates memory and then forgets to free (de-allocate) the memory when it is finished with that task. As a result, the memory block is reserved until the router is reloaded. Over time, more and more memory blocks are allocated by that process until there is no free memory available. Depending on the severity of the low memory situation at this point, the only option you may have is to reload the router to get it operational again.

We are in the process of getting Smartnet on our equipment so as soon as we can get that finished I will request them to do an IOS upgrade.
0
 

Author Comment

by:grevels
ID: 17124655
Ok, here we go... The first section is when the panther line was down and the second is when it was up. At the bottom I listed some others with big differences from when it was up versus being down. The biggest difference was in Virtual Exec. Should I reboot the router and get a reading knowing that the problem won't reoccur for a couple weeks? Would that be helpful for you guys?

PID TTY    Allocated     Free     Holding  GetBufs ReturnBugs Process

show process memory
Total: 62473024, Used: 9368940, Free: 53104084
   5    0       65580          0      90432          0          0 EDDRI_MAIN    
 124   0       79968          0      89820          0          0 CCVPM_HTSP  
 152   0     192544       1268     204160          0          0 PPP Events  

show proc mem
Total: 62473024, Used: 9370332, Free: 53102692
   5   0      65580          0      90432          0          0 EDDRI_MAIN  
 124   0      79968          0      89820          0          0 CCVPM_HTSP  
 152   0     192544       1268     204160          0          0 PPP Events  

_____________________________________________________________________________________________
         PID TTY    Allocated     Free     Holding  GetBufs ReturnBufs Process
Down: 86   0   13593108   13563476      22780          0          0 DHCPD Receive
Up    : 86   0   13632328   13602696      22780          0          0 DHCPD Receive

Down:  131   0  270975508  270974468       9892          0          0 Key Proc  
Up    :  131   0  271774560  271773536       9876          0          0 Key Proc    

Down: 61   0   18820204   18807308      19956          0          0 CDP Protocol
Up    :  61   0   18865476   18851948      20588          0          0 CDP Protocol    

Down:    3 162       1220        580      13492          0          0 Virtual Exec
Up    :    3 162     131736     129740      14848          0          0 Virtual Exec    
0
 
LVL 6

Expert Comment

by:fullerms
ID: 17135867
If you can wait for the Smartnet and the IOS upgrade, then reboot it once.
0
 

Author Comment

by:grevels
ID: 17148570
I uploaded text files showing the output of the show process memory commands before and after I rebooted router. If you don't mind, take a look at them and see if you notice anything. The 4 processes mentioned from the memory analysis do not look much different to me. There is one process that was there before the reboot when the problems were occuring and is not there after the reboot. I also posted another show tech if you could run it through your output interpreter.

THANKS!!!
0
 
LVL 51

Expert Comment

by:Keith Alabaster
ID: 17488804
Out of my depth on this one so I will ask for comments from the experts on closing this one down..

Thanks
Keith
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

In a WLAN, anything you broadcast over the air can be intercepted.  By default a wireless network is wide open to all until security is configured. Even when security is configured information can still be intercepted! It is very important that you …
Creating an OSPF network that automatically (dynamically) reroutes network traffic over other connections to prevent network downtime.
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now