Line protocol will not stay "up" on P2P T1 circuit

I have two point to point T1 circuits to remote location.  One of the T1s has been in service for nearly seven years, and the second is new because there was a need for more bandwidth.  It's setup like this:

Ckt#1
2811 S0/2/0 (WIC-2T) --> DTE Cable --> ADC D-Serv CSU --> AT&T --> WIC-DSU-T1 -->1841 S0/0/0 (WIC-1DSU-T1)

Ckt#2 (problem circuit)
2811 S0/2/1 (WIC-2T) --> DTE Cable --> ADC D-Serv CSU --> AT&T --> WIC-DSU-T1 -->1841 S0/1/0 (WIC-1DSU-T1)

On rebooting the CSU, unplugging and re-plugging the circuit, or issuing a shut/no shut command via CLI from the headquarters end, the line protocol comes up and then goes down about five or ten seconds later.

Circuit seems to be fine when its up briefly.  The routing tables update , and if I type quickly enough, I can squeeze a ping command to the far-side interface and get a packet or two back.  I ran a clear counters command before generating the show interface commands below, then brought the interface down and up via CLI.

It really sounds like a clocking issue, but I thought the network (AT&T) provided this (?).  The carrier swears the two circuits are setup identically which is clear channel ANSI w/ ESF formatting and B8ZS line coding.  We have four other P2P T1 circuits and this is the only one experiencing the trouble.

I did try replacing the ADC D-Serv and all the cabling at the headquarters end.  It's possible there is a bad cable or something at the far end I suppose.  No errors though on that interface.

Here's what the failure looks like:

*Mar 19 19:42:40.264: %LINK-3-UPDOWN: Interface Serial0/2/1, changed state to down
*Mar 19 19:43:36.827: %LINK-3-UPDOWN: Interface Serial0/2/1, changed state to up
*Mar 19 19:43:37.827: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/2/1, changed state to up
*Mar 19 19:44:03.067: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/2/1, changed state to down


Here is the config:

Headquarters:

DET-RTR4# sh run
interface Serial0/2/0
 description T1 to SBY (Circuit #1)
 bandwidth 1544
 ip address 172.20.150.2 255.255.255.0
 ip load-sharing per-packet
 service-policy output POL
!
interface Serial0/2/1
 description T1 to SBY (Circuit #2)
 bandwidth 1544
 ip address 172.20.0.94 255.255.255.248
 ip load-sharing per-packet
 service-policy output POL

DET-RTR4# sh int s0/2/1
Serial0/2/1 is up, line protocol is down
  Hardware is GT96K Serial
  Description: T1 to SBY (Circuit #2)
  Internet address is 172.20.0.94/29
  MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation HDLC, loopback not set
  Keepalive set (10 sec)
  Last input 01:09:58, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:01:01
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: Class-based queueing
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/1/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 772 kilobits/sec
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     22 packets output, 2116 bytes, 0 underruns
     0 output errors, 0 collisions, 4 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up


Branch office:

SBY-RTR1# sh run
interface Serial0/0/0
 description T1 to DET (Circuit #1)
 bandwidth 1544
 ip address 172.20.150.1 255.255.255.0
 ip load-sharing per-packet
 ip nbar protocol-discovery
 ip virtual-reassembly
 service-module t1 timeslots 1-24
 service-policy output POL
!
interface Serial0/1/0
 description T1 to DET (Circuit #2)
 bandwidth 1544
 ip address 172.20.0.93 255.255.255.248
 ip load-sharing per-packet
 ip nbar protocol-discovery
 ip virtual-reassembly
 service-module t1 timeslots 1-24
 service-policy output POL

SBY-RTR1#sh int s0/1/0
Serial0/1/0 is up, line protocol is down
  Hardware is GT96K with integrated T1 CSU/DSU
  Description: T1 to DET (Circuit #2)
  Internet address is 172.20.0.93/29
  MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation HDLC, loopback not set
  Keepalive set (10 sec)
  Last input never, output 00:00:04, output hang never
  Last clearing of "show interface" counters 00:01:02
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: Class-based queueing
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/3/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 772 kilobits/sec
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     8 packets output, 319 bytes, 0 underruns
     0 output errors, 0 collisions, 2 interface resets
LVL 1
David BlairAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Garry GlendownConsulting and Network/Security SpecialistCommented:
Judging from the "0 packets received" on both sides, there may be a problem with either the csu/dsu units, or the actual line ... 10 seconds uptime before line protocol down is caused by the missing keepalive packets ... both sides expect the keepalives to arrive, but none seem to, so the routers take down the interfaces.
To do some diagnosis, try to get a loop first on the dsu/csu towards the local interface. The router should recognize it being looped. Then, set up a loop towards the WAN ... also, you could ask the provider to set up a loop on their equipment, which the router ought to recognize ...
0
TimotiStDatacenter TechnicianCommented:
Just a guess: maybe the WIC-2T only supports one clock source, and it tries to run port 1 with the clock signal of port 0? Cisco sometimes has weird limitations...
Can you try running the second line on another WIC card in the 2811?

Tamas
0
Garry GlendownConsulting and Network/Security SpecialistCommented:
If the clock source were the problem, there ought to be some increase on the "broken" packet counter, like crc or something ... also, I know we used the 2T serial cards before, with dual links (albeit to different remote sites), they also worked fine without a problem ...
0
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

David BlairAuthor Commented:
I put a loopback plug on the CSU/DSU at the headquarters end, and the router kept the interface up.  This morning I put a loop on the WIC-1DSU-T1 at the branch side and the router didn't see it right away.  I did a shut/no shut on S0/1/0 and the line protocol came up and then went down a few seconds later.  WOW!  So, do you guys think it could be a defective WIC card?  It wasn't a brand new card.  I do have another available and can swap this evening if you think that's a good course of action.
0
Garry GlendownConsulting and Network/Security SpecialistCommented:
WIC or cable ... if possible, just plug in the additional card (don't forget to turn the router off first, not hot-pluggable) and check if that works ... if not, it could be the cable ...
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
David BlairAuthor Commented:
Well, I did the loopback right at the WIC interface using just an RJ-45 and a couple inches of twisted pair.  I'll replace the card this evening.
0
David BlairAuthor Commented:
It was the WIC card!  Wow.  Never suspected it would be the WIC.  Thanks for pushing me in that direction!
0
Garry GlendownConsulting and Network/Security SpecialistCommented:
Well, even Cisco hardware may fail sometimes, though it is pretty reliable mostly ...
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Routers

From novice to tech pro — start learning today.