Line is Up, Protocol is Down

I've got four T1 lines that are bonded together apart of a PPP multilink group.  It's the same on both sides, with the actual serial interfaces not having IP addresses and the multilink interface doing all the L3 talking.  A few days ago one of the T1 lines went down.  I checked out the controller on side A and all was good.  I checked out the controller on side B and all was good.  The physical line -being the controller and the link were Up, but the protocol on the serial interface was down.  I opened a ticket with the LEC, and they found nothing wrong.  I did BERT tests for five minutes on both ends, with the far end looped towards me, and absolutely no bit errors.  I also changed the DS1 cards on both ends thinking it could be the controllers.  At this point, I've done everything including rebuilding the channel groups to re-establish the Serial interfaces but got no results.  The only errors I can find are done via sh int serial1/0/1:1, which shows input errors, a few CRCs, a few frames and a whole lot of Aborts; everything on the output side is 0.  I know the Aborts and the errors are bad, but I've checked the config so many times and verified it's not the circuit that I cant figure out whats wrong.

Here's the config on the A side:

no ip address
encap ppp
no fair-queue
ppp multilink
multilink-group 2

on the B side:
no ip address
encap ppp
no fair-queue
ppp multilink group 1

Also: on the B side I don't see any errors on the sh int serialx/x/x:x

Help is appreciated.  Thanks
LVL 2
AM6_Networks_AdamLAsked:
Who is Participating?

Improve company productivity with a Business Account.Sign Up

x
Advertise Here
 
pjtemplinConnect With a Mentor Commented:
Set up a hard loop at the B-side router.  See if the link comes up to a loop ["...is up, line protocol is down (looped)"].  If so, clear the counters and see if it runs clean for a bit.  Move the loop progressively closer (perhaps just to the SmartJack) to see if you can isolate the errors at all.

If that's not real successful but you can get the circuit to see the loop, take that T1 out of the bundle and set it up as a separate, individual T1.  Use HDLC, not PPP, encapsulation (I've seen HDLC work when PPP wouldn't).  Give it an IP address of 1.1.1.1 and 1.1.1.2 on the two ends, mask 255.255.255.252.  Ping the far side.  Does it work?  If so, do an extended ping (just "ping" with no IP address on the tail end, and then answer the individual extra questions).  See below (copied from the Inet-access mailing list):

router#ping
Protocol [ip]:
Target IP address: blah.blah.blah.blah
Repeat count [5]: 1000
Datagram size [100]: 1500
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface:
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]: y
Data pattern [0xABCD]: 0xAAAA
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 1000, 1500-byte ICMP Echos to blah.blah.blah.blah, timeout is 2
seconds:
Packet has data pattern 0xAAAA
Reply data will be validated
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Do this with at least the following patterns:
0x0000 (all zeros)
0xAAAA (alternating 1's and 0's)
0xFFFF (all 1's)

Frequently if some part of the circuit isn't configured properly one of
these tests will largely fail.  At that point you can tell the telco "but
the circuit can't pass all zeros from Z to A (replace with whichever test
fails), it must be a circuit problem!"
0
 
AM6_Networks_AdamLAuthor Commented:
I'll give the hard loop a shot; although I've done an fdl loopback from the A to the Z, ran qrss 20, 0s (to check linecode) and 3in24 (to check timing) and it was all clear.  Then, took down the loop, logged into the Z side and threw a loop towards the A, ran all the same patterns and only took a few progressive errors on qrss20; I took about 50 errors across 5 minutes.  Granted they're errors- but, definitely doesnt seem like enough to tear apart the entire circuit.
0
 
AM6_Networks_AdamLAuthor Commented:
Another side note- on the A side the circuit is going into a multichannel T3- hence why the A side is sort of a known good source.  When i do a show controller on this side I have a huge number of RX THROTTLES; which according to Cisco occur when there's a burst of input errors.  Not sure if this helps anyone...
0
Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Access It Now
 
AM6_Networks_AdamLAuthor Commented:
Alright, so I removed the interface from the bundle, the protocol immediately comes up.  I tried the 1.1.1.0/30 IP test setup and everything pinged perfectly with no errors.  Therefore, I can rule out the issue with the line (which sucks).  Now, the problem is that on the A side, as soon as a put the interface into the multilink bundle, nothing goes wrong.  On the Z side, as soon as i put the interface into the multilink bundle, the protocol dies.  I have no idea why it's doing this because I haven't changed the config in almost a year!

Any ideas?
0
 
pjtemplinCommented:
Bad interface perhaps?  Try swapping the T1 connections on the Z end and see if the problem stays with the interface or with the circuit.
0
 
AM6_Networks_AdamLAuthor Commented:
Did that, no luck.  More fun and games, i can now get the Protocol to remain up, but it wont join the multilink!  Here's how:

A-side (start from blank config
no ip addr
encap ppp
ppp multilink
multilink group 1

Z-side (start from blank
no ip
encap ppp
ppp multilink
multilink group 2 (yes, its different)

-With this, my protocol is up, but the interface has multilink Closed, LCP Open.  In my original config, I never had 'ppp multilink', so I remove it, and the Protocol still stays up.  

Now, if I build the interfaces in this order:

A-side (from blank
no ip
encap ppp

Z-Side (from blank
no ip
encap ppp

The second I put either side in multilink group X, the Protocol crashes.  If I follow the first method the Protocol stays live but wont join the bundle.  Here's some sh ints:

interface Serial0/3/1:1
 no ip address
 encapsulation ppp
 no fair-queue
 ppp multilink group 1
end

Serial0/3/1:1 is up, line protocol is up
  Hardware is GT96K Serial
  MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation PPP, LCP Open, multilink Closed
  Open: CDPCP, loopback not set
  Keepalive set (10 sec)
  Last input 00:00:11, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:37:51
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     1159 packets input, 49060 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     1425 packets output, 45152 bytes, 0 underruns
     0 output errors, 0 collisions, 303 interface resets
     0 output buffer failures, 0 output buffers swapped out
     2 carrier transitions
  Timeslot(s) Used:1-24, SCC: 1, Transmitter delay is 0 flags

On the A side:

interface Serial1/0/1:1
 no ip address
 encapsulation ppp
 no fair-queue
 multilink-group 2
end

Serial1/0/1:1 is up, line protocol is up
  Hardware is PA-MC-T3
  MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation PPP, crc 16, loopback not set
  LCP Open
  Open: CDPCP
  Last input 00:00:19, output 00:00:08, output hang never
  Last clearing of "show interface" counters 00:39:47
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue :0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     1443 packets input, 46153 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     348 input errors, 9 CRC, 37 frame, 0 overrun, 0 ignored, 302 abort
     1190 packets output, 50059 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions no alarm present
  Timeslot(s) Used: 1-24, subrate: 1536Kb/s, transmit delay is 0 flags
  non-inverted data

The multilink on the A side:
Bundle up for 2w5d
  2697 lost fragments, 29742692 reordered, 0 unassigned
  1311 discarded, 1000 lost received, 103/255 load
  0x35B21E received sequence, 0xEFD7A4 sent sequence
  Member links: 2 active, 1 inactive (max not set, min not set)
    Serial1/0/3:3, since 2w5d, last rcvd seq 35B21C
    Serial1/0/2:2, since 2w5d, last rcvd seq 35B21D
    Serial1/0/1:1 (inactive)


The multilink on the Z side:
Bundle up for 2w5d, total bandwidth 3072, load 17/255
  Receive buffer limit 24000 bytes, frag timeout 1000 ms
    0/0 fragments/bytes in reassembly list
    23408 lost fragments, 128600795 reordered
    9464/11573748 discarded fragments/bytes, 0 lost received
    0xF00EC4 received sequence, 0x35D00D sent sequence
  Member links: 2 active, 1 inactive (max not set, min not set)
    Se0/3/0:3, since 2w5d
    Se0/1/1:1, since 2w5d
    Se0/3/1:1 (inactive)


Someone PLEASE tell me they have an idea!
0
 
AM6_Networks_AdamLAuthor Commented:
I've gotta reopen a new subject with not being able to add interfaces to multilink group.  thanks pj
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.