asked on

Line is Up, Protocol is Down

I've got four T1 lines that are bonded together apart of a PPP multilink group. It's the same on both sides, with the actual serial interfaces not having IP addresses and the multilink interface doing all the L3 talking. A few days ago one of the T1 lines went down. I checked out the controller on side A and all was good. I checked out the controller on side B and all was good. The physical line -being the controller and the link were Up, but the protocol on the serial interface was down. I opened a ticket with the LEC, and they found nothing wrong. I did BERT tests for five minutes on both ends, with the far end looped towards me, and absolutely no bit errors. I also changed the DS1 cards on both ends thinking it could be the controllers. At this point, I've done everything including rebuilding the channel groups to re-establish the Serial interfaces but got no results. The only errors I can find are done via sh int serial1/0/1:1, which shows input errors, a few CRCs, a few frames and a whole lot of Aborts; everything on the output side is 0. I know the Aborts and the errors are bad, but I've checked the config so many times and verified it's not the circuit that I cant figure out whats wrong.

Here's the config on the A side:

no ip address
encap ppp
no fair-queue
ppp multilink
multilink-group 2

on the B side:
no ip address
encap ppp
no fair-queue
ppp multilink group 1

Also: on the B side I don't see any errors on the sh int serialx/x/x:x

Help is appreciated. Thanks

ASKER CERTIFIED SOLUTION

pjtemplin

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

AM6_Networks_AdamL

ASKER

I'll give the hard loop a shot; although I've done an fdl loopback from the A to the Z, ran qrss 20, 0s (to check linecode) and 3in24 (to check timing) and it was all clear. Then, took down the loop, logged into the Z side and threw a loop towards the A, ran all the same patterns and only took a few progressive errors on qrss20; I took about 50 errors across 5 minutes. Granted they're errors- but, definitely doesnt seem like enough to tear apart the entire circuit.

AM6_Networks_AdamL

ASKER

Another side note- on the A side the circuit is going into a multichannel T3- hence why the A side is sort of a known good source. When i do a show controller on this side I have a huge number of RX THROTTLES; which according to Cisco occur when there's a burst of input errors. Not sure if this helps anyone...

AM6_Networks_AdamL

ASKER

Alright, so I removed the interface from the bundle, the protocol immediately comes up. I tried the 1.1.1.0/30 IP test setup and everything pinged perfectly with no errors. Therefore, I can rule out the issue with the line (which sucks). Now, the problem is that on the A side, as soon as a put the interface into the multilink bundle, nothing goes wrong. On the Z side, as soon as i put the interface into the multilink bundle, the protocol dies. I have no idea why it's doing this because I haven't changed the config in almost a year!

Any ideas?

pjtemplin

Bad interface perhaps? Try swapping the T1 connections on the Z end and see if the problem stays with the interface or with the circuit.

AM6_Networks_AdamL

ASKER

Did that, no luck. More fun and games, i can now get the Protocol to remain up, but it wont join the multilink! Here's how:

A-side (start from blank config
no ip addr
encap ppp
ppp multilink
multilink group 1

Z-side (start from blank
no ip
encap ppp
ppp multilink
multilink group 2 (yes, its different)

-With this, my protocol is up, but the interface has multilink Closed, LCP Open. In my original config, I never had 'ppp multilink', so I remove it, and the Protocol still stays up.

Now, if I build the interfaces in this order:

A-side (from blank
no ip
encap ppp

Z-Side (from blank
no ip
encap ppp

The second I put either side in multilink group X, the Protocol crashes. If I follow the first method the Protocol stays live but wont join the bundle. Here's some sh ints:

interface Serial0/3/1:1
no ip address
encapsulation ppp
no fair-queue
ppp multilink group 1
end

Serial0/3/1:1 is up, line protocol is up
Hardware is GT96K Serial
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation PPP, LCP Open, multilink Closed
Open: CDPCP, loopback not set
Keepalive set (10 sec)
Last input 00:00:11, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:37:51
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1159 packets input, 49060 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
1425 packets output, 45152 bytes, 0 underruns
0 output errors, 0 collisions, 303 interface resets
0 output buffer failures, 0 output buffers swapped out
2 carrier transitions
Timeslot(s) Used:1-24, SCC: 1, Transmitter delay is 0 flags

On the A side:

interface Serial1/0/1:1
no ip address
encapsulation ppp
no fair-queue
multilink-group 2
end

Serial1/0/1:1 is up, line protocol is up
Hardware is PA-MC-T3
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation PPP, crc 16, loopback not set
LCP Open
Open: CDPCP
Last input 00:00:19, output 00:00:08, output hang never
Last clearing of "show interface" counters 00:39:47
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue :0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1443 packets input, 46153 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
348 input errors, 9 CRC, 37 frame, 0 overrun, 0 ignored, 302 abort
1190 packets output, 50059 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions no alarm present
Timeslot(s) Used: 1-24, subrate: 1536Kb/s, transmit delay is 0 flags
non-inverted data

The multilink on the A side:
Bundle up for 2w5d
2697 lost fragments, 29742692 reordered, 0 unassigned
1311 discarded, 1000 lost received, 103/255 load
0x35B21E received sequence, 0xEFD7A4 sent sequence
Member links: 2 active, 1 inactive (max not set, min not set)
Serial1/0/3:3, since 2w5d, last rcvd seq 35B21C
Serial1/0/2:2, since 2w5d, last rcvd seq 35B21D
Serial1/0/1:1 (inactive)

The multilink on the Z side:
Bundle up for 2w5d, total bandwidth 3072, load 17/255
Receive buffer limit 24000 bytes, frag timeout 1000 ms
0/0 fragments/bytes in reassembly list
23408 lost fragments, 128600795 reordered
9464/11573748 discarded fragments/bytes, 0 lost received
0xF00EC4 received sequence, 0x35D00D sent sequence
Member links: 2 active, 1 inactive (max not set, min not set)
Se0/3/0:3, since 2w5d
Se0/1/1:1, since 2w5d
Se0/3/1:1 (inactive)

Someone PLEASE tell me they have an idea!

AM6_Networks_AdamL

ASKER

I've gotta reopen a new subject with not being able to add interfaces to multilink group. thanks pj