Solved

Line is Up, Protocol is Down

Posted on 2006-11-08
7
2,938 Views
Last Modified: 2008-04-01
I've got four T1 lines that are bonded together apart of a PPP multilink group.  It's the same on both sides, with the actual serial interfaces not having IP addresses and the multilink interface doing all the L3 talking.  A few days ago one of the T1 lines went down.  I checked out the controller on side A and all was good.  I checked out the controller on side B and all was good.  The physical line -being the controller and the link were Up, but the protocol on the serial interface was down.  I opened a ticket with the LEC, and they found nothing wrong.  I did BERT tests for five minutes on both ends, with the far end looped towards me, and absolutely no bit errors.  I also changed the DS1 cards on both ends thinking it could be the controllers.  At this point, I've done everything including rebuilding the channel groups to re-establish the Serial interfaces but got no results.  The only errors I can find are done via sh int serial1/0/1:1, which shows input errors, a few CRCs, a few frames and a whole lot of Aborts; everything on the output side is 0.  I know the Aborts and the errors are bad, but I've checked the config so many times and verified it's not the circuit that I cant figure out whats wrong.

Here's the config on the A side:

no ip address
encap ppp
no fair-queue
ppp multilink
multilink-group 2

on the B side:
no ip address
encap ppp
no fair-queue
ppp multilink group 1

Also: on the B side I don't see any errors on the sh int serialx/x/x:x

Help is appreciated.  Thanks
0
Comment
Question by:AM6_Networks_AdamL
  • 5
  • 2
7 Comments
 
LVL 12

Accepted Solution

by:
pjtemplin earned 500 total points
ID: 17904043
Set up a hard loop at the B-side router.  See if the link comes up to a loop ["...is up, line protocol is down (looped)"].  If so, clear the counters and see if it runs clean for a bit.  Move the loop progressively closer (perhaps just to the SmartJack) to see if you can isolate the errors at all.

If that's not real successful but you can get the circuit to see the loop, take that T1 out of the bundle and set it up as a separate, individual T1.  Use HDLC, not PPP, encapsulation (I've seen HDLC work when PPP wouldn't).  Give it an IP address of 1.1.1.1 and 1.1.1.2 on the two ends, mask 255.255.255.252.  Ping the far side.  Does it work?  If so, do an extended ping (just "ping" with no IP address on the tail end, and then answer the individual extra questions).  See below (copied from the Inet-access mailing list):

router#ping
Protocol [ip]:
Target IP address: blah.blah.blah.blah
Repeat count [5]: 1000
Datagram size [100]: 1500
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface:
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]: y
Data pattern [0xABCD]: 0xAAAA
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 1000, 1500-byte ICMP Echos to blah.blah.blah.blah, timeout is 2
seconds:
Packet has data pattern 0xAAAA
Reply data will be validated
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Do this with at least the following patterns:
0x0000 (all zeros)
0xAAAA (alternating 1's and 0's)
0xFFFF (all 1's)

Frequently if some part of the circuit isn't configured properly one of
these tests will largely fail.  At that point you can tell the telco "but
the circuit can't pass all zeros from Z to A (replace with whichever test
fails), it must be a circuit problem!"
0
 
LVL 2

Author Comment

by:AM6_Networks_AdamL
ID: 17908204
I'll give the hard loop a shot; although I've done an fdl loopback from the A to the Z, ran qrss 20, 0s (to check linecode) and 3in24 (to check timing) and it was all clear.  Then, took down the loop, logged into the Z side and threw a loop towards the A, ran all the same patterns and only took a few progressive errors on qrss20; I took about 50 errors across 5 minutes.  Granted they're errors- but, definitely doesnt seem like enough to tear apart the entire circuit.
0
 
LVL 2

Author Comment

by:AM6_Networks_AdamL
ID: 17910641
Another side note- on the A side the circuit is going into a multichannel T3- hence why the A side is sort of a known good source.  When i do a show controller on this side I have a huge number of RX THROTTLES; which according to Cisco occur when there's a burst of input errors.  Not sure if this helps anyone...
0
What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 
LVL 2

Author Comment

by:AM6_Networks_AdamL
ID: 17916092
Alright, so I removed the interface from the bundle, the protocol immediately comes up.  I tried the 1.1.1.0/30 IP test setup and everything pinged perfectly with no errors.  Therefore, I can rule out the issue with the line (which sucks).  Now, the problem is that on the A side, as soon as a put the interface into the multilink bundle, nothing goes wrong.  On the Z side, as soon as i put the interface into the multilink bundle, the protocol dies.  I have no idea why it's doing this because I haven't changed the config in almost a year!

Any ideas?
0
 
LVL 12

Expert Comment

by:pjtemplin
ID: 17916122
Bad interface perhaps?  Try swapping the T1 connections on the Z end and see if the problem stays with the interface or with the circuit.
0
 
LVL 2

Author Comment

by:AM6_Networks_AdamL
ID: 17916584
Did that, no luck.  More fun and games, i can now get the Protocol to remain up, but it wont join the multilink!  Here's how:

A-side (start from blank config
no ip addr
encap ppp
ppp multilink
multilink group 1

Z-side (start from blank
no ip
encap ppp
ppp multilink
multilink group 2 (yes, its different)

-With this, my protocol is up, but the interface has multilink Closed, LCP Open.  In my original config, I never had 'ppp multilink', so I remove it, and the Protocol still stays up.  

Now, if I build the interfaces in this order:

A-side (from blank
no ip
encap ppp

Z-Side (from blank
no ip
encap ppp

The second I put either side in multilink group X, the Protocol crashes.  If I follow the first method the Protocol stays live but wont join the bundle.  Here's some sh ints:

interface Serial0/3/1:1
 no ip address
 encapsulation ppp
 no fair-queue
 ppp multilink group 1
end

Serial0/3/1:1 is up, line protocol is up
  Hardware is GT96K Serial
  MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation PPP, LCP Open, multilink Closed
  Open: CDPCP, loopback not set
  Keepalive set (10 sec)
  Last input 00:00:11, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:37:51
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     1159 packets input, 49060 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     1425 packets output, 45152 bytes, 0 underruns
     0 output errors, 0 collisions, 303 interface resets
     0 output buffer failures, 0 output buffers swapped out
     2 carrier transitions
  Timeslot(s) Used:1-24, SCC: 1, Transmitter delay is 0 flags

On the A side:

interface Serial1/0/1:1
 no ip address
 encapsulation ppp
 no fair-queue
 multilink-group 2
end

Serial1/0/1:1 is up, line protocol is up
  Hardware is PA-MC-T3
  MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation PPP, crc 16, loopback not set
  LCP Open
  Open: CDPCP
  Last input 00:00:19, output 00:00:08, output hang never
  Last clearing of "show interface" counters 00:39:47
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue :0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     1443 packets input, 46153 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     348 input errors, 9 CRC, 37 frame, 0 overrun, 0 ignored, 302 abort
     1190 packets output, 50059 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions no alarm present
  Timeslot(s) Used: 1-24, subrate: 1536Kb/s, transmit delay is 0 flags
  non-inverted data

The multilink on the A side:
Bundle up for 2w5d
  2697 lost fragments, 29742692 reordered, 0 unassigned
  1311 discarded, 1000 lost received, 103/255 load
  0x35B21E received sequence, 0xEFD7A4 sent sequence
  Member links: 2 active, 1 inactive (max not set, min not set)
    Serial1/0/3:3, since 2w5d, last rcvd seq 35B21C
    Serial1/0/2:2, since 2w5d, last rcvd seq 35B21D
    Serial1/0/1:1 (inactive)


The multilink on the Z side:
Bundle up for 2w5d, total bandwidth 3072, load 17/255
  Receive buffer limit 24000 bytes, frag timeout 1000 ms
    0/0 fragments/bytes in reassembly list
    23408 lost fragments, 128600795 reordered
    9464/11573748 discarded fragments/bytes, 0 lost received
    0xF00EC4 received sequence, 0x35D00D sent sequence
  Member links: 2 active, 1 inactive (max not set, min not set)
    Se0/3/0:3, since 2w5d
    Se0/1/1:1, since 2w5d
    Se0/3/1:1 (inactive)


Someone PLEASE tell me they have an idea!
0
 
LVL 2

Author Comment

by:AM6_Networks_AdamL
ID: 17917808
I've gotta reopen a new subject with not being able to add interfaces to multilink group.  thanks pj
0

Featured Post

Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

Join & Write a Comment

Problem Description:   Couple of months ago we upgraded the ADSL line at our branch office from Home to Business line. The purpose of transforming the service to have static public IP’s. We were in need for public IP’s to publish our web resour…
There are two basic ways to configure a static route for Cisco IOS devices. I've written this article to highlight a case study comparing the configuration of a static route using the next-hop IP and the configuration of a static route using an outg…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now