Nowadays, monitoring is a mixture of tools, systems, and codes—making it a very complex process. And with this complexity, comes variables for failure. Get DZone’s new Guide to Performance to learn how to proactively find these variables and solve them before a disruption occurs.
Course Of The Month
6 days, 19 hours left to enrollBecome a Premium Member and unlock a new, free course in leading technologies each month.
Solved
Line is Up, Protocol is Down
Posted on 2006-11-08
I've got four T1 lines that are bonded together apart of a PPP multilink group. It's the same on both sides, with the actual serial interfaces not having IP addresses and the multilink interface doing all the L3 talking. A few days ago one of the T1 lines went down. I checked out the controller on side A and all was good. I checked out the controller on side B and all was good. The physical line -being the controller and the link were Up, but the protocol on the serial interface was down. I opened a ticket with the LEC, and they found nothing wrong. I did BERT tests for five minutes on both ends, with the far end looped towards me, and absolutely no bit errors. I also changed the DS1 cards on both ends thinking it could be the controllers. At this point, I've done everything including rebuilding the channel groups to re-establish the Serial interfaces but got no results. The only errors I can find are done via sh int serial1/0/1:1, which shows input errors, a few CRCs, a few frames and a whole lot of Aborts; everything on the output side is 0. I know the Aborts and the errors are bad, but I've checked the config so many times and verified it's not the circuit that I cant figure out whats wrong.
Here's the config on the A side:
no ip address
encap ppp
no fair-queue
ppp multilink
multilink-group 2
on the B side:
no ip address
encap ppp
no fair-queue
ppp multilink group 1
Also: on the B side I don't see any errors on the sh int serialx/x/x:x
Help is appreciated. Thanks
Here's the config on the A side:
no ip address
encap ppp
no fair-queue
ppp multilink
multilink-group 2
on the B side:
no ip address
encap ppp
no fair-queue
ppp multilink group 1
Also: on the B side I don't see any errors on the sh int serialx/x/x:x
Help is appreciated. Thanks
[X]
Welcome to Experts Exchange
Add your voice to the tech community where 5M+ people just like you are talking about what matters.
- Help others & share knowledge
- Earn cash & points
- Learn & ask questions
-
5
-
2
7 Comments
Set up a hard loop at the B-side router. See if the link comes up to a loop ["...is up, line protocol is down (looped)"]. If so, clear the counters and see if it runs clean for a bit. Move the loop progressively closer (perhaps just to the SmartJack) to see if you can isolate the errors at all.
If that's not real successful but you can get the circuit to see the loop, take that T1 out of the bundle and set it up as a separate, individual T1. Use HDLC, not PPP, encapsulation (I've seen HDLC work when PPP wouldn't). Give it an IP address of 1.1.1.1 and 1.1.1.2 on the two ends, mask 255.255.255.252. Ping the far side. Does it work? If so, do an extended ping (just "ping" with no IP address on the tail end, and then answer the individual extra questions). See below (copied from the Inet-access mailing list):
router#ping
Protocol [ip]:
Target IP address: blah.blah.blah.blah
Repeat count [5]: 1000
Datagram size [100]: 1500
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface:
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]: y
Data pattern [0xABCD]: 0xAAAA
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 1000, 1500-byte ICMP Echos to blah.blah.blah.blah, timeout is 2
seconds:
Packet has data pattern 0xAAAA
Reply data will be validated
!!!!!!!!!!!!!!!!!!!!!!!!!!
Do this with at least the following patterns:
0x0000 (all zeros)
0xAAAA (alternating 1's and 0's)
0xFFFF (all 1's)
Frequently if some part of the circuit isn't configured properly one of
these tests will largely fail. At that point you can tell the telco "but
the circuit can't pass all zeros from Z to A (replace with whichever test
fails), it must be a circuit problem!"
If that's not real successful but you can get the circuit to see the loop, take that T1 out of the bundle and set it up as a separate, individual T1. Use HDLC, not PPP, encapsulation (I've seen HDLC work when PPP wouldn't). Give it an IP address of 1.1.1.1 and 1.1.1.2 on the two ends, mask 255.255.255.252. Ping the far side. Does it work? If so, do an extended ping (just "ping" with no IP address on the tail end, and then answer the individual extra questions). See below (copied from the Inet-access mailing list):
router#ping
Protocol [ip]:
Target IP address: blah.blah.blah.blah
Repeat count [5]: 1000
Datagram size [100]: 1500
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface:
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]: y
Data pattern [0xABCD]: 0xAAAA
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 1000, 1500-byte ICMP Echos to blah.blah.blah.blah, timeout is 2
seconds:
Packet has data pattern 0xAAAA
Reply data will be validated
!!!!!!!!!!!!!!!!!!!!!!!!!!
Do this with at least the following patterns:
0x0000 (all zeros)
0xAAAA (alternating 1's and 0's)
0xFFFF (all 1's)
Frequently if some part of the circuit isn't configured properly one of
these tests will largely fail. At that point you can tell the telco "but
the circuit can't pass all zeros from Z to A (replace with whichever test
fails), it must be a circuit problem!"
I'll give the hard loop a shot; although I've done an fdl loopback from the A to the Z, ran qrss 20, 0s (to check linecode) and 3in24 (to check timing) and it was all clear. Then, took down the loop, logged into the Z side and threw a loop towards the A, ran all the same patterns and only took a few progressive errors on qrss20; I took about 50 errors across 5 minutes. Granted they're errors- but, definitely doesnt seem like enough to tear apart the entire circuit.
Another side note- on the A side the circuit is going into a multichannel T3- hence why the A side is sort of a known good source. When i do a show controller on this side I have a huge number of RX THROTTLES; which according to Cisco occur when there's a burst of input errors. Not sure if this helps anyone...
Alright, so I removed the interface from the bundle, the protocol immediately comes up. I tried the 1.1.1.0/30 IP test setup and everything pinged perfectly with no errors. Therefore, I can rule out the issue with the line (which sucks). Now, the problem is that on the A side, as soon as a put the interface into the multilink bundle, nothing goes wrong. On the Z side, as soon as i put the interface into the multilink bundle, the protocol dies. I have no idea why it's doing this because I haven't changed the config in almost a year!
Any ideas?
Any ideas?
Bad interface perhaps? Try swapping the T1 connections on the Z end and see if the problem stays with the interface or with the circuit.
Did that, no luck. More fun and games, i can now get the Protocol to remain up, but it wont join the multilink! Here's how:
A-side (start from blank config
no ip addr
encap ppp
ppp multilink
multilink group 1
Z-side (start from blank
no ip
encap ppp
ppp multilink
multilink group 2 (yes, its different)
-With this, my protocol is up, but the interface has multilink Closed, LCP Open. In my original config, I never had 'ppp multilink', so I remove it, and the Protocol still stays up.
Now, if I build the interfaces in this order:
A-side (from blank
no ip
encap ppp
Z-Side (from blank
no ip
encap ppp
The second I put either side in multilink group X, the Protocol crashes. If I follow the first method the Protocol stays live but wont join the bundle. Here's some sh ints:
interface Serial0/3/1:1
no ip address
encapsulation ppp
no fair-queue
ppp multilink group 1
end
Serial0/3/1:1 is up, line protocol is up
Hardware is GT96K Serial
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation PPP, LCP Open, multilink Closed
Open: CDPCP, loopback not set
Keepalive set (10 sec)
Last input 00:00:11, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:37:51
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1159 packets input, 49060 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
1425 packets output, 45152 bytes, 0 underruns
0 output errors, 0 collisions, 303 interface resets
0 output buffer failures, 0 output buffers swapped out
2 carrier transitions
Timeslot(s) Used:1-24, SCC: 1, Transmitter delay is 0 flags
On the A side:
interface Serial1/0/1:1
no ip address
encapsulation ppp
no fair-queue
multilink-group 2
end
Serial1/0/1:1 is up, line protocol is up
Hardware is PA-MC-T3
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation PPP, crc 16, loopback not set
LCP Open
Open: CDPCP
Last input 00:00:19, output 00:00:08, output hang never
Last clearing of "show interface" counters 00:39:47
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue :0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1443 packets input, 46153 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
348 input errors, 9 CRC, 37 frame, 0 overrun, 0 ignored, 302 abort
1190 packets output, 50059 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions no alarm present
Timeslot(s) Used: 1-24, subrate: 1536Kb/s, transmit delay is 0 flags
non-inverted data
The multilink on the A side:
Bundle up for 2w5d
2697 lost fragments, 29742692 reordered, 0 unassigned
1311 discarded, 1000 lost received, 103/255 load
0x35B21E received sequence, 0xEFD7A4 sent sequence
Member links: 2 active, 1 inactive (max not set, min not set)
Serial1/0/3:3, since 2w5d, last rcvd seq 35B21C
Serial1/0/2:2, since 2w5d, last rcvd seq 35B21D
Serial1/0/1:1 (inactive)
The multilink on the Z side:
Bundle up for 2w5d, total bandwidth 3072, load 17/255
Receive buffer limit 24000 bytes, frag timeout 1000 ms
0/0 fragments/bytes in reassembly list
23408 lost fragments, 128600795 reordered
9464/11573748 discarded fragments/bytes, 0 lost received
0xF00EC4 received sequence, 0x35D00D sent sequence
Member links: 2 active, 1 inactive (max not set, min not set)
Se0/3/0:3, since 2w5d
Se0/1/1:1, since 2w5d
Se0/3/1:1 (inactive)
Someone PLEASE tell me they have an idea!
A-side (start from blank config
no ip addr
encap ppp
ppp multilink
multilink group 1
Z-side (start from blank
no ip
encap ppp
ppp multilink
multilink group 2 (yes, its different)
-With this, my protocol is up, but the interface has multilink Closed, LCP Open. In my original config, I never had 'ppp multilink', so I remove it, and the Protocol still stays up.
Now, if I build the interfaces in this order:
A-side (from blank
no ip
encap ppp
Z-Side (from blank
no ip
encap ppp
The second I put either side in multilink group X, the Protocol crashes. If I follow the first method the Protocol stays live but wont join the bundle. Here's some sh ints:
interface Serial0/3/1:1
no ip address
encapsulation ppp
no fair-queue
ppp multilink group 1
end
Serial0/3/1:1 is up, line protocol is up
Hardware is GT96K Serial
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation PPP, LCP Open, multilink Closed
Open: CDPCP, loopback not set
Keepalive set (10 sec)
Last input 00:00:11, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:37:51
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1159 packets input, 49060 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
1425 packets output, 45152 bytes, 0 underruns
0 output errors, 0 collisions, 303 interface resets
0 output buffer failures, 0 output buffers swapped out
2 carrier transitions
Timeslot(s) Used:1-24, SCC: 1, Transmitter delay is 0 flags
On the A side:
interface Serial1/0/1:1
no ip address
encapsulation ppp
no fair-queue
multilink-group 2
end
Serial1/0/1:1 is up, line protocol is up
Hardware is PA-MC-T3
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation PPP, crc 16, loopback not set
LCP Open
Open: CDPCP
Last input 00:00:19, output 00:00:08, output hang never
Last clearing of "show interface" counters 00:39:47
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue :0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1443 packets input, 46153 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
348 input errors, 9 CRC, 37 frame, 0 overrun, 0 ignored, 302 abort
1190 packets output, 50059 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions no alarm present
Timeslot(s) Used: 1-24, subrate: 1536Kb/s, transmit delay is 0 flags
non-inverted data
The multilink on the A side:
Bundle up for 2w5d
2697 lost fragments, 29742692 reordered, 0 unassigned
1311 discarded, 1000 lost received, 103/255 load
0x35B21E received sequence, 0xEFD7A4 sent sequence
Member links: 2 active, 1 inactive (max not set, min not set)
Serial1/0/3:3, since 2w5d, last rcvd seq 35B21C
Serial1/0/2:2, since 2w5d, last rcvd seq 35B21D
Serial1/0/1:1 (inactive)
The multilink on the Z side:
Bundle up for 2w5d, total bandwidth 3072, load 17/255
Receive buffer limit 24000 bytes, frag timeout 1000 ms
0/0 fragments/bytes in reassembly list
23408 lost fragments, 128600795 reordered
9464/11573748 discarded fragments/bytes, 0 lost received
0xF00EC4 received sequence, 0x35D00D sent sequence
Member links: 2 active, 1 inactive (max not set, min not set)
Se0/3/0:3, since 2w5d
Se0/1/1:1, since 2w5d
Se0/3/1:1 (inactive)
Someone PLEASE tell me they have an idea!
Featured Post
![[Webinar] How Hackers Steal Your Credentials](https://filedb.experts-exchange.com/files/public/2017/5/20/3315d9d8-8110-4d52-be94-a5aacdd83e4b.png)
Do You Know How Hackers Steal Your Credentials? Join us and Skyport Systems to learn how hackers steal your credentials and why Active Directory must be secure to stop them. Thursday, July 13, 2017 10:00 A.M. PDT
Question has a verified solution.
If you are experiencing a similar issue, please ask a related question
In the world of WAN, QoS is a pretty important topic for most, if not all, networks. Some WAN technologies have QoS mechanisms built in, but others, such as some L2 WAN's, don't have QoS control in the provider cloud.
Quality of Service (QoS) options are nearly endless when it comes to networks today. This article is merely one example of how it can be handled in a hub-n-spoke design using a 3-tier configuration.
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
Suggested Courses
Course of the Month6 days, 19 hours left to enroll
Earn Certification
Cisco CCNA R&S: ICND2 v3
$197.00$177.30
Earn Certification
Cisco CCENT R&S: ICND1 v3
$197.00$177.30
724 members asked questions and received personalized solutions in the past 7 days.
Join the community of 500,000 technology professionals and ask your questions.