Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.
COURSE of the MONTH
12 days, 15 hours left to enrollBecome a Premium Member and unlock a new, free course in leading technologies each month.
Solved
Line is Up, Protocol is Down
Posted on 2006-11-08
I've got four T1 lines that are bonded together apart of a PPP multilink group. It's the same on both sides, with the actual serial interfaces not having IP addresses and the multilink interface doing all the L3 talking. A few days ago one of the T1 lines went down. I checked out the controller on side A and all was good. I checked out the controller on side B and all was good. The physical line -being the controller and the link were Up, but the protocol on the serial interface was down. I opened a ticket with the LEC, and they found nothing wrong. I did BERT tests for five minutes on both ends, with the far end looped towards me, and absolutely no bit errors. I also changed the DS1 cards on both ends thinking it could be the controllers. At this point, I've done everything including rebuilding the channel groups to re-establish the Serial interfaces but got no results. The only errors I can find are done via sh int serial1/0/1:1, which shows input errors, a few CRCs, a few frames and a whole lot of Aborts; everything on the output side is 0. I know the Aborts and the errors are bad, but I've checked the config so many times and verified it's not the circuit that I cant figure out whats wrong.
Here's the config on the A side:
no ip address
encap ppp
no fair-queue
ppp multilink
multilink-group 2
on the B side:
no ip address
encap ppp
no fair-queue
ppp multilink group 1
Also: on the B side I don't see any errors on the sh int serialx/x/x:x
Help is appreciated. Thanks
Here's the config on the A side:
no ip address
encap ppp
no fair-queue
ppp multilink
multilink-group 2
on the B side:
no ip address
encap ppp
no fair-queue
ppp multilink group 1
Also: on the B side I don't see any errors on the sh int serialx/x/x:x
Help is appreciated. Thanks
[X]
Welcome to Experts Exchange
Add your voice to the tech community where 5M+ people just like you are talking about what matters.
- Help others & share knowledge
- Earn cash & points
- Learn & ask questions
-
5
-
2
7 Comments
Set up a hard loop at the B-side router. See if the link comes up to a loop ["...is up, line protocol is down (looped)"]. If so, clear the counters and see if it runs clean for a bit. Move the loop progressively closer (perhaps just to the SmartJack) to see if you can isolate the errors at all.
If that's not real successful but you can get the circuit to see the loop, take that T1 out of the bundle and set it up as a separate, individual T1. Use HDLC, not PPP, encapsulation (I've seen HDLC work when PPP wouldn't). Give it an IP address of 1.1.1.1 and 1.1.1.2 on the two ends, mask 255.255.255.252. Ping the far side. Does it work? If so, do an extended ping (just "ping" with no IP address on the tail end, and then answer the individual extra questions). See below (copied from the Inet-access mailing list):
router#ping
Protocol [ip]:
Target IP address: blah.blah.blah.blah
Repeat count [5]: 1000
Datagram size [100]: 1500
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface:
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]: y
Data pattern [0xABCD]: 0xAAAA
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 1000, 1500-byte ICMP Echos to blah.blah.blah.blah, timeout is 2
seconds:
Packet has data pattern 0xAAAA
Reply data will be validated
!!!!!!!!!!!!!!!!!!!!!!!!!!
Do this with at least the following patterns:
0x0000 (all zeros)
0xAAAA (alternating 1's and 0's)
0xFFFF (all 1's)
Frequently if some part of the circuit isn't configured properly one of
these tests will largely fail. At that point you can tell the telco "but
the circuit can't pass all zeros from Z to A (replace with whichever test
fails), it must be a circuit problem!"
If that's not real successful but you can get the circuit to see the loop, take that T1 out of the bundle and set it up as a separate, individual T1. Use HDLC, not PPP, encapsulation (I've seen HDLC work when PPP wouldn't). Give it an IP address of 1.1.1.1 and 1.1.1.2 on the two ends, mask 255.255.255.252. Ping the far side. Does it work? If so, do an extended ping (just "ping" with no IP address on the tail end, and then answer the individual extra questions). See below (copied from the Inet-access mailing list):
router#ping
Protocol [ip]:
Target IP address: blah.blah.blah.blah
Repeat count [5]: 1000
Datagram size [100]: 1500
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface:
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]: y
Data pattern [0xABCD]: 0xAAAA
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 1000, 1500-byte ICMP Echos to blah.blah.blah.blah, timeout is 2
seconds:
Packet has data pattern 0xAAAA
Reply data will be validated
!!!!!!!!!!!!!!!!!!!!!!!!!!
Do this with at least the following patterns:
0x0000 (all zeros)
0xAAAA (alternating 1's and 0's)
0xFFFF (all 1's)
Frequently if some part of the circuit isn't configured properly one of
these tests will largely fail. At that point you can tell the telco "but
the circuit can't pass all zeros from Z to A (replace with whichever test
fails), it must be a circuit problem!"
I'll give the hard loop a shot; although I've done an fdl loopback from the A to the Z, ran qrss 20, 0s (to check linecode) and 3in24 (to check timing) and it was all clear. Then, took down the loop, logged into the Z side and threw a loop towards the A, ran all the same patterns and only took a few progressive errors on qrss20; I took about 50 errors across 5 minutes. Granted they're errors- but, definitely doesnt seem like enough to tear apart the entire circuit.
Another side note- on the A side the circuit is going into a multichannel T3- hence why the A side is sort of a known good source. When i do a show controller on this side I have a huge number of RX THROTTLES; which according to Cisco occur when there's a burst of input errors. Not sure if this helps anyone...
Alright, so I removed the interface from the bundle, the protocol immediately comes up. I tried the 1.1.1.0/30 IP test setup and everything pinged perfectly with no errors. Therefore, I can rule out the issue with the line (which sucks). Now, the problem is that on the A side, as soon as a put the interface into the multilink bundle, nothing goes wrong. On the Z side, as soon as i put the interface into the multilink bundle, the protocol dies. I have no idea why it's doing this because I haven't changed the config in almost a year!
Any ideas?
Any ideas?
Bad interface perhaps? Try swapping the T1 connections on the Z end and see if the problem stays with the interface or with the circuit.
Did that, no luck. More fun and games, i can now get the Protocol to remain up, but it wont join the multilink! Here's how:
A-side (start from blank config
no ip addr
encap ppp
ppp multilink
multilink group 1
Z-side (start from blank
no ip
encap ppp
ppp multilink
multilink group 2 (yes, its different)
-With this, my protocol is up, but the interface has multilink Closed, LCP Open. In my original config, I never had 'ppp multilink', so I remove it, and the Protocol still stays up.
Now, if I build the interfaces in this order:
A-side (from blank
no ip
encap ppp
Z-Side (from blank
no ip
encap ppp
The second I put either side in multilink group X, the Protocol crashes. If I follow the first method the Protocol stays live but wont join the bundle. Here's some sh ints:
interface Serial0/3/1:1
no ip address
encapsulation ppp
no fair-queue
ppp multilink group 1
end
Serial0/3/1:1 is up, line protocol is up
Hardware is GT96K Serial
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation PPP, LCP Open, multilink Closed
Open: CDPCP, loopback not set
Keepalive set (10 sec)
Last input 00:00:11, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:37:51
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1159 packets input, 49060 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
1425 packets output, 45152 bytes, 0 underruns
0 output errors, 0 collisions, 303 interface resets
0 output buffer failures, 0 output buffers swapped out
2 carrier transitions
Timeslot(s) Used:1-24, SCC: 1, Transmitter delay is 0 flags
On the A side:
interface Serial1/0/1:1
no ip address
encapsulation ppp
no fair-queue
multilink-group 2
end
Serial1/0/1:1 is up, line protocol is up
Hardware is PA-MC-T3
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation PPP, crc 16, loopback not set
LCP Open
Open: CDPCP
Last input 00:00:19, output 00:00:08, output hang never
Last clearing of "show interface" counters 00:39:47
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue :0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1443 packets input, 46153 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
348 input errors, 9 CRC, 37 frame, 0 overrun, 0 ignored, 302 abort
1190 packets output, 50059 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions no alarm present
Timeslot(s) Used: 1-24, subrate: 1536Kb/s, transmit delay is 0 flags
non-inverted data
The multilink on the A side:
Bundle up for 2w5d
2697 lost fragments, 29742692 reordered, 0 unassigned
1311 discarded, 1000 lost received, 103/255 load
0x35B21E received sequence, 0xEFD7A4 sent sequence
Member links: 2 active, 1 inactive (max not set, min not set)
Serial1/0/3:3, since 2w5d, last rcvd seq 35B21C
Serial1/0/2:2, since 2w5d, last rcvd seq 35B21D
Serial1/0/1:1 (inactive)
The multilink on the Z side:
Bundle up for 2w5d, total bandwidth 3072, load 17/255
Receive buffer limit 24000 bytes, frag timeout 1000 ms
0/0 fragments/bytes in reassembly list
23408 lost fragments, 128600795 reordered
9464/11573748 discarded fragments/bytes, 0 lost received
0xF00EC4 received sequence, 0x35D00D sent sequence
Member links: 2 active, 1 inactive (max not set, min not set)
Se0/3/0:3, since 2w5d
Se0/1/1:1, since 2w5d
Se0/3/1:1 (inactive)
Someone PLEASE tell me they have an idea!
A-side (start from blank config
no ip addr
encap ppp
ppp multilink
multilink group 1
Z-side (start from blank
no ip
encap ppp
ppp multilink
multilink group 2 (yes, its different)
-With this, my protocol is up, but the interface has multilink Closed, LCP Open. In my original config, I never had 'ppp multilink', so I remove it, and the Protocol still stays up.
Now, if I build the interfaces in this order:
A-side (from blank
no ip
encap ppp
Z-Side (from blank
no ip
encap ppp
The second I put either side in multilink group X, the Protocol crashes. If I follow the first method the Protocol stays live but wont join the bundle. Here's some sh ints:
interface Serial0/3/1:1
no ip address
encapsulation ppp
no fair-queue
ppp multilink group 1
end
Serial0/3/1:1 is up, line protocol is up
Hardware is GT96K Serial
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation PPP, LCP Open, multilink Closed
Open: CDPCP, loopback not set
Keepalive set (10 sec)
Last input 00:00:11, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:37:51
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1159 packets input, 49060 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
1425 packets output, 45152 bytes, 0 underruns
0 output errors, 0 collisions, 303 interface resets
0 output buffer failures, 0 output buffers swapped out
2 carrier transitions
Timeslot(s) Used:1-24, SCC: 1, Transmitter delay is 0 flags
On the A side:
interface Serial1/0/1:1
no ip address
encapsulation ppp
no fair-queue
multilink-group 2
end
Serial1/0/1:1 is up, line protocol is up
Hardware is PA-MC-T3
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation PPP, crc 16, loopback not set
LCP Open
Open: CDPCP
Last input 00:00:19, output 00:00:08, output hang never
Last clearing of "show interface" counters 00:39:47
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue :0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1443 packets input, 46153 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
348 input errors, 9 CRC, 37 frame, 0 overrun, 0 ignored, 302 abort
1190 packets output, 50059 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions no alarm present
Timeslot(s) Used: 1-24, subrate: 1536Kb/s, transmit delay is 0 flags
non-inverted data
The multilink on the A side:
Bundle up for 2w5d
2697 lost fragments, 29742692 reordered, 0 unassigned
1311 discarded, 1000 lost received, 103/255 load
0x35B21E received sequence, 0xEFD7A4 sent sequence
Member links: 2 active, 1 inactive (max not set, min not set)
Serial1/0/3:3, since 2w5d, last rcvd seq 35B21C
Serial1/0/2:2, since 2w5d, last rcvd seq 35B21D
Serial1/0/1:1 (inactive)
The multilink on the Z side:
Bundle up for 2w5d, total bandwidth 3072, load 17/255
Receive buffer limit 24000 bytes, frag timeout 1000 ms
0/0 fragments/bytes in reassembly list
23408 lost fragments, 128600795 reordered
9464/11573748 discarded fragments/bytes, 0 lost received
0xF00EC4 received sequence, 0x35D00D sent sequence
Member links: 2 active, 1 inactive (max not set, min not set)
Se0/3/0:3, since 2w5d
Se0/1/1:1, since 2w5d
Se0/3/1:1 (inactive)
Someone PLEASE tell me they have an idea!
Featured Post
Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.
One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.
Question has a verified solution.
If you are experiencing a similar issue, please ask a related question
I recently attended Cisco Live! in Las Vegas, a conference that boasted over 28,000 techies in attendance, and a week of hands-on learning hosted by a solid partner with which Concerto goes to market. Every year, Cisco displays cutting-edge technol…
Shadow IT is coming out of the shadows as more businesses are choosing cloud-based applications. It is now a multi-cloud world for most organizations. Simultaneously, most businesses have yet to consolidate with one cloud provider or define an offic…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
Suggested Courses
Course of the Month12 days, 15 hours left to enroll
777 members asked questions and received personalized solutions in the past 7 days.
Join the community of 500,000 technology professionals and ask your questions.