PTP T1 circuits clocking issue

I have a 3725 core router with three PTP T1 circuits to three 1721 routers.  We are using the new v2.0 T1 cards.  The ISP indicates there is no line clocking.  Two CCIEs and Cisco have tried to solve my problem for over 40 hours with no luck.  I am not very technical so I will do my best to describe the symptoms.  Currently each branch router is providing clocking although we have also tried having the core router provide the clocking.  Each T1 has been tested with hard loopback plugs several times and the ISP indicates lines are fine.  Here are the symptoms:

The most common symptom is a reload of the core router results in not all three T1 circuits coming back up.  Sometimes all three come up , most often a random two do, and sometimes only one does.  The core router interface that does not come back up is in a down down state and the branch router is in an up down state.   Issuing a clear service-module command on the down down interface brings the circuit back up.

Second and rare symptom has happened twice in one week on separate circuits:  after several core router reloads the core router interface and branch router both come back with an Up Down state and CRCs start building on both sides.  After about 40 minutes and simultaneous power recycles on both sides the circuit comes back up.

Third symptom is occasionally about 30 CRCs per night build up on the core router interfaces.  There are no data loads on the circuits.  They are not yet in production.   Massive ping tests produce no CRCs.

ISP testing on one T1 circuit this afternoon showed the following results:  testing back to the core router was clean.  Testing to the branch router resulted in errors.  Hard loop back on branch side circuit tested clean.  When ISP puts branch router in loopback they see framing errors.

CCIEs now suspect a compatibility problem with the 1721 and v2.0 cards.  The 3725 and all three T1 cards were chnaged out at the beginning becuase of suspected faulty hardware.
Who is Participating?
mikebernhardtConnect With a Mentor Commented:
Sounds like the only clear indication of the source of your problem is the test of the T1 WICs on the 1721. So I would focus my efforts there. As lrmoore said, make sure the IOS is correct for the cards. I couldn't find any bugs in cisco's bug tool but a call to TAC might reveal other issues.

I also think he's right about moving your clock source for all circuits to the 3725. But, a question- you said your ISP isn't providing line clock. I assume you mean the telco or other circuit provider. Why aren't they providing that? Normally they do on a T1. I'm wondering what would happen if both a router AND the telco were providing clock... Just a thought.
lrmooreConnect With a Mentor Commented:
why not have the core 3725 provide the clocking for all T1's? It does not make sense to me to have each remote provide the clock and have the 3725 dealing with 3 different clock sources. Are all of your T1 cards the WIC-1DSU-T1 v.2 cards so that you have the same at each end? Do you have the right version of IOS on the 1700's?
Are you using HDLC or PPP data encapsulation on the T1's?
spontoAuthor Commented:
Yes, we had confirmed the telco is not providing clocking.  I'm told that is common on a PTP.  We started with the 3725 providing clocking and then switched it to the 1721s because of the problems we were having.  TAC also had us add a "invert txclock" statement into the router configs that were providing clocking.  They said "this has resolved this issue in the past".  Yes, all T1 cards are WIC-1DSU-T1 v.2.  Encapsulation started as PPP and then we switched to HDLC.  I do not know the version of IOS on the 1721s but my engineer changed it out several times with TAC.  TAC has been elevated to several levels but they have no more answers.  Their last comment was it had to be a line voltage issue but the telco confirms line voltage is normal via onsite tests at 16.59 and 16.60db.

I have now changed out the T1 cards in the three 1721s with v1.0 cards and all circuits now come up consistently in a normal manner when I repeatedly restart the 3725.   This would seem to indicate the v1.0 and v2.0 cards are indeed behaving differently in the 1721.  We still have v2.0 cards in the 3725.  Does this pinpoint an IOS issue on the 1721s?   I'm now monitoring for CRCs to see if that issue has also resolved itself with the v1.0 cards.  The v1.0 cards are on loan to me and I think I need to get the v2.0 cards working since they are current and the ones I own.

I think you answered your own last question. If you are only having the v2.0 problem in the 1721, then the 1721s most likely have an IOS issue- some kind of hardware support bug. The only other test you could do is to put those v2 cards in the 3735 and see if they work properly. Then you'll know if you somehow found yourself with 3 bad cards. If they are good, you'll need to push Cisco to open a bug report for your problem.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.