Cisco 3750X Fiber Ports don't recover after neighbor switch loses power.
Posted on 2016-08-24
First I appreciate the grace as I do not consider myself a Cisco professional at all. We have a couple folks working this problem and I wanted to ask the Experts.
We have 2 3750X switches in a stack configuration, with 1Gb fiber runs to 11 closets throughout our building. The 11 closets all have 2960S 48port switches.
This morning I unplugged the 2960S switches to move them to new individual UPS systems, and while it appeared they came up as normal, about half of them did not have a link light on the 3750X stack. Yes you read that right, one end had a link light, the 2960S, but the 3750 showed no link light for about half of the connections. The configurations were not changed on any of the 2960S's, whether they were working or not after the power back up.
Seeing as it was more than one closet having the issue we turned our focus on the 3750X stack. The configuration was also unchanged, and it was reporting no errors on the ports having the issue. No ULDL errors were present on the 3750X at all. The ports were showing no connection, although we were getting a link light on the other end.
We were able to temporarily remedy the problem by moving the problem closet connections to other ports on the 3750X Stack to get them to light up, with the configurations being the same on the new port. The issue still remains that the old ports, 5 of them, seem to be dead to the world.
This evening we are going to perform a reboot of the 3750X stack in hopes it will clear up the issue with the dead ports, but honestly we're really trying to find out how this happened and if anyone has seen this before. We brought in a vendor to work on it today, but he's as stumped as we are. I'll try to update with firmware information.
Thanks for reading my novel.