Solved

Cisco 3750X Fiber Ports don't recover after neighbor switch loses power.

Posted on 2016-08-24
7
72 Views
Last Modified: 2016-08-25
Good afternoon,

First I appreciate the grace as I do not consider myself a Cisco professional at all. We have a couple folks working this problem and I wanted to ask the Experts.

We have 2 3750X switches in a stack configuration, with 1Gb fiber runs to 11 closets throughout our building. The 11 closets all have 2960S 48port switches.

This morning I unplugged the 2960S switches to move them to new individual UPS systems, and while it appeared they came up as normal, about half of them did not have a link light on the 3750X stack. Yes you read that right, one end had a link light, the 2960S, but the 3750 showed no link light for about half of the connections. The configurations were not changed on any of the 2960S's, whether they were working or not after the power back up.

Seeing as it was more than one closet having the issue we turned our focus on the 3750X stack. The configuration was also unchanged, and it was reporting no errors on the ports having the issue. No ULDL errors were present on the 3750X at all. The ports were showing no connection, although we were getting a link light on the other end.

We were able to temporarily remedy the problem by moving the problem closet connections to other ports on the 3750X Stack to get them to light up, with the configurations being the same on the new port. The issue still remains that the old ports, 5 of them, seem to be dead to the world.

This evening we are going to perform a reboot of the 3750X stack in hopes it will clear up the issue with the dead ports, but honestly we're really trying to find out how this happened and if anyone has seen this before. We brought in a vendor to work on it today, but he's as stumped as we are. I'll try to update with firmware information.

Thanks for reading my novel.
0
Comment
Question by:ksfrist
  • 4
  • 2
7 Comments
 
LVL 13

Expert Comment

by:SIM50
ID: 41769391
So ports are not in errdisabled state? Have you tried to do shut no shut?
0
 

Author Comment

by:ksfrist
ID: 41769476
We did attempt shut no shut to no avail.

All ports have the config of:

interface GigabitEthernet1/0/5
 switchport trunk encapsulation dot1q
 switchport mode trunk
 udld port aggressive

Here is a line status of a few of the ports:

Gi2/0/1                      connected    trunk      a-full a-1000 1000BaseSX SFP
Gi2/0/2                      notconnect   1            auto   auto Not Present
Gi2/0/3                      connected    trunk      a-full a-1000 1000BaseSX SFP
Gi2/0/4                      connected    trunk      a-full a-1000 1000BaseSX SFP
Gi2/0/5                      notconnect   1            auto   auto Not Present


05 has a closet plugged into it, 02 does not. It looks like nothing is plugged in, although we're getting a link light on the remote switch. The rest are closets that came up normally.
0
 

Author Comment

by:ksfrist
ID: 41769501
Version Info:

Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 12.2(58)SE2, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Thu 21-Jul-11 01:23 by prod_rel_team

ROM: Bootstrap program is C3750E boot loader
BOOTLDR: C3750E Boot Loader (C3750X-HBOOT-M) Version 12.2(58r)SE1, RELEASE SOFTWARE (fc1)

MDF202-CORE-FCVA uptime is 2 years, 52 weeks, 15 hours, 36 minutes
System returned to ROM by power-on
System restarted at 01:37:24 CST Mon Aug 26 2013
System image file is "flash:/c3750e-universalk9-mz.122-58.SE2/c3750e-universalk9-mz.122-58.SE2.bin"
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 45

Accepted Solution

by:
Craig Beck earned 500 total points
ID: 41769608
It's odd that you see a link light on the 2960 end as it only acknowledges a link UP if the opposite end is sending a pulse of light to it, so that would tell me that the 3750X was transmitting to the 2960 but the 2960 wasn't transmitting to the 3750X, or the fibre was damaged.  It's doubtful that the fibre is damaged as it works when you move the SFP at the 3750X end.

Did you see anything in the logs on the 3750X?
What did show int Gi1/0/x show on each 'dead' port at the 3750X?
Can you post the show diagnostic post, show inteface transceiver and show inventory output from the 3750X, please (hide any serial numbers though)?
0
 

Author Comment

by:ksfrist
ID: 41770186
Good morning,

Thank you for your replies. After limping through the day yesterday we powered off the 3750 stack last night and powered it back on. This seems to have cleared whatever issue up, as I tested a neighbor switch this morning and it came back up as it should. This was one of the problem children yesterday. Craig I ran the commands this morning, which probably won't give you any information after it cleared up last night, but see below:

Switch 1
---------

POST: CPU MIC register Tests : Begin
POST: CPU MIC register Tests : End, Status Passed

POST: MA BIST : Begin
POST: MA BIST : End, Status Passed

POST: TCAM BIST : Begin
POST: TCAM BIST : End, Status Passed

POST: SF ASIC BIST : Begin
POST: SF ASIC BIST : End, Status Passed

POST: Switch Fabric Memory Tests : Begin
POST: Switch Fabric Memory Tests : End, Status Passed

POST: CPU MIC interface Loopback Tests : Begin
POST: CPU MIC interface Loopback Tests : End, Status Passed

POST: PortASIC RingLoopback Tests : Begin
POST: PortASIC RingLoopback Tests : End, Status Passed

POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed

POST: PortASIC Port Loopback Tests : Begin
POST: Failed PortPhyLoopback Packet Test asic_index 1 port_hardware_index 22
POST: PortASIC Port Loopback Tests : End, Status Passed

POST: EMAC Loopback Tests : Begin
POST: EMAC Loopback Tests : End, Status Passed

Switch 2
---------

POST: CPU MIC register Tests : Begin
POST: CPU MIC register Tests : End, Status Passed

POST: MA BIST : Begin
POST: MA BIST : End, Status Passed

POST: TCAM BIST : Begin
POST: TCAM BIST : End, Status Passed

POST: SF ASIC BIST : Begin
POST: SF ASIC BIST : End, Status Passed

POST: Switch Fabric Memory Tests : Begin
POST: Switch Fabric Memory Tests : End, Status Passed

POST: CPU MIC interface Loopback Tests : Begin
POST: CPU MIC interface Loopback Tests : End, Status Passed

POST: PortASIC RingLoopback Tests : Begin
POST: PortASIC RingLoopback Tests : End, Status Passed

POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed

POST: PortASIC Port Loopback Tests : Begin
POST: PortASIC Port Loopback Tests : End, Status Passed

POST: EMAC Loopback Tests : Begin
POST: EMAC Loopback Tests : End, Status Passed

-------------------------------------------------------

If device is externally calibrated, only calibrated values are printed.
++ : high alarm, +  : high warning, -  : low warning, -- : low alarm.
NA or N/A: not applicable, Tx: transmit, Rx: receive.
mA: milliamperes, dBm: decibels (milliwatts).

                                 Optical   Optical
           Temperature  Voltage  Tx Power  Rx Power
Port       (Celsius)    (Volts)  (dBm)     (dBm)
---------  -----------  -------  --------  --------
Gi1/0/1      33.8       3.28      -5.0      -6.5
Gi1/0/2      31.4       3.31      -5.0      -5.2
Gi1/0/3      34.5       3.29      -5.0      -5.7
Gi1/0/4      39.3       3.29      -5.1      -5.7
Gi1/0/5      34.0       3.29      -5.0      -5.7
Gi1/0/6      36.5       3.27      -5.0      -5.7
Gi1/0/11     29.7       3.29      -4.9      -5.2
Gi1/0/12     37.5       3.33      -5.0      -5.2
Gi2/0/1      39.3       3.30      -5.0      -5.6
Gi2/0/2      34.4       3.27      -5.0      -5.3
Gi2/0/3      36.7       3.28      -5.0      -5.7
Gi2/0/4      35.1       3.26      -5.0      -5.6
Gi2/0/11     35.3       3.28      -5.0      -5.0
Gi2/0/12     33.1       3.28      -5.0      -4.8


---------------------------------------------------------------------------------------

NAME: "1", DESCR: "WS-C3750X-12S"
PID: WS-C3750X-12S-S   , VID: V04  , SN:

NAME: "Switch 1 - Power Supply 0", DESCR: "FRU Power Supply"
PID: C3KX-PWR-350WAC   , VID: V02L , SN:

NAME: "GigabitEthernet1/0/1", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD          , VID: V01  , SN:

NAME: "GigabitEthernet1/0/2", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD          , VID: V01  , SN:

NAME: "GigabitEthernet1/0/3", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD          , VID: V01  , SN:

NAME: "GigabitEthernet1/0/4", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD          , VID: V01  , SN:

NAME: "GigabitEthernet1/0/5", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD          , VID: V01  , SN:

NAME: "GigabitEthernet1/0/6", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD          , VID: V01  , SN:

NAME: "GigabitEthernet1/0/11", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD          , VID: V01  , SN:

NAME: "GigabitEthernet1/0/12", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD          , VID: V01  , SN:

NAME: "2", DESCR: "WS-C3750X-12S"
PID: WS-C3750X-12S-S   , VID: V04  , SN:

NAME: "Switch 2 - Power Supply 0", DESCR: "FRU Power Supply"
PID: C3KX-PWR-350WAC   , VID: V02L , SN:

NAME: "GigabitEthernet2/0/1", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD          , VID: V01  , SN:

NAME: "GigabitEthernet2/0/2", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD          , VID: V01  , SN:

NAME: "GigabitEthernet2/0/3", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD          , VID: V01  , SN:

NAME: "GigabitEthernet2/0/4", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD          , VID: V01  , SN:

NAME: "GigabitEthernet2/0/5", DESCR: "1000BaseSX SFP"
PID: Unspecified       , VID:      , SN:

NAME: "GigabitEthernet2/0/11", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD          , VID: V01  , SN:

NAME: "GigabitEthernet2/0/12", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD          , VID: V01  , SN:


Thankfully we had enough un-used ports on the 3750 stack to move the non functioning connections to in order to get by yesterday. I'm still trying to see if I can find out what happened yesterday or if there are any other potential logs files or commands I can run on the 3750.

Again thanks for your time and input.
0
 
LVL 45

Assisted Solution

by:Craig Beck
Craig Beck earned 500 total points
ID: 41770196
Thanks for the info.

Were all of the DOWN interfaces on Switch 1?  It looks like you need to seek assistance from TAC...

POST: PortASIC Port Loopback Tests : Begin
POST: Failed PortPhyLoopback Packet Test asic_index 1 port_hardware_index 22
POST: PortASIC Port Loopback Tests : End, Status Passed
0
 

Author Comment

by:ksfrist
ID: 41770269
They were actually on both switches. We had that thought initially, because it was almost exactly half of our closets that went down. 3 on the top, 2 on the bottom if I remember correctly.

I will certainly open a TAC case for the above message to see what that's all about.

Again thanks for the guidance and if I have a follow up within a reasonable time I'll update the thread.

Thanks Craig.
0

Featured Post

Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

Join & Write a Comment

Suggested Solutions

If you have an ASA5510 then this sort of thing would be better handled with a CSC Module, however on an ASA5505 thats not an option, and if you want to throw in a quick solution to stop your staff going to facebook during work time, then this is the…
Security is one of the biggest concerns when moving and migrating your data from your on-premise location to the Public Cloud.  Where is your data? Who can access it? Will it be safe from accidental deletion?  All of these questions and more are imp…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now