Cisco 3750X Fiber Ports don't recover after neighbor switch loses power.
Good afternoon,
First I appreciate the grace as I do not consider myself a Cisco professional at all. We have a couple folks working this problem and I wanted to ask the Experts.
We have 2 3750X switches in a stack configuration, with 1Gb fiber runs to 11 closets throughout our building. The 11 closets all have 2960S 48port switches.
This morning I unplugged the 2960S switches to move them to new individual UPS systems, and while it appeared they came up as normal, about half of them did not have a link light on the 3750X stack. Yes you read that right, one end had a link light, the 2960S, but the 3750 showed no link light for about half of the connections. The configurations were not changed on any of the 2960S's, whether they were working or not after the power back up.
Seeing as it was more than one closet having the issue we turned our focus on the 3750X stack. The configuration was also unchanged, and it was reporting no errors on the ports having the issue. No ULDL errors were present on the 3750X at all. The ports were showing no connection, although we were getting a link light on the other end.
We were able to temporarily remedy the problem by moving the problem closet connections to other ports on the 3750X Stack to get them to light up, with the configurations being the same on the new port. The issue still remains that the old ports, 5 of them, seem to be dead to the world.
This evening we are going to perform a reboot of the 3750X stack in hopes it will clear up the issue with the dead ports, but honestly we're really trying to find out how this happened and if anyone has seen this before. We brought in a vendor to work on it today, but he's as stumped as we are. I'll try to update with firmware information.
Thanks for reading my novel.
First I appreciate the grace as I do not consider myself a Cisco professional at all. We have a couple folks working this problem and I wanted to ask the Experts.
We have 2 3750X switches in a stack configuration, with 1Gb fiber runs to 11 closets throughout our building. The 11 closets all have 2960S 48port switches.
This morning I unplugged the 2960S switches to move them to new individual UPS systems, and while it appeared they came up as normal, about half of them did not have a link light on the 3750X stack. Yes you read that right, one end had a link light, the 2960S, but the 3750 showed no link light for about half of the connections. The configurations were not changed on any of the 2960S's, whether they were working or not after the power back up.
Seeing as it was more than one closet having the issue we turned our focus on the 3750X stack. The configuration was also unchanged, and it was reporting no errors on the ports having the issue. No ULDL errors were present on the 3750X at all. The ports were showing no connection, although we were getting a link light on the other end.
We were able to temporarily remedy the problem by moving the problem closet connections to other ports on the 3750X Stack to get them to light up, with the configurations being the same on the new port. The issue still remains that the old ports, 5 of them, seem to be dead to the world.
This evening we are going to perform a reboot of the 3750X stack in hopes it will clear up the issue with the dead ports, but honestly we're really trying to find out how this happened and if anyone has seen this before. We brought in a vendor to work on it today, but he's as stumped as we are. I'll try to update with firmware information.
Thanks for reading my novel.
SIM50
So ports are not in errdisabled state? Have you tried to do shut no shut?
ASKER
We did attempt shut no shut to no avail.
All ports have the config of:
interface GigabitEthernet1/0/5
switchport trunk encapsulation dot1q
switchport mode trunk
udld port aggressive
Here is a line status of a few of the ports:
Gi2/0/1 connected trunk a-full a-1000 1000BaseSX SFP
Gi2/0/2 notconnect 1 auto auto Not Present
Gi2/0/3 connected trunk a-full a-1000 1000BaseSX SFP
Gi2/0/4 connected trunk a-full a-1000 1000BaseSX SFP
Gi2/0/5 notconnect 1 auto auto Not Present
05 has a closet plugged into it, 02 does not. It looks like nothing is plugged in, although we're getting a link light on the remote switch. The rest are closets that came up normally.
All ports have the config of:
interface GigabitEthernet1/0/5
switchport trunk encapsulation dot1q
switchport mode trunk
udld port aggressive
Here is a line status of a few of the ports:
Gi2/0/1 connected trunk a-full a-1000 1000BaseSX SFP
Gi2/0/2 notconnect 1 auto auto Not Present
Gi2/0/3 connected trunk a-full a-1000 1000BaseSX SFP
Gi2/0/4 connected trunk a-full a-1000 1000BaseSX SFP
Gi2/0/5 notconnect 1 auto auto Not Present
05 has a closet plugged into it, 02 does not. It looks like nothing is plugged in, although we're getting a link light on the remote switch. The rest are closets that came up normally.
ASKER
Version Info:
Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 12.2(58)SE2, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Thu 21-Jul-11 01:23 by prod_rel_team
ROM: Bootstrap program is C3750E boot loader
BOOTLDR: C3750E Boot Loader (C3750X-HBOOT-M) Version 12.2(58r)SE1, RELEASE SOFTWARE (fc1)
MDF202-CORE-FCVA uptime is 2 years, 52 weeks, 15 hours, 36 minutes
System returned to ROM by power-on
System restarted at 01:37:24 CST Mon Aug 26 2013
System image file is "flash:/c3750e-universalk9
Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 12.2(58)SE2, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Thu 21-Jul-11 01:23 by prod_rel_team
ROM: Bootstrap program is C3750E boot loader
BOOTLDR: C3750E Boot Loader (C3750X-HBOOT-M) Version 12.2(58r)SE1, RELEASE SOFTWARE (fc1)
MDF202-CORE-FCVA uptime is 2 years, 52 weeks, 15 hours, 36 minutes
System returned to ROM by power-on
System restarted at 01:37:24 CST Mon Aug 26 2013
System image file is "flash:/c3750e-universalk9
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Good morning,
Thank you for your replies. After limping through the day yesterday we powered off the 3750 stack last night and powered it back on. This seems to have cleared whatever issue up, as I tested a neighbor switch this morning and it came back up as it should. This was one of the problem children yesterday. Craig I ran the commands this morning, which probably won't give you any information after it cleared up last night, but see below:
Switch 1
---------
POST: CPU MIC register Tests : Begin
POST: CPU MIC register Tests : End, Status Passed
POST: MA BIST : Begin
POST: MA BIST : End, Status Passed
POST: TCAM BIST : Begin
POST: TCAM BIST : End, Status Passed
POST: SF ASIC BIST : Begin
POST: SF ASIC BIST : End, Status Passed
POST: Switch Fabric Memory Tests : Begin
POST: Switch Fabric Memory Tests : End, Status Passed
POST: CPU MIC interface Loopback Tests : Begin
POST: CPU MIC interface Loopback Tests : End, Status Passed
POST: PortASIC RingLoopback Tests : Begin
POST: PortASIC RingLoopback Tests : End, Status Passed
POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed
POST: PortASIC Port Loopback Tests : Begin
POST: Failed PortPhyLoopback Packet Test asic_index 1 port_hardware_index 22
POST: PortASIC Port Loopback Tests : End, Status Passed
POST: EMAC Loopback Tests : Begin
POST: EMAC Loopback Tests : End, Status Passed
Switch 2
---------
POST: CPU MIC register Tests : Begin
POST: CPU MIC register Tests : End, Status Passed
POST: MA BIST : Begin
POST: MA BIST : End, Status Passed
POST: TCAM BIST : Begin
POST: TCAM BIST : End, Status Passed
POST: SF ASIC BIST : Begin
POST: SF ASIC BIST : End, Status Passed
POST: Switch Fabric Memory Tests : Begin
POST: Switch Fabric Memory Tests : End, Status Passed
POST: CPU MIC interface Loopback Tests : Begin
POST: CPU MIC interface Loopback Tests : End, Status Passed
POST: PortASIC RingLoopback Tests : Begin
POST: PortASIC RingLoopback Tests : End, Status Passed
POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed
POST: PortASIC Port Loopback Tests : Begin
POST: PortASIC Port Loopback Tests : End, Status Passed
POST: EMAC Loopback Tests : Begin
POST: EMAC Loopback Tests : End, Status Passed
--------------------------
If device is externally calibrated, only calibrated values are printed.
++ : high alarm, + : high warning, - : low warning, -- : low alarm.
NA or N/A: not applicable, Tx: transmit, Rx: receive.
mA: milliamperes, dBm: decibels (milliwatts).
Optical Optical
Temperature Voltage Tx Power Rx Power
Port (Celsius) (Volts) (dBm) (dBm)
--------- ----------- ------- -------- --------
Gi1/0/1 33.8 3.28 -5.0 -6.5
Gi1/0/2 31.4 3.31 -5.0 -5.2
Gi1/0/3 34.5 3.29 -5.0 -5.7
Gi1/0/4 39.3 3.29 -5.1 -5.7
Gi1/0/5 34.0 3.29 -5.0 -5.7
Gi1/0/6 36.5 3.27 -5.0 -5.7
Gi1/0/11 29.7 3.29 -4.9 -5.2
Gi1/0/12 37.5 3.33 -5.0 -5.2
Gi2/0/1 39.3 3.30 -5.0 -5.6
Gi2/0/2 34.4 3.27 -5.0 -5.3
Gi2/0/3 36.7 3.28 -5.0 -5.7
Gi2/0/4 35.1 3.26 -5.0 -5.6
Gi2/0/11 35.3 3.28 -5.0 -5.0
Gi2/0/12 33.1 3.28 -5.0 -4.8
--------------------------
NAME: "1", DESCR: "WS-C3750X-12S"
PID: WS-C3750X-12S-S , VID: V04 , SN:
NAME: "Switch 1 - Power Supply 0", DESCR: "FRU Power Supply"
PID: C3KX-PWR-350WAC , VID: V02L , SN:
NAME: "GigabitEthernet1/0/1", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet1/0/2", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet1/0/3", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet1/0/4", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet1/0/5", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet1/0/6", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet1/0/11", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet1/0/12", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "2", DESCR: "WS-C3750X-12S"
PID: WS-C3750X-12S-S , VID: V04 , SN:
NAME: "Switch 2 - Power Supply 0", DESCR: "FRU Power Supply"
PID: C3KX-PWR-350WAC , VID: V02L , SN:
NAME: "GigabitEthernet2/0/1", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet2/0/2", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet2/0/3", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet2/0/4", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet2/0/5", DESCR: "1000BaseSX SFP"
PID: Unspecified , VID: , SN:
NAME: "GigabitEthernet2/0/11", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet2/0/12", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
Thankfully we had enough un-used ports on the 3750 stack to move the non functioning connections to in order to get by yesterday. I'm still trying to see if I can find out what happened yesterday or if there are any other potential logs files or commands I can run on the 3750.
Again thanks for your time and input.
Thank you for your replies. After limping through the day yesterday we powered off the 3750 stack last night and powered it back on. This seems to have cleared whatever issue up, as I tested a neighbor switch this morning and it came back up as it should. This was one of the problem children yesterday. Craig I ran the commands this morning, which probably won't give you any information after it cleared up last night, but see below:
Switch 1
---------
POST: CPU MIC register Tests : Begin
POST: CPU MIC register Tests : End, Status Passed
POST: MA BIST : Begin
POST: MA BIST : End, Status Passed
POST: TCAM BIST : Begin
POST: TCAM BIST : End, Status Passed
POST: SF ASIC BIST : Begin
POST: SF ASIC BIST : End, Status Passed
POST: Switch Fabric Memory Tests : Begin
POST: Switch Fabric Memory Tests : End, Status Passed
POST: CPU MIC interface Loopback Tests : Begin
POST: CPU MIC interface Loopback Tests : End, Status Passed
POST: PortASIC RingLoopback Tests : Begin
POST: PortASIC RingLoopback Tests : End, Status Passed
POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed
POST: PortASIC Port Loopback Tests : Begin
POST: Failed PortPhyLoopback Packet Test asic_index 1 port_hardware_index 22
POST: PortASIC Port Loopback Tests : End, Status Passed
POST: EMAC Loopback Tests : Begin
POST: EMAC Loopback Tests : End, Status Passed
Switch 2
---------
POST: CPU MIC register Tests : Begin
POST: CPU MIC register Tests : End, Status Passed
POST: MA BIST : Begin
POST: MA BIST : End, Status Passed
POST: TCAM BIST : Begin
POST: TCAM BIST : End, Status Passed
POST: SF ASIC BIST : Begin
POST: SF ASIC BIST : End, Status Passed
POST: Switch Fabric Memory Tests : Begin
POST: Switch Fabric Memory Tests : End, Status Passed
POST: CPU MIC interface Loopback Tests : Begin
POST: CPU MIC interface Loopback Tests : End, Status Passed
POST: PortASIC RingLoopback Tests : Begin
POST: PortASIC RingLoopback Tests : End, Status Passed
POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed
POST: PortASIC Port Loopback Tests : Begin
POST: PortASIC Port Loopback Tests : End, Status Passed
POST: EMAC Loopback Tests : Begin
POST: EMAC Loopback Tests : End, Status Passed
--------------------------
If device is externally calibrated, only calibrated values are printed.
++ : high alarm, + : high warning, - : low warning, -- : low alarm.
NA or N/A: not applicable, Tx: transmit, Rx: receive.
mA: milliamperes, dBm: decibels (milliwatts).
Optical Optical
Temperature Voltage Tx Power Rx Power
Port (Celsius) (Volts) (dBm) (dBm)
--------- ----------- ------- -------- --------
Gi1/0/1 33.8 3.28 -5.0 -6.5
Gi1/0/2 31.4 3.31 -5.0 -5.2
Gi1/0/3 34.5 3.29 -5.0 -5.7
Gi1/0/4 39.3 3.29 -5.1 -5.7
Gi1/0/5 34.0 3.29 -5.0 -5.7
Gi1/0/6 36.5 3.27 -5.0 -5.7
Gi1/0/11 29.7 3.29 -4.9 -5.2
Gi1/0/12 37.5 3.33 -5.0 -5.2
Gi2/0/1 39.3 3.30 -5.0 -5.6
Gi2/0/2 34.4 3.27 -5.0 -5.3
Gi2/0/3 36.7 3.28 -5.0 -5.7
Gi2/0/4 35.1 3.26 -5.0 -5.6
Gi2/0/11 35.3 3.28 -5.0 -5.0
Gi2/0/12 33.1 3.28 -5.0 -4.8
--------------------------
NAME: "1", DESCR: "WS-C3750X-12S"
PID: WS-C3750X-12S-S , VID: V04 , SN:
NAME: "Switch 1 - Power Supply 0", DESCR: "FRU Power Supply"
PID: C3KX-PWR-350WAC , VID: V02L , SN:
NAME: "GigabitEthernet1/0/1", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet1/0/2", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet1/0/3", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet1/0/4", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet1/0/5", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet1/0/6", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet1/0/11", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet1/0/12", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "2", DESCR: "WS-C3750X-12S"
PID: WS-C3750X-12S-S , VID: V04 , SN:
NAME: "Switch 2 - Power Supply 0", DESCR: "FRU Power Supply"
PID: C3KX-PWR-350WAC , VID: V02L , SN:
NAME: "GigabitEthernet2/0/1", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet2/0/2", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet2/0/3", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet2/0/4", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet2/0/5", DESCR: "1000BaseSX SFP"
PID: Unspecified , VID: , SN:
NAME: "GigabitEthernet2/0/11", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
NAME: "GigabitEthernet2/0/12", DESCR: "1000BaseSX SFP"
PID: GLC-SX-MMD , VID: V01 , SN:
Thankfully we had enough un-used ports on the 3750 stack to move the non functioning connections to in order to get by yesterday. I'm still trying to see if I can find out what happened yesterday or if there are any other potential logs files or commands I can run on the 3750.
Again thanks for your time and input.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
They were actually on both switches. We had that thought initially, because it was almost exactly half of our closets that went down. 3 on the top, 2 on the bottom if I remember correctly.
I will certainly open a TAC case for the above message to see what that's all about.
Again thanks for the guidance and if I have a follow up within a reasonable time I'll update the thread.
Thanks Craig.
I will certainly open a TAC case for the above message to see what that's all about.
Again thanks for the guidance and if I have a follow up within a reasonable time I'll update the thread.
Thanks Craig.