Link to home
Start Free TrialLog in
Avatar of Dan
DanFlag for United States of America

asked on

Cisco Switch error for spanning ree

I've been having some network issues, where for a moment or so, the internet and just network access stops working for almost everyone.

I started digging in the logs on some of the switches, and I came across this at about the exact time when this occurred.
What exactly does this mean?

      2486: Sep 21 19:57:57.069: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking GigabitEthernet1/0/1 on VLAN0001. Port consistency restored.

How can I further troubleshoot my network issue?
Avatar of Jane Updegraff
Jane Updegraff
Flag of United States of America image

what model of switch and what version of the ios are you using here?
Avatar of Dan

ASKER

catalyst 3750

Ver 12.2(44) SE5
ok in the same log, do you see adjacent messages that say

"%SPANTREE-2-BLOCK_PVID_PEER: Blocking GigabitEthernet1/0/1 on VLAN0001. Inconsistent peer vlan.

OR maybe

%SPANTREE-2-BLOCK_PVID_LOCAL: Blocking GigabitEthernet1/0/1 on VLAN0001. Inconsistent local vlan.

or even

%SPANTREE-2-RECV_PVID_ERR: Received BPDU with inconsistent peer vlan id 1 onGigabitEthernet1/0/1 VLAN0001.

or the last one might reference a different vlan on the end. is there anything looking like that in the log?

perhaps it would be best to post the whole log from several minutes both before and after the event?
Avatar of Dan

ASKER

Here's everything that is in the log from yesterday until now.

This error occured at 12:49 pm today.

9/21/2017 2:08:16 PM      192.168.100.39      2496: Sep 21 21:08:16.384: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/2, changed state to up                  5            23                        
9/21/2017 2:08:16 PM      192.168.100.39      2495: Sep 21 21:08:15.378: %LINK-3-UPDOWN: Interface FastEthernet1/0/2, changed state to up                  3            23                        
9/21/2017 2:08:13 PM      192.168.100.39      2494: Sep 21 21:08:12.106: %LINK-3-UPDOWN: Interface FastEthernet1/0/2, changed state to down                  3            23                        
9/21/2017 2:08:12 PM      192.168.100.39      2493: Sep 21 21:08:11.108: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/2, changed state to down                  5            23                        
9/21/2017 2:07:53 PM      192.168.100.39      2492: Sep 21 21:07:52.250: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/2, changed state to up                  5            23                        
9/21/2017 2:07:52 PM      192.168.100.39      2491: Sep 21 21:07:51.243: %LINK-3-UPDOWN: Interface FastEthernet1/0/2, changed state to up                  3            23                        
9/21/2017 2:07:24 PM      192.168.100.39      2490: Sep 21 21:07:22.763: %LINK-3-UPDOWN: Interface FastEthernet1/0/2, changed state to down                  3            23                        
9/21/2017 2:07:22 PM      192.168.100.39      2489: Sep 21 21:07:21.765: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/2, changed state to down                  5            23                        
9/21/2017 12:57:58 PM      192.168.100.39      2486: Sep 21 19:57:57.069: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking GigabitEthernet1/0/1 on VLAN0001. Port consistency restored.                  2            23                        
9/19/2017 6:08:56 AM      192.168.100.39      2485: Sep 19 13:08:56.035: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/3, changed state to up
Interesting that the spanning tree protocol block message isn't shown at all before the unblock message. Not sure why that would be unless the block was triggered way before ... which you would have definitely noticed. Anyhow ... OK well normally anything to do with spanning tree has to do with the device's STP (spanning tree protocol) or the related BPDU guard.. so that means that you may have a spanning tree loop. In my experience those have been caused by using unmanaged ("dumb") switches or those cheap little 4-port hubs that are not capable of STP and therefore cannot block or prevent spanning tree loops. Why are ST loops bad? Well, when you have devices that are not STP capable, you can have something happen such as an end user (or a cleaning person) may knock a cable out of a wall socket and inadvertently plugs it back into the wrong place so that both ends of the ethernet cable are plugged into the same dumb hub or switch. The loop between the two ports on the "dumb" hub feeds back (just like a microphone squealing when a speaker is being picked up by a microphone) and loads up the port traffic to where it becomes unusable. A "real" managed switch then detects the problem and as a fail-safe, it blocks the traffic on a port or interface.

Have you already looked for obvious places where a loop may have been created? Are you using unmanaged hubs or switches in your environment?

If not then it may be just a bug. if so then see if you can find it in the bug db at Cisco. While there, you might want to update your ios, which is a bit out of date. Bugs are regularly addressed in ios updates so it's always advisable to keep your ios updated. You can get the updates and recommendations for whatever license you have at Cisco's site.
Avatar of Dan

ASKER

I do have some unmanaged dumb switches because we have over grown and have to many people in one office, so the cheapest solution was to install the PNP switches.
I have about 11 managed switches and in the next few weeks, I'll be upgrading all our switches to only 3750 G or 2960S switches with newer IOS installed.

It was just strange as for like a minute or two, the network just dropped, so it's very hard to troubleshoot where the problem starts.
Are there any software programs out there that might assist me in finding the root cause?
That's a really good question. I don't think so but I'll look anyway. That would be great if there were such a thing. I'll see what i can find.
Avatar of Dan

ASKER

it just happened again.  Here's what I found on one switch.  Since I'm using PRTG for my syslog, I have to check the logs of every single switch, so I just bought kiwi syslog, so I'm hoping I'll configure that in the next few days, as I will have one repository for all switches that should make it easier to see all the messages at once.

9/25/2017 5:03:36 PM      192.168.100.45      2219: Sep 25 17:03:34.543 pst: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking GigabitEthernet3/0/1 on VLAN0001. Port consistency restored.                  2            23                        
9/25/2017 5:02:53 PM      192.168.100.45      2216: Sep 25 17:02:52.063 pst: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking GigabitEthernet3/0/1 on VLAN0001. Port consistency restored.                  2            23                        
9/25/2017 9:49:13 AM      192.168.100.45      2214: Sep 25 09:49:13.600 pst: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet3/0/32, changed state to up
ASKER CERTIFIED SOLUTION
Avatar of Jane Updegraff
Jane Updegraff
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of Dan

ASKER

There's no Gi32 port, there's only 4 gig ports on this switch.  Let me know if you need to see anything else?



3750_48_POE_100.45#sh int gi3/0/1
GigabitEthernet3/0/1 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 001e.4abc.a681 (bia 001e.4abc.a681)
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 1000Mb/s, link type is auto, media type is 10/100/1000BaseTX SFP
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:02, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/276/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 570000 bits/sec, 157 packets/sec
  5 minute output rate 279000 bits/sec, 101 packets/sec
     861759611 packets input, 699699162526 bytes, 0 no buffer
     Received 144910695 broadcasts (36277257 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 36277257 multicast, 0 pause input
     0 input packets with dribble condition detected
     491799543 packets output, 170304775472 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
3750_48_POE_100.45#


3750_48_POE_100.45#
3750_48_POE_100.45#sh int gi3/0/1 trunk

Port        Mode             Encapsulation  Status        Native vlan
Gi3/0/1     auto             negotiate      not-trunking  1

Port        Vlans allowed on trunk
Gi3/0/1     1

Port        Vlans allowed and active in management domain
Gi3/0/1     1

Port        Vlans in spanning tree forwarding state and not pruned
Gi3/0/1     1
3750_48_POE_100.45#

3750_48_POE_100.45#sh ip int brief
Interface              IP-Address      OK? Method Status                Protocol
Vlan1                  192.168.100.45  YES NVRAM  up                    up
FastEthernet3/0/1      unassigned      YES unset  down                  down
FastEthernet3/0/2      unassigned      YES unset  up                    up
FastEthernet3/0/3      unassigned      YES unset  down                  down
FastEthernet3/0/4      unassigned      YES unset  up                    up
FastEthernet3/0/5      unassigned      YES unset  up                    up
FastEthernet3/0/6      unassigned      YES unset  up                    up
FastEthernet3/0/7      unassigned      YES unset  down                  down
FastEthernet3/0/8      unassigned      YES unset  up                    up
FastEthernet3/0/9      unassigned      YES unset  up                    up
FastEthernet3/0/10     unassigned      YES unset  up                    up
FastEthernet3/0/11     unassigned      YES unset  down                  down
FastEthernet3/0/12     unassigned      YES unset  down                  down
FastEthernet3/0/13     unassigned      YES unset  up                    up
FastEthernet3/0/14     unassigned      YES unset  up                    up
FastEthernet3/0/15     unassigned      YES unset  down                  down
FastEthernet3/0/16     unassigned      YES unset  down                  down
FastEthernet3/0/17     unassigned      YES unset  up                    up
FastEthernet3/0/18     unassigned      YES unset  up                    up
FastEthernet3/0/19     unassigned      YES unset  down                  down
FastEthernet3/0/20     unassigned      YES unset  down                  down
FastEthernet3/0/21     unassigned      YES unset  down                  down
FastEthernet3/0/22     unassigned      YES unset  down                  down
FastEthernet3/0/23     unassigned      YES unset  down                  down
FastEthernet3/0/24     unassigned      YES unset  down                  down
FastEthernet3/0/25     unassigned      YES unset  down                  down
FastEthernet3/0/26     unassigned      YES unset  down                  down
FastEthernet3/0/27     unassigned      YES unset  up                    up
FastEthernet3/0/28     unassigned      YES unset  down                  down
FastEthernet3/0/29     unassigned      YES unset  up                    up
FastEthernet3/0/30     unassigned      YES unset  up                    up
FastEthernet3/0/31     unassigned      YES unset  down                  down
FastEthernet3/0/32     unassigned      YES unset  up                    up
FastEthernet3/0/33     unassigned      YES unset  down                  down
FastEthernet3/0/34     unassigned      YES unset  up                    up
FastEthernet3/0/35     unassigned      YES unset  down                  down
FastEthernet3/0/36     unassigned      YES unset  up                    up
FastEthernet3/0/37     unassigned      YES unset  down                  down
FastEthernet3/0/38     unassigned      YES unset  down                  down
FastEthernet3/0/39     unassigned      YES unset  down                  down
FastEthernet3/0/40     unassigned      YES unset  up                    up
FastEthernet3/0/41     unassigned      YES unset  down                  down
FastEthernet3/0/42     unassigned      YES unset  up                    up
FastEthernet3/0/43     unassigned      YES unset  up                    up
FastEthernet3/0/44     unassigned      YES unset  down                  down
FastEthernet3/0/45     unassigned      YES unset  up                    up
FastEthernet3/0/46     unassigned      YES unset  up                    up
FastEthernet3/0/47     unassigned      YES unset  down                  down
FastEthernet3/0/48     unassigned      YES unset  up                    up
GigabitEthernet3/0/1   unassigned      YES unset  up                    up
GigabitEthernet3/0/2   unassigned      YES unset  down                  down
GigabitEthernet3/0/3   unassigned      YES unset  down                  down
GigabitEthernet3/0/4   unassigned      YES unset  down                  down
3750_48_POE_100.45#


3750_48_POE_100.45#sh int fa3/0/32
FastEthernet3/0/32 is up, line protocol is up (connected)
  Hardware is Fast Ethernet, address is 001e.4abc.a6a4 (bia 001e.4abc.a6a4)
  MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 10Mb/s, media type is 10/100BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 526762
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 17000 bits/sec, 21 packets/sec
     617798 packets input, 60469616 bytes, 0 no buffer
     Received 49760 broadcasts (36910 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 36910 multicast, 0 pause input
     0 input packets with dribble condition detected
     172774496 packets output, 31579120882 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
3750_48_POE_100.45#



3750_48_POE_100.45#sh int fa3/0/32 trunk

Port        Mode             Encapsulation  Status        Native vlan
Fa3/0/32    auto             negotiate      not-trunking  1

Port        Vlans allowed on trunk
Fa3/0/32    1

Port        Vlans allowed and active in management domain
Fa3/0/32    1

Port        Vlans in spanning tree forwarding state and not pruned
Fa3/0/32    1
3750_48_POE_100.45#
Avatar of Dan

ASKER

Thanks for your input. I found out what the issue was, or at least I think I did.
I replaced my core switch, it was a Dell, and since then, Monday night, the issue has not resurfaced at all, and it's been 2 days.
I think there was an incompatibility for spanning-tree between my dell and my cisco.
Sorry i didn't get back to you on this question!

that would not surprise me that a Dell uses a different spanning tree setup than a Cisco, but you would think that they would both use STP ... although he Cisco also uses bpdu guard, and that might have had something to do with it. But glad you got it resolved one way or another!