Network freezing - all based on C2950-24-EI switches + canopy mirowaves

Posted on 2011-10-26
Last Modified: 2012-05-12
architecture:  16 locations (subcenter aka SC) goes through 70MT switch (C2950 too) to  1 main center (aka WCC) (another C2950)

- SCADA system distributed through the 16 SC's &  WCC:

-in each SC there is 1 cisco 2950 switch
- each switch in all places are having 3 Vlans: 1 (native) ,31 and 32 (separated network desgin)

each device (d1,d2..) in the network has 2 interfaces (separated)
 one goes to vlan 31, start with 172.31.x.x
the other goes to vlan 32, starts with 172.32.x.x
(for  network redundancy)

-between each SC & WCC the connection through microwave link
(Motorola canopy cyclone, 2.4 GHz & 5.7 GHz radio modules)

-switch ports are assigned as follows:

*Fa0/1 - 2 are trunks, vlan 1, fa0/1 have the 2.4 Ghz link connected (+ power injector) 5.7 on fa0/2

*Fa0/3-12 are Vlan 31
*Fa0/13-24 are Vlan32
*VTP mode is transparent
*spanning tree mode is pvst

-comm. tower has (3x120 degree + 1 P2P )AP's for 2.4 Ghz, and another (3x120 degree + 1 P2P) AP's for 5.7
- P2P used for very far location

-all 4 2.4 Ghz cables goes through canopy board (+gps cables for sync) to canopy cube switch that has 1 cable output goes to 70MT switch's Fa0/1 which is configured as trunk

-all 4 5.7 Ghz cables goes through canopy board (+gps cables for sync) to canopy cube switch that has 1 cable output goes to 70MT switch's Fa0/2 which is configured as trunk

-Gi0/1 connects 70MT switch to WCC switch

Gi0/1 connects to 70MT tower
Gi0/2 disabled for troubleshooting
1-12 vlan 31
13-24 vlan 32
servers and workstations with dual ports connects to both vlans via same address scheme.


-network is jamming/freezing, sometime telneting / pinging an adjacent switch will result in total failure/ slowness and delays while typing in the telnet terminal

- there was bridged devices in the SC's, ive disabled them all either physically or via shutdown the port

- through wireshark and monitor session, i've disabled all broadcasting devices

-when new bridged device is intalled in SC, the 70MT switch in show spanning tree status will show one of the Fa0/1 or 2 is blocked, i would do some commands in the SC to block the binstalled bridge, and both ports in 70MT switch will be in FWD mode.

- since yesterday network was jamming, 70MT tower had both ports to the SC tree in FWD, even now one of the ports is blocked, yet there is jamming

-notes: some SC won't communicate on both 2.4 / 5.7 as the module/signal at the SC or corresponding AP at WCC side might have a problem

shouldn't default cisco config will take care of loops and detect 70T as the root switch by default ?

-installed firmware (IOS) goes back to 2005, any good for installing the most recent ones ? (2010) ?

 comm architevture
Question by:mohsh86
    LVL 28

    Expert Comment

    There's a lot in your question, and a lot of things that could be wrong, but let's start at the end:
    1. Yes, installing more recent software in your switches may be a good idea. But I'd guessing that this is not the problem
    2. Regarding the default Cisco config: Yes, it will take care of loops. No, it will NOT figure out which switch you wanted to act as the root. Spanning Tree Protocol will pick the switch with the lowest MAC address as the root unless you do something to influence it. the way to do this is (on the switch that you want as root):
    config t
    spanning-tree vlan 1-999 priority 0

    If you want a secondary root on another switch:
    spanning-tree vlan 1-999 priority 4096

    If they support it, I suggest you move to rapid spanning tree. This will give you sub-second failover times instead of up to 40 seconds, but your switches may or may not support it, depending on IOS version (maybe a reason to install more recent software):
    spanning-tree mode rapid-pvst
    LVL 28

    Expert Comment

    Also, I'm not sure why you have 2 vlans and then connect all of your devices to both of them. This has the potential to cause routing or firewall issues with your hosts, and also to create spanning tree loops if the devices start bridging between the vlans. A much better way, if the end hosts support it, is to do NIC teaming (Windows) or bonding (Linux) and put both NICs in the same vlan with a shared virtual IP address. Set them up as active/standby.

    Author Comment

    thanks for the reply i'll try the root command:

    1- the network configuration( 2vlans) are for the sake of (stupid redundancy), the SCADA software will keep try to communicate on the 172.31.x.x then if failed on the 172.32.x.x

    2- there is no routers/firewalls (actually its connected to WCC switch in the figure, but i disconnect it for the sake of troubleshooting (still problem is there after disconnection)

    Author Comment

    ok, the spanning-tree vlan 1-999 priority 0 works, but herer is the interesting thing and a thing i forgot to mention:

    1. mac access list is created on SC switches's Fa0/1 & Fa0/2 ports so that it will allow only WCC servers and workstations to access.

    2. do i have to inclue the mac address of the 70MT switch ? so that what every spanning tree protocol messages should be allowed into the peer SC switch ? of the SC has a new switch installed (C2960), this switch specifcally has both Fa0/1 & Fa0/2 in FWD mode, i think the problem was from it, because in keeping one of them only, the network freezing is gone.

    why this is happening in the C2960 switch ?

    solve it and the 500 points is yours :)
    LVL 28

    Expert Comment

    Regarding the MAC access list- yes, you absolutely have to include the MAC of every connected device. A cleaner solution than a MAC address access list is Port Security, which locks down the port to the first MAC address seen on the port, unless you shut down the port first. But you should not apply this to the port where the 70MT switch connects. It's always best to apply this kind of security as close to the adge as possible, which means the actual ports that rogue devices could connect on.

    I have no idea about the 2960 switch without a lot more information. But of both ports were forwarding, I suspect that it was the root bridge. The network freezing could have been due to spanning tree topology changes if the Microwave connectivity was interrupted. It would be interesting now that you have set a root, to turn on both ports and see if both are still forwarding.

    Author Comment

    thank you for the information,

    the thing is why i used access list and not port security is that i need only specific devices to be able to access the SC's switches, why is that because previously by mistake and Engineering workstation for PLC stopped a running process in another SC which was about to lead to a disaster.

    which i don't understand is that the C2960 had the same configration like the rest of the SC switches, why it thought it's the root bridge ?

    how to force a switch to be a peer (end of the tree) bridge and NOT to be a root bridge ?

    LVL 28

    Expert Comment

    The way the spanning tree protocol works is that ALL switches regardless of brand, model, etc. have the same default settings for root priority. How is the root chosen? The lowest MAC address wins! That switch must have the lowest MAC address so it became the root. That's how spanning tree works and there is nothing wrong.

    The default root priority value is 32768. If you lower that value on a switch, it will become the root if the others are all default. If you raise that value on a switch, it will never become root if there are other switches around with a lower value.

    In my earlier post I told you how to set the root, and also a secondary root in case the primary root fails. It's much better to choose your root by setting a lower priority number than to tell a switch NOT to become root.

    Author Comment

    the thing is i told a root switch to become root, the new C2960 switch when connected (peer switch in SC) had the 2 ports in FW and start the network interruption, maybe because the root bridge MAC address is not on the C2960 access list ?
    LVL 28

    Accepted Solution

    That is very likely. If the MAC addresses aren't permitted, spanning tree traffic would not be permitted and that switch would not see a loop from the 2 connected ports, so it would forward on both of them, creating a loop.

    I would suggest un-applying your mac address access lists temporarily and see what happens. If that solves the problem, look at the mac addresses learned on the ports and add them to the access list.

    The problem with MAC address lists though is that if you have to replace a device or a NIC, you will have to update your MAC access lists. If you don't want rogue devices in your network, why not just shut down the ports that should not be in use. And with port security, only the device that is plugged in when you enable the port can use that port unless it's shut down first. So on a live port, someone can't unplug something and plug in another device.

    Author Comment

    because as per the drawn architecture , the peers (SC's) are able to access each other which we want to prevent

    i'll check the Mac Access list

    Author Comment

    ok here is the thing:

    - in the old switches C2950 its allowing STP packets even with mac access list enabled (latest IOS)
    - in the new C2960 switch, its preventing the STP packets

    for some reason adding the mac address for 70MT tower to access list didn't solve the problem, by creating monitoring seassion and monitor via wireshark, i've seen the STP packets comes at the same MAC address of 70MT tower mac but ends with +1 & +2 of the Switch Main mac address, maybe thats Fa0/1 & Fa0/2 mac address of the 70MT through which the 2.4 GHz and 5.7 GHz links is connected

    problem solved thank you

    Author Comment

    I've requested that this question be closed as follows:

    Accepted answer: 500 points for mikebernhardt's comment http:/Q_27416992.html#37059668
    Assisted answer: 0 points for mohsh86's comment http:/Q_27416992.html#37061534

    for the following reason:

    both tricks worked
    LVL 28

    Expert Comment

    If you want to accept my answer, you should click on the accept button, not ask to close the question.

    By the way, you are likely correct about the +1 and +2 incemements of the mac address on the switch. I'm sure the moderator will tell you that too. Glad I could help!
    LVL 28

    Expert Comment

    If you want to accept my answer, you should click on the accept button, not ask to close the question. Glad I could help!

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    Scale it in WD Gold

    With up to ten times the workload capacity of desktop drives, WD Gold hard drives employ advanced technology to deliver among the best in reliability, capacity, power efficiency and performance.

    AWS has developed and created its highly available global infrastructure allowing users to deploy and manage their estates all across the world through the use of the following geographical components   RegionsAvailability ZonesEdge Locations  Wh…
    Join Greg Farro and Ethan Banks from Packet Pushers ( and Greg Ross from Paessler ( for a discussion about smart network …
    After creating this article (, I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
    In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor ( If you're interested in additional methods for monitoring bandwidt…

    761 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    8 Experts available now in Live!

    Get 1:1 Help Now