Network freezing - all based on C2950-24-EI switches + canopy mirowaves
architecture: 16 locations (subcenter aka SC) goes through 70MT switch (C2950 too) to 1 main center (aka WCC) (another C2950)
- SCADA system distributed through the 16 SC's & WCC:
-in each SC there is 1 cisco 2950 switch
- each switch in all places are having 3 Vlans: 1 (native) ,31 and 32 (separated network desgin)
each device (d1,d2..) in the network has 2 interfaces (separated)
one goes to vlan 31, start with 172.31.x.x
the other goes to vlan 32, starts with 172.32.x.x
(for network redundancy)
-between each SC & WCC the connection through microwave link
(Motorola canopy cyclone, 2.4 GHz & 5.7 GHz radio modules)
SC:
-switch ports are assigned as follows:
*Fa0/1 - 2 are trunks, vlan 1, fa0/1 have the 2.4 Ghz link connected (+ power injector) 5.7 on fa0/2
*Fa0/3-12 are Vlan 31
*Fa0/13-24 are Vlan32
*VTP mode is transparent
*spanning tree mode is pvst
70MT:
-comm. tower has (3x120 degree + 1 P2P )AP's for 2.4 Ghz, and another (3x120 degree + 1 P2P) AP's for 5.7
- P2P used for very far location
-all 4 2.4 Ghz cables goes through canopy board (+gps cables for sync) to canopy cube switch that has 1 cable output goes to 70MT switch's Fa0/1 which is configured as trunk
-all 4 5.7 Ghz cables goes through canopy board (+gps cables for sync) to canopy cube switch that has 1 cable output goes to 70MT switch's Fa0/2 which is configured as trunk
-Gi0/1 connects 70MT switch to WCC switch
WCC:
Gi0/1 connects to 70MT tower
Gi0/2 disabled for troubleshooting
1-12 vlan 31
13-24 vlan 32
servers and workstations with dual ports connects to both vlans via same address scheme.
problems:
-network is jamming/freezing, sometime telneting / pinging an adjacent switch will result in total failure/ slowness and delays while typing in the telnet terminal
- there was bridged devices in the SC's, ive disabled them all either physically or via shutdown the port
- through wireshark and monitor session, i've disabled all broadcasting devices
-when new bridged device is intalled in SC, the 70MT switch in show spanning tree status will show one of the Fa0/1 or 2 is blocked, i would do some commands in the SC to block the binstalled bridge, and both ports in 70MT switch will be in FWD mode.
- since yesterday network was jamming, 70MT tower had both ports to the SC tree in FWD, even now one of the ports is blocked, yet there is jamming
-notes: some SC won't communicate on both 2.4 / 5.7 as the module/signal at the SC or corresponding AP at WCC side might have a problem
shouldn't default cisco config will take care of loops and detect 70T as the root switch by default ?
-installed firmware (IOS) goes back to 2005, any good for installing the most recent ones ? (2010) ?
- SCADA system distributed through the 16 SC's & WCC:
-in each SC there is 1 cisco 2950 switch
- each switch in all places are having 3 Vlans: 1 (native) ,31 and 32 (separated network desgin)
each device (d1,d2..) in the network has 2 interfaces (separated)
one goes to vlan 31, start with 172.31.x.x
the other goes to vlan 32, starts with 172.32.x.x
(for network redundancy)
-between each SC & WCC the connection through microwave link
(Motorola canopy cyclone, 2.4 GHz & 5.7 GHz radio modules)
SC:
-switch ports are assigned as follows:
*Fa0/1 - 2 are trunks, vlan 1, fa0/1 have the 2.4 Ghz link connected (+ power injector) 5.7 on fa0/2
*Fa0/3-12 are Vlan 31
*Fa0/13-24 are Vlan32
*VTP mode is transparent
*spanning tree mode is pvst
70MT:
-comm. tower has (3x120 degree + 1 P2P )AP's for 2.4 Ghz, and another (3x120 degree + 1 P2P) AP's for 5.7
- P2P used for very far location
-all 4 2.4 Ghz cables goes through canopy board (+gps cables for sync) to canopy cube switch that has 1 cable output goes to 70MT switch's Fa0/1 which is configured as trunk
-all 4 5.7 Ghz cables goes through canopy board (+gps cables for sync) to canopy cube switch that has 1 cable output goes to 70MT switch's Fa0/2 which is configured as trunk
-Gi0/1 connects 70MT switch to WCC switch
WCC:
Gi0/1 connects to 70MT tower
Gi0/2 disabled for troubleshooting
1-12 vlan 31
13-24 vlan 32
servers and workstations with dual ports connects to both vlans via same address scheme.
problems:
-network is jamming/freezing, sometime telneting / pinging an adjacent switch will result in total failure/ slowness and delays while typing in the telnet terminal
- there was bridged devices in the SC's, ive disabled them all either physically or via shutdown the port
- through wireshark and monitor session, i've disabled all broadcasting devices
-when new bridged device is intalled in SC, the 70MT switch in show spanning tree status will show one of the Fa0/1 or 2 is blocked, i would do some commands in the SC to block the binstalled bridge, and both ports in 70MT switch will be in FWD mode.
- since yesterday network was jamming, 70MT tower had both ports to the SC tree in FWD, even now one of the ports is blocked, yet there is jamming
-notes: some SC won't communicate on both 2.4 / 5.7 as the module/signal at the SC or corresponding AP at WCC side might have a problem
shouldn't default cisco config will take care of loops and detect 70T as the root switch by default ?
-installed firmware (IOS) goes back to 2005, any good for installing the most recent ones ? (2010) ?
Also, I'm not sure why you have 2 vlans and then connect all of your devices to both of them. This has the potential to cause routing or firewall issues with your hosts, and also to create spanning tree loops if the devices start bridging between the vlans. A much better way, if the end hosts support it, is to do NIC teaming (Windows) or bonding (Linux) and put both NICs in the same vlan with a shared virtual IP address. Set them up as active/standby.
ASKER
thanks for the reply i'll try the root command:
1- the network configuration( 2vlans) are for the sake of (stupid redundancy), the SCADA software will keep try to communicate on the 172.31.x.x then if failed on the 172.32.x.x
2- there is no routers/firewalls (actually its connected to WCC switch in the figure, but i disconnect it for the sake of troubleshooting (still problem is there after disconnection)
1- the network configuration( 2vlans) are for the sake of (stupid redundancy), the SCADA software will keep try to communicate on the 172.31.x.x then if failed on the 172.32.x.x
2- there is no routers/firewalls (actually its connected to WCC switch in the figure, but i disconnect it for the sake of troubleshooting (still problem is there after disconnection)
ASKER
ok, the spanning-tree vlan 1-999 priority 0 works, but herer is the interesting thing and a thing i forgot to mention:
1. mac access list is created on SC switches's Fa0/1 & Fa0/2 ports so that it will allow only WCC servers and workstations to access.
2. do i have to inclue the mac address of the 70MT switch ? so that what every spanning tree protocol messages should be allowed into the peer SC switch ?
3.one of the SC has a new switch installed (C2960), this switch specifcally has both Fa0/1 & Fa0/2 in FWD mode, i think the problem was from it, because in keeping one of them only, the network freezing is gone.
why this is happening in the C2960 switch ?
solve it and the 500 points is yours :)
1. mac access list is created on SC switches's Fa0/1 & Fa0/2 ports so that it will allow only WCC servers and workstations to access.
2. do i have to inclue the mac address of the 70MT switch ? so that what every spanning tree protocol messages should be allowed into the peer SC switch ?
3.one of the SC has a new switch installed (C2960), this switch specifcally has both Fa0/1 & Fa0/2 in FWD mode, i think the problem was from it, because in keeping one of them only, the network freezing is gone.
why this is happening in the C2960 switch ?
solve it and the 500 points is yours :)
Regarding the MAC access list- yes, you absolutely have to include the MAC of every connected device. A cleaner solution than a MAC address access list is Port Security, which locks down the port to the first MAC address seen on the port, unless you shut down the port first. But you should not apply this to the port where the 70MT switch connects. It's always best to apply this kind of security as close to the adge as possible, which means the actual ports that rogue devices could connect on.
I have no idea about the 2960 switch without a lot more information. But of both ports were forwarding, I suspect that it was the root bridge. The network freezing could have been due to spanning tree topology changes if the Microwave connectivity was interrupted. It would be interesting now that you have set a root, to turn on both ports and see if both are still forwarding.
I have no idea about the 2960 switch without a lot more information. But of both ports were forwarding, I suspect that it was the root bridge. The network freezing could have been due to spanning tree topology changes if the Microwave connectivity was interrupted. It would be interesting now that you have set a root, to turn on both ports and see if both are still forwarding.
ASKER
thank you for the information,
the thing is why i used access list and not port security is that i need only specific devices to be able to access the SC's switches, why is that because previously by mistake and Engineering workstation for PLC stopped a running process in another SC which was about to lead to a disaster.
which i don't understand is that the C2960 had the same configration like the rest of the SC switches, why it thought it's the root bridge ?
how to force a switch to be a peer (end of the tree) bridge and NOT to be a root bridge ?
thanks
the thing is why i used access list and not port security is that i need only specific devices to be able to access the SC's switches, why is that because previously by mistake and Engineering workstation for PLC stopped a running process in another SC which was about to lead to a disaster.
which i don't understand is that the C2960 had the same configration like the rest of the SC switches, why it thought it's the root bridge ?
how to force a switch to be a peer (end of the tree) bridge and NOT to be a root bridge ?
thanks
The way the spanning tree protocol works is that ALL switches regardless of brand, model, etc. have the same default settings for root priority. How is the root chosen? The lowest MAC address wins! That switch must have the lowest MAC address so it became the root. That's how spanning tree works and there is nothing wrong.
The default root priority value is 32768. If you lower that value on a switch, it will become the root if the others are all default. If you raise that value on a switch, it will never become root if there are other switches around with a lower value.
In my earlier post I told you how to set the root, and also a secondary root in case the primary root fails. It's much better to choose your root by setting a lower priority number than to tell a switch NOT to become root.
The default root priority value is 32768. If you lower that value on a switch, it will become the root if the others are all default. If you raise that value on a switch, it will never become root if there are other switches around with a lower value.
In my earlier post I told you how to set the root, and also a secondary root in case the primary root fails. It's much better to choose your root by setting a lower priority number than to tell a switch NOT to become root.
ASKER
the thing is i told a root switch to become root, the new C2960 switch when connected (peer switch in SC) had the 2 ports in FW and start the network interruption, maybe because the root bridge MAC address is not on the C2960 access list ?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
because as per the drawn architecture , the peers (SC's) are able to access each other which we want to prevent
i'll check the Mac Access list
i'll check the Mac Access list
ASKER
ok here is the thing:
- in the old switches C2950 its allowing STP packets even with mac access list enabled (latest IOS)
- in the new C2960 switch, its preventing the STP packets
for some reason adding the mac address for 70MT tower to access list didn't solve the problem, by creating monitoring seassion and monitor via wireshark, i've seen the STP packets comes at the same MAC address of 70MT tower mac but ends with +1 & +2 of the Switch Main mac address, maybe thats Fa0/1 & Fa0/2 mac address of the 70MT through which the 2.4 GHz and 5.7 GHz links is connected
problem solved thank you
- in the old switches C2950 its allowing STP packets even with mac access list enabled (latest IOS)
- in the new C2960 switch, its preventing the STP packets
for some reason adding the mac address for 70MT tower to access list didn't solve the problem, by creating monitoring seassion and monitor via wireshark, i've seen the STP packets comes at the same MAC address of 70MT tower mac but ends with +1 & +2 of the Switch Main mac address, maybe thats Fa0/1 & Fa0/2 mac address of the 70MT through which the 2.4 GHz and 5.7 GHz links is connected
problem solved thank you
ASKER
I've requested that this question be closed as follows:
Accepted answer: 500 points for mikebernhardt's comment http:/Q_27416992.html#37059668
Assisted answer: 0 points for mohsh86's comment http:/Q_27416992.html#37061534
for the following reason:
both tricks worked
Accepted answer: 500 points for mikebernhardt's comment http:/Q_27416992.html#37059668
Assisted answer: 0 points for mohsh86's comment http:/Q_27416992.html#37061534
for the following reason:
both tricks worked
If you want to accept my answer, you should click on the accept button, not ask to close the question.
By the way, you are likely correct about the +1 and +2 incemements of the mac address on the switch. I'm sure the moderator will tell you that too. Glad I could help!
By the way, you are likely correct about the +1 and +2 incemements of the mac address on the switch. I'm sure the moderator will tell you that too. Glad I could help!
If you want to accept my answer, you should click on the accept button, not ask to close the question. Glad I could help!
1. Yes, installing more recent software in your switches may be a good idea. But I'd guessing that this is not the problem
2. Regarding the default Cisco config: Yes, it will take care of loops. No, it will NOT figure out which switch you wanted to act as the root. Spanning Tree Protocol will pick the switch with the lowest MAC address as the root unless you do something to influence it. the way to do this is (on the switch that you want as root):
config t
spanning-tree vlan 1-999 priority 0
If you want a secondary root on another switch:
spanning-tree vlan 1-999 priority 4096
If they support it, I suggest you move to rapid spanning tree. This will give you sub-second failover times instead of up to 40 seconds, but your switches may or may not support it, depending on IOS version (maybe a reason to install more recent software):
spanning-tree mode rapid-pvst