We are having issues with one of our customers. The network is designed as follows:
Two Cisco 3750 Layer 3 switches; these switches are configured as stackable - Primary & Member (provides redundancy in case of failure)
The Cisco switches route traffic between VLANs and only send traffic to firewall when internet request is made.
3 network segments (VLANs) are connected to these switches:
Network 10 has one cable connected to primary Cisco switch and a second cable to member Cisco switch
Network 20 has one cable connected to primary Cisco switch and a second cable to member Cisco switch
Network 30 has one cable connected to primary Cisco switch and a second cable to member Cisco switch
Each VLAN has 2 x stackable Dell Powerconnect 5548 switches
Spanning tree protocol is enabled on the Cisco switches (mode Rapid-pvst)
Spanning tree protocol is enabled on all Dell switches (mode rapid STP)
This configuration has been active for 14 months without issue.
However, during the past week there have been two instances where the 10 and 20 networks connected to the primary and member Cisco went into block mode enabled by STP protocol. In addition network 10 and 20 lost communication to all networks , both private (internal) and public (internet).
Network 30 was still functional (could not reach network 10 or 20), but was able to connect to internet.
We originally thought that this problem was due to a device which bridged network 10 and network 20. However, we were unable to locate such a device. This happened during production hours.
We temporary fixed the problem by disabling the member Cisco switch.
6 hours later (when business was closed) we activated the member Cisco switch with the hope that we could reproduce the problem and this time spanning tree did not block any ports, nor was any network communication lost. Logs on both the Cisco switch and Dell switches do not show any irregularities.
1. I thought Spanning tree was supposed to avoid a situation which happened as described above--> the shutdown of two VLANs. Is this true?
2. What steps can one do to locate the problem point which caused STP to block ports ?
Any advice on how to troubleshoot this problem would be appreciated. If there is a software program or tool that can help, please recommend one.
Thanks in advance.