I am designing a network for a new building. There will be a main server room and 5 IDFs. In each IDF I want to have as much redundancy as possible, so in each IDF there will be 2 XG/aggregate switches. The idea is that no matter how many 48-port switches are in a cabinet, EACH one will connect to 2 XG switches (one fiber cable to each). Then, to get from one IDF to the next, each XG switch in turn will have a dedicated fiber connection to the other XG switch in the next cabinet. The IDFs are connected in a star topology, meaning one "central" IDF has 2 fiber connections to each of the other 4 IDFs - one in each corner of the building. So the redundancy here is that every 48 port switch itself is connected to 2 XG switches, and in turn every IDF has 2 paths to the central IDF.
Here is a partial diagram to illustrate what I mean:
The red lines indicate fiber connections.
The blue lines also indicate fiber connections but may be overkill (Within each IDF, a single fiber cable connects the 2 XG switches to each other)
I am using RSTP on all switches, with the default settings except I made the following change: I gave each of the 48-port switches a priority of "32,768" while all of the 16-port switches have a priority of "16,384". My assumption is that STP will choose each XG switch over a 48 port switch when building out its path (but I suspect there is more to it than that)
A few questions:
1. Is "Priority" the same as "Cost"? meaning, the faster switches should have a LOWER number (as I indicated above). Is this correct?
2. I started out with one fiber connection between each IDF. At some point I plugged in all of the second (redundant) connections between IDFs and then I noticed some of my servers went offline. Maybe that's because STP was busy re-calculating the new path(s) and maybe I didn't wait long enough (I waited about 20 minutes). As soon as I unplugged one of the redundant paths the servers come back online. I suspect this is because when it comes to Ethernet/IP networking you should not have more than one path to anything. So if that's the case, how can I program/design a redundant network? (In other words, what additional programming must be done to tell the switches to use one path or the other, don't try to use both? I thought the whole point of STP was to calculate and elect a single path and stick with it until either (a) something goes offline or (b) there is a topology change)
3. Not shown in the diagram is the actual main server room. Should the switches in the server room have an even lower cost than all others? (assuming most traffic is between a client and a server (or router))
4. I know that if I had 2 cables connecting Switch A to Switch B, I could select both ports and create a LAG. The LAG will provide a bigger pipe (not necessary in this scenario), load balancing and also redundancy (and "redundancy" is what I am after here). But in my scenario, I don't have 2 fiber connections going to a single XG switch. Instead, I have one fiber going to XG Switch number 1 and another fiber going to XG Switch number 2. Can the 2 XG switches in a sense form the LAG? (Or is a LAG only allowed between 2 devices).
My concern is that in an attempt to add redundancy to my network, I have created too many paths and loops.