Link to home
Create AccountLog in
Avatar of SAGE Dining
SAGE Dining

asked on

pfsense redundant gateway NIC to redundant switches to redundant server NIC

I have a LAGG setup, in pfSense, from two NIC ports that feed two separate managed switches (DGS-1210-24) that each feed into the CentOS server that ultimately bonds the connections from both switches.

hardware/software:
- pfSense Gateway with 6 NICs -- 2 configured with link aggregation to appear as one interface called LAGG0.
- 2 x DLINK 1210-24 managed switches.
- Server running CentOS 6.4 with 2 1Gbit NICs in a bond.

I cannot, for the life of me, figure out the proper LAGG/LACP setup and centos NIC bonding modes

Here is the setup I have in a stupid picture:

            Gateway1
           NIC1    NIC2
              \        /
               LAGG0                ---Mode?
              /        \
   DLINK1p24   DLINK2p24     ---Mode?
              \        /
               BOND0                ---Mode?
              /        \
           NIC1     NIC2
              Server1


I tried:
LAGG0 - LACP(802.3ad)
DLINK 1 and DLINK2 - LACP(802.3ad) - Port Trunking sw1p24 <--> sw2p24
BOND0 - Mode 4 - (802.3ad)

Does not work. I pull the plug on the first switch and all traffic halts and then starts doing weird stuff like 20000ms response time and so forth. Any ideas?!?!
Avatar of skullnobrains
skullnobrains

are you trying to setup a lag that jumps over the switches ?

- if yes beware that this may or may not work. in this case you should not configure any trunking in the switch, and the switch do not have to be connected

- if no, you should try laggs one after the other (i assume your dlinks are stacked) so we know what to debug, and you should set 2 different sets of lags one on each "side" of the switches.

---

as a side note, i do not know what your requirements are, but this looks a little complicated as a setup. you can achieve reasonable failover for a hosted service in a number of much simpler ways including giving 2 addresses to your linux box and load-balance among them in pfsense and deal with outgoing connections using multiple default gateways (possibly with monitoring) on the linux box
Avatar of SAGE Dining

ASKER

are you trying to setup a lag that jumps over the switches ?

I don't know what you mean by jump over. I'll assume you mean feed traffic through the switches?

- if yes beware that this may or may not work. in this case you should not configure any trunking in the switch, and the switch do not have to be connected

OK I had a feeling that no trunking in the switches could be an option. I can disabled the ports to disable the trunks for now. I will try this.

- if no, you should try laggs one after the other (i assume your dlinks are stacked) so we know what to debug, and you should set 2 different sets of lags one on each "side" of the switches.

I am not sure if they are stacked. Yes, I set them up, but no I do not know much about the usage. Documentation does not mention much about stacking. I would assume by stacking I would see switch 1 and switch 2 in some form of an interface, thus allowing me to create trunks that span across the switches.



as a side note, i do not know what your requirements are, but this looks a little complicated as a setup.
Mostly just uptime. Cannot have a single switch fail and cause everything to go down. Devices such as UPS webUI, temp monitor, etc. will go down but the servers should never lose connection if using bonded interfaces. I did leave out a lot of factors for the sake of posting. We actually have two gateways, two switches and 5 servers that sit behind this, for now. We also have two dedicated power circuits to the units so the idea is that we can pull the power on one circuit (or it dies) and the stack will work 100%. Given this, throughput isn't really on my mind here, just uptime. The 2 web servers are load balanced using a floating hostname and the 3 mysql servers are clustered in such a way to handle load balancing without external config. After some time our ESXi servers will be moved over to this setup, furthering the possibility of incorrect setup, etc. Just want to make this a smooth process.
ASKER CERTIFIED SOLUTION
Avatar of skullnobrains
skullnobrains

Link to home
membership
Create a free account to see this answer
Signing up is free and takes 30 seconds. No credit card required.
See answer
feel free to post back if you either have a working solution as i'm interested in pfsense ( i usually do the same king of things in freebsd or openbsd directly ) or if you need help setting up something or thinking about something workable. it will only be trivial if he switches are stacked