nginx reverse proxy error

CSRF failure means you are most probably losing a cookie.
You will get a cookie during initial connection, if that cookie gets lost then the session will be disconnected.
The CSRF check is to prevent session hijacking.
(It might also happen when your source IP changes or some other checks fail, of if there are multiple servers that don;t share a database with issued cookies.).
Before logon you just get a new cookie to show afterwards, once logged on the cookie must be there.

Thanks for the info about the cookie. That makes sense because I am seeing the same behavior on other servers after login. Do you know what I need to do to get that cookie to pass through the proxy? Login works fine locally so I know the servers are functioning fine.

Thanks again for the help.

Stuart

Both servers need to have the same session info to check the cookie. So the App needs to support that part.
HTTP is stateless in and of itself. So every request is on it's own. When a server receives the packet (including cookies) the web app can stitch all request together. To prevent session hijacking CSRF cookies can be used, in that case ALL backends need to be able to validate all requests.

Maybe you should look at haproxy as reverse proxy, that proxy can ensure that if you start with server 1, that you session continues to go to server 1... and doesn't swap per  request.

@noci, So I feel I have messed with this long enough with nginx although I am sure its totally possible my time is probably better sent in HAProxy

I have installed HAProxy on an Ubuntu server VM and I'm attempting to configure it.

It seems that most information I find about configuration of HAProxy seems to be with load balancing. And I'm a totally newb so that doesn't really make it easy for me.

I have 4 servers that need to be accessed via port 443 we'll call them servers1-4

Here is my attempt at a working configuration. The top portion is from the default config file from HAProxy

global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    # Default ciphers to use on SSL-enabled listening sockets.
    # For more information, see ciphers(1SSL). This list is from:
    #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES25$
    ssl-default-bind-options no-sslv3

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5000
    timeout client  50000
    timeout server  50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

frontend localhost
    mode http
    use_backend server1 if { hdr(host) -i subdomain1.domain.com }
    use_backend server2 if { hdr(host) -i subdomain2.domain.com }
    use_backend server3 if { hdr(host) -i subdomain3.domain.com}
    use_backend server4 if { hdr(host) -i subdomain4.domain.com }

backend server1
    server server1 192.168.1.100

backend server2
    server server2 192.168.1.101

backend server3
    server server3 192.168.1.102

backend server4
    server server4 192.168.1.103

Select allOpen in new window

If you could help me I would be very grateful.

Thanks,
Stuart

This should be workable...

after you add the TLS / SSL stuff:  

In options:

        ssl-default-bind-options ssl-min-ver TLSv1.0  ssl-max-ver TLSv1.3 no-tls-tickets
        ssl-default-bind-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS:!kRSA:!3DES
        ssl-default-server-options no-sslv3 no-tls-tickets ssl-min-ver TLSv1.0  ssl-max-ver TLSv1.3
        ssl-default-server-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS:!eNULL:!DES:!3DES
        tune.ssl.default-dh-param 2048

Select allOpen in new window

(You may need to upgrade the minimal TLS version to 1.2 in the near future.)

The frontend needs to listen to port 443...

frontend myfront
        bind :443  ssl crt /etc/haproxy/crt/ alpn h2,http/1.1
        #bind :::443 v4v6 ssl crt /etc/haproxy/crt/ alpn h2,http/1.1    if IPv6 is involved....
        log global
        mode http
        option forwardfor

Select allOpen in new window

All certificates needed can be put into a directory named /etc/haproxy/crt/ and maybe remove the "alpn" as that requires some more work to get fingerprints from the certificates.
Then there is the question how to access the actual servers....
if those use http your method is fine
if the use https then you may need:

server server1  192.168.1.100:443  ssl verify none

Select allOpen in new window

The " verify none " can be skipped if there is a valid, verifyable certificate on the backend.

You could add this entry to redirect all http calls to https using somthing like:

listen http-to-https
        bind *:80 v4v6
        mode http
        redirect scheme https code 301 #if !{ ssl_fc }

Select allOpen in new window

remove the v4v6 if no IPv6 is used.

Thanks so much for all the hlep just to clearify can I put the above config in any place in haproxy.cfg or does it need to go in specific sections?

It seems that 2 of the above sections are listed clearly but the other two aren't. I'll just repost and see what you think.

And are there any dependancies I need to install? I dont seem to be able to start haproxy via systemctl start haproxy.

global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    # Default ciphers to use on SSL-enabled listening sockets.
    # For more information, see ciphers(1SSL). This list is from:
    #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES12$
    ssl-default-bind-options no-sslv3
    ssl-default-bind-options ssl-min-ver TLSv1.0  ssl-max-ver TLSv1.3 no-tls-tickets
    ssl-default-bind-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1$
    ssl-default-server-options no-sslv3 no-tls-tickets ssl-min-ver TLSv1.0  ssl-max$
    ssl-default-server-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES$
    tune.ssl.default-dh-param 2048

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5000
    timeout client  50000
    timeout server  50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

frontend myfront
    bind :443  ssl crt /etc/haproxy/crt/ alpn h2,http/1.1
    #bind :::443 v4v6 ssl crt /etc/haproxy/crt/ alpn h2,http/1.1    if IPv6 is invo$
    log global
    mode http
    option forwardfor

frontend localhost
    mode http
    use_backend server1 if { hdr(host) -i subdomain1.domain.com }
    use_backend server14 if { hdr(host) -i subdomain2.domain.com }
    use_backend server3 if { hdr(host) -i subdomain3.domain.com}
    use_backend server4 if { hdr(host) -i subdomain4.domain.com }

backend server1
    server server1 192.168.1.100:443

backend server2
    server server2 192.168.1.101:443

backend server3
    server server3 192.168.1.102:443

backend server4
    server server4 192.168.1.103:443

Select allOpen in new window

In specific I get this.

Job for haproxy.service failed because the control process exited with error code.
See "systemctl status haproxy.service" and "journalctl -xe" for details.

Select allOpen in new window

@haproxy:/etc/haproxy$ systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enab
   Active: failed (Result: exit-code) since Tue 2019-01-08 18:47:44 UTC; 52s ago
     Docs: man:haproxy(1)
           file:/usr/share/doc/haproxy/configuration.txt.gz
  Process: 9007 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exi
 Main PID: 7117 (code=exited, status=143)

Jan 08 18:47:44 haproxy systemd[1]: haproxy.service: Control process exited, code=ex
Jan 08 18:47:44 haproxy systemd[1]: haproxy.service: Failed with result 'exit-code'.
Jan 08 18:47:44 haproxy systemd[1]: Failed to start HAProxy Load Balancer.
Jan 08 18:47:44 haproxy systemd[1]: haproxy.service: Service hold-off time over, sch
Jan 08 18:47:44 haproxy systemd[1]: haproxy.service: Scheduled restart job, restart
Jan 08 18:47:44 haproxy systemd[1]: Stopped HAProxy Load Balancer.
Jan 08 18:47:44 haproxy systemd[1]: haproxy.service: Start request repeated too quic
Jan 08 18:47:44 haproxy systemd[1]: haproxy.service: Failed with result 'exit-code'.
Jan 08 18:47:44 haproxy systemd[1]: Failed to start HAProxy Load Balancer.

Select allOpen in new window

-- Subject: Unit haproxy.service has finished shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit haproxy.service has finished shutting down.
Jan 08 18:50:03 haproxy systemd[1]: Starting HAProxy Load Balancer...
-- Subject: Unit haproxy.service has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit haproxy.service has begun starting up.
Jan 08 18:50:03 haproxy haproxy[9130]: [ALERT] 007/185003 (9130) : parsing [/etc/hap
Jan 08 18:50:03 haproxy haproxy[9130]: [ALERT] 007/185003 (9130) : parsing [/etc/hap
Jan 08 18:50:03 haproxy haproxy[9130]: [ALERT] 007/185003 (9130) : Error(s) found in
Jan 08 18:50:03 haproxy haproxy[9130]: [ALERT] 007/185003 (9130) : Fatal errors foun
Jan 08 18:50:03 haproxy systemd[1]: haproxy.service: Control process exited, code=ex
Jan 08 18:50:03 haproxy systemd[1]: haproxy.service: Failed with result 'exit-code'.
Jan 08 18:50:03 haproxy systemd[1]: Failed to start HAProxy Load Balancer.
-- Subject: Unit haproxy.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit haproxy.service has failed.
--
-- The result is RESULT.
Jan 08 18:50:04 haproxy systemd[1]: haproxy.service: Service hold-off time over, sch
Jan 08 18:50:04 haproxy systemd[1]: haproxy.service: Scheduled restart job, restart
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Automatic restarting of the unit haproxy.service has been scheduled, as the resul
-- the configured Restart= setting for the unit.
Jan 08 18:50:04 haproxy systemd[1]: Stopped HAProxy Load Balancer.
-- Subject: Unit haproxy.service has finished shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit haproxy.service has finished shutting down.
Jan 08 18:50:04 haproxy systemd[1]: haproxy.service: Start request repeated too quic
Jan 08 18:50:04 haproxy systemd[1]: haproxy.service: Failed with result 'exit-code'.
Jan 08 18:50:04 haproxy systemd[1]: Failed to start HAProxy Load Balancer.
-- Subject: Unit haproxy.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit haproxy.service has failed.
--
-- The result is RESULT.

Select allOpen in new window

to connect to an https port (443) the server line should contain ssl

The line for server2 (use_backend)...  server14...

Try to avoid using names for multiple purposes, use meaning full names.

frontend fe_ssl        # the SSL frontend.
 backend be_domain1      # because you disect on domains...
 server  se_server1 ...     # clear this is a server.

Yeah 14 was a typo. I have been renaming everything for privacy reasons. So in the end I will have everything named to make sense.

Do you mean "ssl verify none"

just ssl, the verify none will prevent verifying the ssl certificates on a server (allowing to use selfsigned certificates).

frontend myfront
    bind :443  ssl crt /etc/haproxy/crt/ alpn h2,http/1.1
    #bind :::443 v4v6 ssl crt /etc/haproxy/crt/ alpn h2,http/1.1    if IPv6 is invo$
    log global
    mode http
    option forwardfor

frontend localhost
    mode http
    use_backend lpr1 if { hdr(host) -i lpr1.domain.com }
    use_backend lpr14 if { hdr(host) -i lpr14.domain.com }
    use_backend avigilon if { hdr(host) -i avigilon.domain.com }
    use_backend hassio if { hdr(host) -i hass.domain.com }

backend lpr1
    server lpr1 192.168.1.83:443  ssl

backend lpr14
    server lpr14 192.168.1.114:443  ssl

backend avigilon
    server avigilon 192.168.1.197:443  ssl

backend hassio
    server hassio 192.168.20.8:443  ssl

Select allOpen in new window

Is this correct because haproxy still wont start.

try to use be_lpr1  for all backend references and se_lpr1 for the server line....
So to make the above look better... (also in logging below will show better is a backend lpr1 is in volved or a server lpr1.)

frontend myfront
    bind :443  ssl crt /etc/haproxy/crt/ h2,http/1.1
    #bind :::443 v4v6 ssl crt /etc/haproxy/crt/ h2,http/1.1   ## if IPv6 is involved, without IPv4 you can drop this line.
    log global
    mode http
    option forwardfor
    use_backend be_lpr1 if { hdr(host) -i lpr1.domain.com }
    use_backend be_lpr14 if { hdr(host) -i lpr14.domain.com }
    use_backend be_avigilon if { hdr(host) -i avigilon.domain.com }
    use_backend be_hassio if { hdr(host) -i hass.domain.com }

backend be_lpr1
    server se_lpr1 192.168.1.83:443  ssl

backend be_lpr14
    server se_lpr14 192.168.1.114:443  ssl

backend be_avigilon
    server se_avigilon 192.168.1.197:443  ssl

backend be_hassio
    server se_hassio 192.168.20.8:443  ssl

Select allOpen in new window

The alpn is optional and requires more setup outside of haproxy. (so drop it for now).
please provide the error messages if anything fails those do help explaining what is happening (or rather why it isn;t happening).

Oh ok I understand now that makes sense.

frontend localhost
    mode http
    use_backend be_lpr1 if { hdr(host) -i lpr1.domain.com }
    use_backend be_lpr14 if { hdr(host) -i lpr14.domain.com }
    use_backend be_avigilon if { hdr(host) -i avigilon.domain.com }
    use_backend be_hassio if { hdr(host) -i hass.domain.com }

backend be_lpr1
    server se_lpr1 192.168.1.83:443  ssl

backend be_lpr14
    server se_lpr14 192.168.1.114:443  ssl

backend be_avigilon
    server se_avigilon 192.168.1.197:443  ssl

backend be_hassio
    server se_hassio 192.168.20.8:443  ssl

Select allOpen in new window

I know that may be exactly what you put above but just want to verify its right.

I'm still unable to start it though.

beatified@haproxy:/etc/haproxy$ systemctl start haproxy
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to start 'haproxy.service'.
Authenticating as: Stuart (beatified)
Password:
==== AUTHENTICATION COMPLETE ===
Job for haproxy.service failed because the control process exited with error code.
See "systemctl status haproxy.service" and "journalctl -xe" for details.
beatified@haproxy:/etc/haproxy$ systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enab
   Active: failed (Result: exit-code) since Tue 2019-01-08 20:23:51 UTC; 2min 41s ag
     Docs: man:haproxy(1)
           file:/usr/share/doc/haproxy/configuration.txt.gz
  Process: 1728 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exi

Jan 08 20:23:50 haproxy systemd[1]: haproxy.service: Control process exited, code=ex
Jan 08 20:23:50 haproxy systemd[1]: haproxy.service: Failed with result 'exit-code'.
Jan 08 20:23:50 haproxy systemd[1]: Failed to start HAProxy Load Balancer.
Jan 08 20:23:51 haproxy systemd[1]: haproxy.service: Service hold-off time over, sch
Jan 08 20:23:51 haproxy systemd[1]: haproxy.service: Scheduled restart job, restart
Jan 08 20:23:51 haproxy systemd[1]: Stopped HAProxy Load Balancer.
Jan 08 20:23:51 haproxy systemd[1]: haproxy.service: Start request repeated too quic
Jan 08 20:23:51 haproxy systemd[1]: haproxy.service: Failed with result 'exit-code'.
Jan 08 20:23:51 haproxy systemd[1]: Failed to start HAProxy Load Balancer.

Select allOpen in new window

Oh wait did I put it in the wrong section? I put it in "frontend localhost" you seem to have it in "frontend myfront".

You need a bind at least in a front end...
(localhost may not be the best name choice, myfront neither....
better is fe_firewall1 or likewise...)  (In consolidated logging localhost doesn't add value).

haproxy messages seem to be missing. i only see systemd messages. The earlier log had haproxy messages   haproxy[xxx]

hm..., maybe remove the "h2,"  part from the config as well, not sure if your haproxy already supports it.
(h2 = http/2 support (the official version of the SPDY protocol).   Before 1.9.x it isn't actually supported on the backends yet).