OEGrasshopper
asked on
nginx instantly returns "took too long to respond HTTP 504 error"
I'm using aws elasticbeanstalk with an nginx proxy server to my application. I've got it setup on port 5000. I am getting a "took too long to respond HTTP ERROR 504", however, it occurs instantly.
Locally on the server, I can curl http://localhost:3000/login and it returns the html of the login page.
Here are my nginx.conf
# Elastic Beanstalk Nginx Configuration File
#conf.d/myconf.conf
conf
Finally here are my nginx error log when I try to access it from the web
Locally on the server, I can curl http://localhost:3000/login and it returns the html of the login page.
Here are my nginx.conf
# Elastic Beanstalk Nginx Configuration File
user nginx;
error_log /var/log/nginx/error.log debug;
pid /var/run/nginx.pid;
worker_processes auto;
worker_rlimit_nofile 200000;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
include conf.d/*.conf;
map $http_upgrade $connection_upgrade {
default "upgrade";
}
server {
listen 80 default_server;
access_log /var/log/nginx/access.log main;
client_header_timeout 60;
client_body_timeout 60;
keepalive_timeout 60;
gzip off;
gzip_comp_level 4;
gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript;
# Include the Elastic Beanstalk generated locations
include conf.d/elasticbeanstalk/*.conf;
}
}
#conf.d/myconf.conf
# Accept connections via the load balancer
server {
listen 80 proxy_protocol;
server_name _; #Default virtual host
}
server {
listen 8080 proxy_protocol;
client_header_timeout 60;
client_body_timeout 60;
keepalive_timeout 90;
client_max_body_size 25m;
client_header_buffer_size 64k;
large_client_header_buffers 4 64k;
# gzip on;
# gzip_comp_level 5;
# gzip_http_version 1.1;
# gzip_min_length 1000;
# gzip_proxied any;
# gzip_vary on;
# gzip_types application/javascript text/javascript;
location / {
proxy_pass http://127.0.0.1:5000;
proxy_http_version 1.1;
# proxy_set_header Cache-Control no-cache;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;
}
}
conf.d/elasticbeanstalk/*.location / {
proxy_pass http://127.0.0.1:5000;
proxy_http_version 1.1;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto http;
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;
proxy_read_timeout 300;
proxy_connect_timeout 300;
}
Finally here are my nginx error log when I try to access it from the web
2019/10/09 17:01:12 [debug] 25417#0: epoll: fd:6 ev:0001 d:0000000001ED2380
2019/10/09 17:01:12 [debug] 25416#0: epoll: fd:6 ev:0001 d:0000000001ED2380
2019/10/09 17:01:12 [debug] 25415#0: epoll: fd:6 ev:0001 d:0000000001ED2380
2019/10/09 17:01:12 [debug] 25418#0: epoll: fd:6 ev:0001 d:0000000001ED2380
2019/10/09 17:01:12 [debug] 25417#0: accept on 0.0.0.0:80, ready: 0
2019/10/09 17:01:12 [debug] 25416#0: accept on 0.0.0.0:80, ready: 0
2019/10/09 17:01:12 [debug] 25415#0: accept on 0.0.0.0:80, ready: 0
2019/10/09 17:01:12 [debug] 25418#0: accept on 0.0.0.0:80, ready: 0
2019/10/09 17:01:12 [debug] 25417#0: posix_memalign: 0000000001EBBA00:512 @16
2019/10/09 17:01:12 [debug] 25415#0: accept() not ready (11: Resource temporarily unavailable)
2019/10/09 17:01:12 [debug] 25416#0: accept() not ready (11: Resource temporarily unavailable)
2019/10/09 17:01:12 [debug] 25416#0: timer delta: 8960
2019/10/09 17:01:12 [debug] 25418#0: accept() not ready (11: Resource temporarily unavailable)
2019/10/09 17:01:12 [debug] 25417#0: *105866 accept: 172.31.2.137:36474 fd:11
2019/10/09 17:01:12 [debug] 25415#0: timer delta: 8960
2019/10/09 17:01:12 [debug] 25416#0: worker cycle
2019/10/09 17:01:12 [debug] 25418#0: timer delta: 8960
2019/10/09 17:01:12 [debug] 25417#0: *105866 event timer add: 11: 60000:432387461
2019/10/09 17:01:12 [debug] 25415#0: worker cycle
2019/10/09 17:01:12 [debug] 25416#0: epoll timer: -1
2019/10/09 17:01:12 [debug] 25418#0: worker cycle
2019/10/09 17:01:12 [debug] 25417#0: *105866 reusable connection: 1
2019/10/09 17:01:12 [debug] 25415#0: epoll timer: -1
2019/10/09 17:01:12 [debug] 25418#0: epoll timer: 49904
2019/10/09 17:01:12 [debug] 25417#0: *105866 epoll add event: fd:11 op:1 ev:80002001
2019/10/09 17:01:12 [debug] 25417#0: timer delta: 8960
2019/10/09 17:01:12 [debug] 25417#0: worker cycle
2019/10/09 17:01:12 [debug] 25417#0: epoll timer: 51040
2019/10/09 17:01:12 [debug] 25417#0: epoll: fd:11 ev:2001 d:0000000001ED29D9
2019/10/09 17:01:12 [debug] 25417#0: *105866 http wait request handler
2019/10/09 17:01:12 [debug] 25417#0: *105866 malloc: 0000000001EBBC10:1024
2019/10/09 17:01:12 [debug] 25417#0: *105866 recv: eof:1, avail:1
2019/10/09 17:01:12 [debug] 25417#0: *105866 recv: fd:11 47 of 1024
2019/10/09 17:01:12 [debug] 25417#0: *105866 PROXY protocol address: 172.31.2.137 36474
2019/10/09 17:01:12 [debug] 25417#0: *105866 post event 0000000001F0C630
2019/10/09 17:01:12 [debug] 25417#0: timer delta: 0
2019/10/09 17:01:12 [debug] 25417#0: posted event 0000000001F0C630
2019/10/09 17:01:12 [debug] 25417#0: *105866 delete posted event 0000000001F0C630
2019/10/09 17:01:12 [debug] 25417#0: *105866 http wait request handler
2019/10/09 17:01:12 [debug] 25417#0: *105866 recv: eof:1, avail:0
2019/10/09 17:01:12 [debug] 25417#0: *105866 recv: fd:11 0 of 1024
2019/10/09 17:01:12 [info] 25417#0: *105866 client closed connection while waiting for request, client: 172.31.2.137, server: 0.0.0.0:80
2019/10/09 17:01:12 [debug] 25417#0: *105866 close http connection: 11
2019/10/09 17:01:12 [debug] 25417#0: *105866 event timer del: 11: 432387461
2019/10/09 17:01:12 [debug] 25417#0: *105866 reusable connection: 0
2019/10/09 17:01:12 [debug] 25417#0: *105866 free: 0000000001EBBC10
2019/10/09 17:01:12 [debug] 25417#0: *105866 free: 0000000001EBBA00, unused: 124
2019/10/09 17:01:12 [debug] 25417#0: worker cycle
2019/10/09 17:01:12 [debug] 25417#0: epoll timer: 51040
ASKER
Thanks David! When I run netstat -pluten | grep :5000 on the server I get
tcp 0 0 :::5000 :::* LISTEN 497 14439 3430/java
It's not super clear to me what exactly this means. It returned so it seems to be listening to something. Also, when I run
curl http://localhost:5000/login
the html of the login page is returned.
tcp 0 0 :::5000 :::* LISTEN 497 14439 3430/java
It's not super clear to me what exactly this means. It returned so it seems to be listening to something. Also, when I run
curl http://localhost:5000/login
the html of the login page is returned.
This means pid==3420, a Java process, is listening via TCP on port 5000.
You can find more information by doing something like the following to show more information...
In your case you'll use -p 3420.
My guess is this shows you're running an elasticbeanstalk instance on port 5000, as elasticbeanstalk is written in Java.
So you'll move onto connecting to port 5000 using curl or wget or some other code, to test your elasticbeanstalk instance functionality.
You can find more information by doing something like the following to show more information...
lxd: # ps -p 713405 -o args
COMMAND
php-fpm: master process (/etc/php/7.3/fpm/php-fpm.conf)
In your case you'll use -p 3420.
My guess is this shows you're running an elasticbeanstalk instance on port 5000, as elasticbeanstalk is written in Java.
So you'll move onto connecting to port 5000 using curl or wget or some other code, to test your elasticbeanstalk instance functionality.
ASKER
Ok I finally figured out what was wrong.
When deploying to Elastic Beanstalk you can set up options in .ebextentions/securelisten er.config
In mine there was
This created a Load Balancer listener on port 443 but outbound to port 80. However, the strange thing is that when I am using the ELB web console this auto-created listener doesn't show up. So I added my 443->8080 listener but it was never using it, therefore my site never loaded. It was only when I deleted the listeners, logged out, logged back in, did I notice suddenly there was this auto-created listener. Once I deleted it and properly set up the listener it all worked.
So you were close. It wasn't that my server wasn't listening on the correct port. It was that the Load Balancer wasn't forwarding to the port my application was listening on, and the aws elastic beanstalk web console wasn't displaying the listener it was actually using. There were no errors of any kind in the logs and I basically had to find this by randomly removing and re-adding everything.
I also added
InstanceProtocol: TCP
InstancePort: 8080
to my .ebextentions/securelisten er.config file so that the correct listener is created automatically.
Finally, the reason I am using port 8080 to begin with is because my web app uses websockets. The only way to use web sockets with an aws classic load balancer under ssl is to use basic ssl-tcp (rather than https-http) and you need another port because port 80 is reserved for the http protocol.
When deploying to Elastic Beanstalk you can set up options in .ebextentions/securelisten
In mine there was
option_settings:
aws:elb:listener:443:
SSLCertificateId: arn:aws:acm:us-east-1:909057688394:certificate/ad1d91fc-5cc2-4c91-8615-6b8d8c2728b0
ListenerProtocol: SSL
This created a Load Balancer listener on port 443 but outbound to port 80. However, the strange thing is that when I am using the ELB web console this auto-created listener doesn't show up. So I added my 443->8080 listener but it was never using it, therefore my site never loaded. It was only when I deleted the listeners, logged out, logged back in, did I notice suddenly there was this auto-created listener. Once I deleted it and properly set up the listener it all worked.
So you were close. It wasn't that my server wasn't listening on the correct port. It was that the Load Balancer wasn't forwarding to the port my application was listening on, and the aws elastic beanstalk web console wasn't displaying the listener it was actually using. There were no errors of any kind in the logs and I basically had to find this by randomly removing and re-adding everything.
I also added
InstanceProtocol: TCP
InstancePort: 8080
to my .ebextentions/securelisten
Finally, the reason I am using port 8080 to begin with is because my web app uses websockets. The only way to use web sockets with an aws classic load balancer under ssl is to use basic ssl-tcp (rather than https-http) and you need another port because port 80 is reserved for the http protocol.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
In your case elasticbeanstalk is the process which can't be accessed.
First test access to elasticbeanstalk via the IP + port you expect, looks like localhost:5000 is the port.
Make sure the process is actually listening...
Open in new window
Then ssh into this machine + interact with elasticbeanstalk through curl or some other tool to verify your elasticbeanstalk instance is working.
Likely once you get elasticbeanstalk running correctly, NGINX will connect.