troubleshooting Question

nginx error:- upstream timed out (110: Connection timed out) while reading response header from upstream, client:

Avatar of coder
coderFlag for Australia asked on
LinuxLinux OS DevLinux DistributionsDockerUbuntu
5 Comments2 Solutions16 ViewsLast Modified:
Hi Experts,

I get this error from Nginx error logs.  There are two Nginx running.
a) Nginx running in the Ubuntu host machine
b) Nginx inside the docker container (which runs the application inside)

the error logs from outside Nginx are as follows:

2021/07/12 00:00:20 [error] 26211#26211: *698772 upstream timed out (110: Connection timed out) while reading response header from upstream, client: *.*.*.*, server: territ******.au, request: "GET /a****?group=0&handle=10070%2F****&page=0 HTTP/1.1", upstream: "http://127.0.0.1:808/***?group=0&handle=10070%2F6*****&page=0", host: "territ****.au"
2021/07/12 00:36:34 [crit] 26211#26211: *705032 SSL_do_handshake() failed (SSL: error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol) while SSL handshaking, client: 119.*.*.*, server: 0.0.0.0:443
2021/07/12 00:36:35 [crit] 26211#26211: *705034 SSL_do_handshake() failed (SSL: error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol) while SSL handshaking, client: 139.*.*.*, server: 0.0.0.0:443
2021/07/12 01:07:34 [error] 26211#26211: *710644 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 52.*.*.*, server: territ****.au, request: "GET /api/****/ts8*****?query= HTTP/1.1", upstream: "http://127.0.0.1:8008/****/ts8851df53-56*****1?query=", host: "territ****u"

the logs from the inside Nginx (inside docker container) is as follows:-

2021/07/12 01:04:45 [error] 39#39: *789701 open() "/home/ntdl/static/favicon.ico" failed (2: No such file or directory), client: 172.17.0.1, server: , request: "GET /static/favicon.ico HTTP/1.0", host: "127.0.0.1:8008", referrer: "https://territ**au/bitstream/10070/****/1/****.pdf"
2021/07/12 01:07:34 [error] 39#39: *790048 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 172.17.0.1, server: , request: "GET /a***/ts******?query= HTTP/1.0", upstream: "uwsgi://unix://tmp/n***k", host: "127.0.0.1:8008"
2021/07/12 01:07:35 [error] 39#39: *790050 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 172.17.0.1, server: , request: "GET /a****?group=0&handle=10070%2F6***&page=0 HTTP/1.0", upstream: "uwsgi://unix://tmp/nt***k", host: "127.0.0.1:8008"  

The command which I run inside the docker container is as follows:-

npm run start-prod
Start-prod in package.json is as follow:-

"start-prod": "concurrently --no-color \"uwsgi --ini /home/ntdl/code/uwsgi.ini\" \"nginx\" \"npm run render-prod\"",  

uwsgi.ini setting inside the docker container is as follows:-

[uwsgi]
chdir=/home/ntdl/code
module=ntdl.wsgi:application
master=True
pidfile=/tmp/ntdl.pid
vacuum=True
max-requests=5000
socket = /tmp/n****k
chmod-socket = 664
uid = www-data
gid = www-data
enable-threads = true
threads = 20
req-logger = file:/home/ntdl/code/req.log
logger = file:/home/ntdl/code/err.log
the err.log and req.log for uwsgi don't have any errors.
nginx settings inside the docker container are as follows:-

root@2f96610fd1ee:/home/ntdl/code# cat /etc/nginx/sites-available/nginx-app.conf
# NTDL nginx application config file

upstream django {
    server unix://tmp/nt***k;
}

server {
    listen 80 default_server;
    charset utf-8;
    client_max_body_size 50M;

    location /media {
        alias /home/ntdl/media;
    }

    location /static {
        alias /home/ntdl/static;
    }

    location / {
        uwsgi_pass django;
        proxy_connect_timeout 159s;
        proxy_send_timeout 600;
        proxy_read_timeout 600;
        include /home/ntdl/code/uwsgi_params;
    }
}

nginx settings in the ubuntu machine (host machine Nginx - outside docker container)

server {
    listen 443;
    ssl on;

    # Tenable Nessus 84502 HSTS required
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

    # Ensure the host name has no www
    server_name territ*****u;
    if ($host != $server_name) {
        #rewrite ^ $scheme://$server_name permanent;
        return 301 $scheme://$server_name$request_uri;
    }

    # The main **** application on port 8008
    location / {
        proxy_pass http://127.0.0.1:8008;
        proxy_set_header X-Forwarded-For $remote_addr;
        #add_header Access-Control-Allow-Origin *;
        add_header Access-Control-Allow-Origin https://territ***au/;
        add_header Access-Control-Allow-Origin https://www.terri****u/;
        add_header Access-Control-Max-Age 3600;
        add_header Access-Control-Expose-Headers Content-Length;
        add_header Access-Control-Allow-Headers Range;
    }


    # Cantaloupe IIIF Server
    location /cantaloupe {
        proxy_redirect off;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host "territ****u";
        proxy_set_header Accept-Encoding "";
        sub_filter_types application/json;
        # How to write the substitution filter rules:
        # sub_filter <internal_url> <external_url>
        sub_filter "http://terri****u/" "https://territ*****u/can****e/";
        sub_filter_once off;
        # This ensures '%2F' in S3 paths is not decoded by Nginx
        rewrite ^ $request_uri; # The original URI
        rewrite ^/cantaloupe(/.*) $1 break; # The / must be in the pattern
        return 400; # Only if the second rewrite fails
        proxy_pass http://172.17.0.1:8080$uri; # With no / in the URI
    }

    # The Harvest API - /harvest/items
    location /harvest/items {
        proxy_redirect off;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host "territ*****u/h****t";
        rewrite /harvest/(.*) /$1 break;
        proxy_set_header Accept-Encoding "";
        sub_filter_types application/json;
        # How to write the substitution filter rules:
        # sub_filter <internal_url> <external_url>
        sub_filter "http://terri*****au/" "https://terri****au/h****t/";
        sub_filter_once off;        
        proxy_pass http://127.0.0.1:10080;
    }
}

Please suggest how to fix the upstream timeout error.
I am not getting any errors in test and dev environment.  I only get from production environment
can anyone suggest how to get this error in dev environment, so that I can find the right reason for error and fix in production
Any help is highly appreciated.
ASKER CERTIFIED SOLUTION
nociSoftware Engineer
Join our community to see this answer!
Unlock 2 Answers and 5 Comments.
Start Free Trial
Learn from the best

Network and collaborate with thousands of CTOs, CISOs, and IT Pros rooting for you and your success.

Andrew Hancock - VMware vExpert
See if this solution works for you by signing up for a 7 day free trial.
Unlock 2 Answers and 5 Comments.
Try for 7 days

”The time we save is the biggest benefit of E-E to our team. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange.

-Mike Kapnisakis, Warner Bros