Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

I am running docker-nginx on ECS server. My nginx service is suddenly stopped because the proxy_pass of one of the servers got unreachable. The error is as follows:

[emerg] 1#1: host not found in upstream "dev-example.io" in /etc/nginx/conf.d/default.conf:988
My config file is as below:
 server {
       listen      80;
       server_name     test.com;
       location / {
          proxy_pass         http://dev-exapmle.io:5016/;
          proxy_redirect     off;
          ##proxy_set_header   Host             $host;
          proxy_set_header   X-Real-IP        $remote_addr;
          proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
          client_max_body_size       10m;
          client_body_buffer_size    128k;
          proxy_connect_timeout      90;
          proxy_send_timeout         90;
          proxy_read_timeout         90;
          proxy_buffer_size          4k;
          proxy_buffers              4 32k;
          proxy_busy_buffers_size    64k;
          proxy_temp_file_write_size 64k;
server {
   listen       80 default_server;
   server_name  localhost;
   #charset koi8-r;
   #access_log  /var/log/nginx/log/host.access.log  main;
   location / {
      root   /usr/share/nginx/html;
      index  index.html index.htm;
   #error_page  404              /404.html;
   # redirect server error pages to the static page /50x.html
   error_page   500 502 503 504  /50x.html;
   location = /50x.html {
      root   /usr/share/nginx/html;
I have many servers in the config file, even if one server was down, I need to have running nginx. Is there any way to fix it?
Any suggestion to fix this issue would be appreciated.
                typically this can be for two reasons: either the service in dev-exapmle.io:5016 is down or not responding to requests; or either you have some selinux configuration which is preventing nginx from proxying the requests. I'd look into the logs for for the app serving dev-example.io. If you don't see anything coming in, it's probably a selinux issue, in which case you may want to have a look at this answer: stackoverflow.com/questions/27435655/…
– Jose Haro Peralta
                Mar 10, 2017 at 14:37
                @JoséAntonioHaroPeralta thanks for the comment. yes that service, dev-exapmle.io:5016, is down. But I have many other services in my nginx config file that works but because of this service, the nginx stopped. Is there any way to fix it?
– Matrix
                Mar 10, 2017 at 14:39
                include a resolver directive (pointing for example to google, 8.8.8.8). Nginx will resolve to that if it can't reach your site, as indicated in answers to the following question: stackoverflow.com/questions/32845674/…
– Jose Haro Peralta
                Mar 10, 2017 at 14:48
Just adding a resolver did not resolve the issue in my case. But I was able to work around it by using a variable for the host.
Also, I guess it makes more sense to use Docker's DNS at 127.0.0.11 (this is a fixed IP).
Example:
server {
  listen 80;
  server_name test.com;
  location / {
    resolver 127.0.0.11;
    set $example dev-example.io:5016;
    proxy_pass http://$example;
I found the variable workaround on this page.
                Works well. In the case of using Docker: 1) echo "resolver 127.0.0.11 valid=60s;" > ./conf.d/resolvers.conf 2) docker-compose.yml: volumes: - ./conf.d/resolvers.conf:/etc/nginx/conf.d/resolvers.conf
– Igor Leonovich
                Aug 1, 2021 at 23:57
Include to prevent Nginx from crashing if your site is down, include a resolver directive, as follows:
 server {
       listen                80;
       server_name           test.com;
       location / {
          resolver           8.8.8.8;
          proxy_pass         http://dev-exapmle.io:5016/;
          proxy_redirect     off;
WARNING! Using a public DNS create a security risk in your backend since your DNS requests can be spoofed. If this is an issue, you should point the resolver to a secure DNS server.
                Thanks. what if I use resolver 127.0.0.1  ? and if the server gets reachable again, would it routes the correct path?
– Matrix
                Mar 10, 2017 at 15:42
                Hi, presumably this just adds another DNS resolution endpoint. In my case, the DNS address had been deleted and would not be resolvable anywhere. Is there not a way to have nginx simply treat the page as down rather than crashing the entire instance?
– duct_tape_coder
                Sep 3, 2020 at 18:19
This usually means that the dns name you provided as upstream server cannot be resolved. To test it, log on nginx server and try pinging upstream server provided and see if the name resolution completes correctly, If its a docker container try docker exec -it to get a shell, then try pinging the upstream to test the name resolution. If the contianer is stopped try  to use IP address instead of dns name in your server block.
proxy_pass         http://<IP ADDRESS>:5016/;
You can also use the resolver directive if you want to use different dns server for this location than the host system:
resolver 8.8.8.8;
                Bring up a default config nginx container, then try pinging the dns to see if the resolution happens correctly.
– Farhad Farahi
                Mar 10, 2017 at 14:54
                Thanks. Then I use resolver instead. but if I use resolver 127.0.0.1; would it work fine as well? What will happen if the DNS gets to work later? Also doesn't it need to have valid=30s; ?
– Matrix
                Mar 11, 2017 at 8:57
                resolver 127.0.0.1 means you are running a dns server on localhost, are you?  when the dns server is fixed remove the resolver line. valid 30s means do not requery for 30s use the existing value.
– Farhad Farahi
                Mar 11, 2017 at 9:46
                Isn't there any way to do it automatically? Use resolver when it's down, use the DNS when it's up again. It could be done automatically only using a script?
– Matrix
                Mar 11, 2017 at 10:39
When using nginx plus, you can get around this as well by adding a zone to your upstream with resolve. When use this test in your proxypass. When the server some-server starts resolving, it will starting pass traffic to it.
Make sure to as stated above, put a resolver in other parts of your config. For docker, I use
resolver 127.0.0.11 valid=1s;
upstream test {
        zone test-zone 64k;
        server some-server:1234 resolve;
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.