Traffic is sent to a terminated server

[ninjaPixel]

When using the loadBalancing option (with two servers, named 'one' and 'two'), I expected nginx to route all traffic to server 'one' when server 'two' is terminated.
Zero downtime deploys do work for me with this setup, however, if server 'two' is terminated (i.e. I just shut it down from my cloud hosting provider) then clients that were connected to 'two' are not connected to 'one' instead (after doing a browser refresh).

I've tried this by using clients with lots of different IP addresses (by using Tor browser) and my guess is that nginx is routing traffic based on the client's IP address, and it isn't checking the health of the server first.

Based on this snippet from your 1.5 blog post, I would expect traffic to only be sent to healthy servers:

While the app is being restarted on a server, Nginx detects that it is down and will send requests to a different server.

Mup version (mup --version): 1.5.1

Mup config

{
  "servers": {
    "one": {
      "host": "1.2.3.4",
      "username": "root",
      "password": "password"
    },
    "two": {
      "host": "1.2.3.4",
      "username": "root",
      "password": "password"
    }
  },
  "app": {
    "name": "my-app",
    "path": "../app",
    "servers": {
      "one": {},
      "two": {}
    },
    "buildOptions": {
      "serverOnly": true
    },
    "env": {
      "ROOT_URL": "https://subdomain.host.com",
      "MONGO_URL": "mongodb+srv://user:pass@subdomain.host.com/test",
      "VIRTUAL_HOST": "staging02.barber-shop-booking.com",
      "HTTPS_METHOD": "noredirect",
      "LETSENCRYPT_HOST": "staging02.barber-shop-booking.com",
      "LETSENCRYPT_EMAIL": "xxx",
      "PORT": 19254,
      "VIRTUAL_PORT": 3000,
      "HTTP_FORWARDED_COUNT": 1
    },
    "docker": {
      "image": "abernix/meteord:node-8.15.1-base",
      "prepareBundle": true,
      "useBuildKit": true,
      "stopAppDuringPrepareBundle": true,
      "imagePort": 3000
    },
    "enableUploadProgressBar": true,
    "type": "meteor"
  },
  "proxy": {
    "domains": "staging02.barber-shop-booking.com",
    "ssl": {
      "letsEncryptEmail": "xxx",
      "forceSSL": false
    },
    "loadBalancing": true,
    "servers": {
      "one": {}
    }
  }
}

[eugle]

One and two servers in your servers are two different servers, how do you achieve that both servers can automatically deploy SSL certificates?

[eugle]

I now use the manual certificate creation method

ssl: {
			forceSSL: true,
			crt: './chain.crt',
			key: './private.key',
		},

[ninjaPixel]

One and two servers in your servers are two different servers, how do you achieve that both servers can automatically deploy SSL certificates?

The proxy only runs on server "one".

[zodern]

@ninjaPixel could you please explain how you tested this? I had tried to test this scenario, but there must have been a flaw in how I did it.

[ninjaPixel]

1 - I deployed my application to two servers ('one' and 'two' in my config) on my cloud hosting provider (Vultr).
2 - I have a page in my app which displays the name of the server that the client is connected to. I made sure that I had an instance of a browser connected to server 'two'
3 - I then stopped server 'two', using my hosting provider's dashboard (clicked the button called 'stop').
4 - The browser which was connected to server 'two' could no longer load the app (I expected the app to load and that the browser would now be connected to server 'one')
5 - Any browsers that were previously connected to server 'one' continue to work.