Updating the router kills existing connections #167
Description
Currently when you update to a newer version of jumpstarter, the jumpstarter-router is restarted, killing any existing connections from clients to exporters, which will recover but in some cases jobs will fail.
To avoid this we need to control the rollout strategy for the router updates.
- Ensure that the newer version is started first.
- Provide readyness/liveness indicators from the router, to avoid the cluster sending new connections to a router that has been requested to stop.
- Once a router is asked to stop, it must not stop until the existing connections are ended (probably with a timeout, something reasonable 1-2h?)
- Because the existing router will move into non-ready state the cluster must send connections to the new router.
- When the old router has finished will all active connections, will then end.