High Availaibility with pinball

[Benoss]

After reading about pinball, I can't see how I can prevent full failure overnight if the server with the master or the scheduler dies.
Is there any way to have multi master? and multi schedulers?

[csliu]

The Pinterest practice about this is to set some alert about these services, if they died, just restart them or rebuild them in another box, as they are stateless services.

It's possible to build some fault tolerant feature, like fail over to secondary master if the primary failed.

[Benoss]

Auto failover would be a must for me. We don't have a 24/7 tech support in my company and we have some very important jobs overnight, we can always recover in the morning but having a second master/scheduler just waiting on a second server in case of hardware failure would solve 99% of HA needed.

[derek-adair]

Fault tolerance is a must for us as well, although there are some very specific implementation details that make me lean towards rolling our own workflow.