torchelastic Rendezvous Backend

[d4l3k]

We want to be able to leverage torchft's fast quorum implementation for Lighthouse in order to do faster dynamic rendezvous for torchelastic.

Torchelastic has an entrypoints based mechanism for registering new backends at https://github.com/pytorch/pytorch/blob/main/torch/distributed/elastic/rendezvous/registry.py#L63-L64

Key features we want to support:

• flexible lighthouse config: external lighthouse support + automatically starting lighthouse similar to c10d's TCPStore using the address
• scale up / scale down operations
• hot spares for fast restarts

References:

• https://packaging.python.org/en/latest/specifications/entry-points/
• https://github.com/pytorch/pytorch/blob/main/torch/distributed/elastic/rendezvous/registry.py#L63-L64
• https://pytorch.org/docs/stable/elastic/rendezvous.html
• c10d rendezvous https://github.com/pytorch/pytorch/blob/main/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py#L214

[MattKotzbauer]

I'd like to try and pick this up - for an initial commit I'll make a
LighthouseRendezvousBackend with flexible config (can either connect to a user-supplied Lighthouse endpoint or, when the rendezvous address ends in :0, automatically launch a local Lighthouse process and advertise bound port to workers). Will post a commit by end of week 🤙.

Warmly,
Matt

[d4l3k]

@MattKotzbauer sounds good -- @tushar00jain is also a good POC.

Are you in the PyTorch slack? Often easier to collaborate via chat -- can invite you via email if you're not