Have the nodes ping out their dataloader state before the all-reduce.

[Jackmin801]

• If a node leaves by crashing, we cannot exactly recover its dataloader state.
• This forces us to manually skip shards to avoid duplicates
• The ideal state is that they can resume automatically from a remote dataloader state
• The dataloader state is not that big and this should not cost too much overhead
• We could interleave it with the all-reduce, completing the all-reduce validates the dataloader state as latest