Multi-GPU Training Support

[evandrosks]

Hi,

Is there support for multi-gpu single-node or multi-node training? If so, an example in the example page would be helpful.

I'm able to follow the image dataloading example for multi-gpu, but it's not obvious how to incorporate spdl for distributed training.

Thanks

[mthrok]

Hi @evandrosks

Thanks for trying out SPDL.

Data loading in distributed training is splitting the dataset into non-overlapping sets and let each rank iterate one.
In SPDL, this can be achieved by having the source iterable incorporate the rank and the world size.

A simple way is to slice the source list with dataset[rank::world_size] (assuming that the size of the dataset is divisible by world_size.).
This is basically what DistributedSampler does.
So you can create a source object using sampler.

I will try to write an example.

[ricksun2023]

@mthrok
Could you post an example for data loading in multi-node & multi-gpu training?
Thank you.

[mthrok]

@ricksun2023 I plan to do this, but I am in a sort of sabbatical so give me some time.

Meanwhile, we open sourced sampler implementations we use.

https://facebookresearch.github.io/spdl/main/generated/spdl.source.DistributedRandomSampler.html#spdl.source.DistributedRandomSampler

https://facebookresearch.github.io/spdl/main/generated/spdl.source.DistributedDeterministicSampler.html#spdl.source.DistributedDeterministicSampler