Skip to content

Commit 3726e54

Browse files
SkafteNickipre-commit-ci[bot]Borda
authored
Update webdataset integration example with missing ddp params (#21243)
* fix example * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Jirka Borovec <[email protected]>
1 parent 4e262bc commit 3726e54

File tree

1 file changed

+6
-1
lines changed

1 file changed

+6
-1
lines changed

docs/source-pytorch/data/alternatives.rst

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,12 @@ The webdataset library contains a small wrapper (``WebLoader``) that adds a flui
9999
import lightning as L
100100
import webdataset as wds
101101
102-
dataset = wds.WebDataset(urls)
102+
dataset = wds.WebDataset(
103+
urls,
104+
# needed for multi-gpu or multi-node training
105+
workersplitter=wds.shardlists.split_by_worker,
106+
nodesplitter=wds.shardlists.split_by_node,
107+
)
103108
train_dataloader = wds.WebLoader(dataset)
104109
105110
model = ...

0 commit comments

Comments
 (0)