Skip to content

Fix save_to_disk writing no shard for a dataset with no rows - #8385

Open
LeSingh1 wants to merge 1 commit into
huggingface:mainfrom
LeSingh1:fix/save-to-disk-empty-dataset
Open

Fix save_to_disk writing no shard for a dataset with no rows#8385
LeSingh1 wants to merge 1 commit into
huggingface:mainfrom
LeSingh1:fix/save-to-disk-empty-dataset

Conversation

@LeSingh1

@LeSingh1 LeSingh1 commented Aug 4, 2026

Copy link
Copy Markdown

save_to_disk computes the number of shards from the estimated size and then clamps it to the number of rows (added in #7912):

num_shards = min(len(self.data), num_shards)

For a dataset with no rows that gives 0 shards. No .arrow file is written, state.json gets an empty "_data_files", and loading it back fails:

ds = Dataset.from_dict({"col_1": []}, features=Features({"col_1": Value("int64")}))
ds.save_to_disk(path)   # succeeds, writes only state.json + dataset_info.json
load_from_disk(path)    # IndexError: list index out of range

save_to_disk reports success, so the data loss is silent until the load fails later.

It is reachable from an ordinary pipeline rather than only from a hand-built empty dataset:

ds.filter(lambda x: False).flatten_indices().save_to_disk(path)

and the same applies through DatasetDict.save_to_disk when one split filters down to nothing.

Passing num_shards=1 explicitly already worked, so only the automatic path was affected. This clamps the computed value to at least one shard.

Added a regression test asserting the shard file exists and the dataset round-trips; it fails on main with assert (dataset_path / "data-00000-of-00001.arrow").exists() and passes with the change. tests/test_arrow_dataset.py and tests/test_dataset_dict.py pass (469 passed, 26 skipped).

save_to_disk computes the number of shards from the estimated size and then
clamps it to the number of rows, which gives 0 shards for a dataset with no
rows. In that case the directory only gets state.json and dataset_info.json,
with an empty "_data_files" list, and load_from_disk raises
"IndexError: list index out of range" while rebuilding the table.

This is reachable from a normal pipeline, e.g. a filter that keeps nothing
followed by flatten_indices, and passing num_shards=1 explicitly already
worked, so only the automatic path was affected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant