Skip to content

Fix StopIteration when the JSON data files are empty - #8444

Open
LeSingh1 wants to merge 1 commit into
huggingface:mainfrom
LeSingh1:fix-json-empty-file-stopiteration
Open

Fix StopIteration when the JSON data files are empty#8444
LeSingh1 wants to merge 1 commit into
huggingface:mainfrom
LeSingh1:fix-json-empty-file-stopiteration

Conversation

@LeSingh1

@LeSingh1 LeSingh1 commented Aug 9, 2026

Copy link
Copy Markdown

Loading a JSON dataset whose data file is empty fails with a bare StopIteration:

open("empty.jsonl", "w").close()
load_dataset("json", data_files="empty.jsonl", split="train", streaming=True)
# StopIteration
File "src/datasets/packaged_modules/json/json.py", line 101, in _split_generators
StopIteration

_split_generators() infers the features from the first table of the first split:

if self.info.features is None:
    try:
        pa_table = next(iter(self._generate_tables(**splits[0].gen_kwargs, allow_full_read=False)))[1]
        ...
    except FullReadDisallowed:
        pass

Only FullReadDisallowed is caught. When the data files hold no data the generator yields nothing, so next() raises StopIteration and it propagates out of load_dataset — an exception with no message that says nothing about the file being empty. It is also the kind of exception that gets swallowed or converted to RuntimeError (PEP 479) depending on where the call ends up.

The other loaders don't do this — text streams an empty file as zero examples.

Fix

Pass a default to next() and only set the features when a table was actually produced. Behaviour afterwards:

before after
streaming=True StopIteration yields no example, like the text loader
streaming=False StopIteration the usual "specify features" DatasetGenerationError, like csv / text

A file containing only newlines is unaffected — it already produces a zero-row table and infers features={}, so it never hit this path.

Tests

  • test_json_split_generators_with_empty_file_split_generators() leaves info.features as None and generates no table.
  • test_json_load_dataset_streaming_empty_file — end to end, streaming an empty file yields [].

Both fail on main with StopIteration.

_split_generators() infers the features from the first table of the first
split with next(iter(...)), catching only FullReadDisallowed. When the
data files contain no data at all the generator is empty, so next() raises
StopIteration, which escapes load_dataset() as a bare StopIteration with
no message.

Use the default argument of next() and only set the features when a table
was produced. Streaming an empty JSON file now yields no example, like the
text loader already does, and loading it non-streaming reports the usual
'you need to specify features' error instead of StopIteration.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant