Fix StopIteration when the JSON data files are empty - #8444
Open
LeSingh1 wants to merge 1 commit into
Open
Conversation
_split_generators() infers the features from the first table of the first split with next(iter(...)), catching only FullReadDisallowed. When the data files contain no data at all the generator is empty, so next() raises StopIteration, which escapes load_dataset() as a bare StopIteration with no message. Use the default argument of next() and only set the features when a table was produced. Streaming an empty JSON file now yields no example, like the text loader already does, and loading it non-streaming reports the usual 'you need to specify features' error instead of StopIteration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Loading a JSON dataset whose data file is empty fails with a bare
StopIteration:_split_generators()infers the features from the first table of the first split:Only
FullReadDisallowedis caught. When the data files hold no data the generator yields nothing, sonext()raisesStopIterationand it propagates out ofload_dataset— an exception with no message that says nothing about the file being empty. It is also the kind of exception that gets swallowed or converted toRuntimeError(PEP 479) depending on where the call ends up.The other loaders don't do this —
textstreams an empty file as zero examples.Fix
Pass a default to
next()and only set the features when a table was actually produced. Behaviour afterwards:streaming=TrueStopIterationtextloaderstreaming=FalseStopIterationfeatures"DatasetGenerationError, likecsv/textA file containing only newlines is unaffected — it already produces a zero-row table and infers
features={}, so it never hit this path.Tests
test_json_split_generators_with_empty_file—_split_generators()leavesinfo.featuresasNoneand generates no table.test_json_load_dataset_streaming_empty_file— end to end, streaming an empty file yields[].Both fail on
mainwithStopIteration.