Reject features with different columns in IterableDataset.cast - #8453
Open
LeSingh1 wants to merge 1 commit into
Open
Reject features with different columns in IterableDataset.cast#8453LeSingh1 wants to merge 1 commit into
LeSingh1 wants to merge 1 commit into
Conversation
cast() replaced info.features with whatever it was given, without checking that the columns match the ones of the dataset. Passing features with an extra column silently invented it: the column showed up in .features and was yielded as None in every example. Passing features that miss or rename a column was also accepted, and only failed later when iterating, with 'CastError: Couldn't cast'. Dataset.cast() checks this up front, so add the same check and message here when the features are known. Datasets whose features are unknown are unaffected, and reordering the columns is still allowed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
IterableDataset.cast()assigns the given features directly:There is no check that the columns match the dataset's, which
Dataset.cast()does perform. The worst case is an extra column: it is accepted silently and then fabricated in the data, because the declared features are applied to the examples on the fly.No error at any point — the dataset just grows a column that was never in it. The map-style call raises:
Missing and renamed columns are accepted too, and fail later with a
CastError: Couldn't castfrom deep in the Arrow cast, which doesn't say the column set is wrong.Fix
Apply the same check and the same message as
Dataset.cast()when the features are known:sorted()is used on both sides, exactly like the map-style version, so reordering the columns keeps working. Datasets whose features aren't resolved yet (info.features is None) are untouched.Test
test_iterable_dataset_cast_with_different_columns, parametrized over a missing column, an extra column and a renamed column. All three fail onmain.