Skip to content

Reject features with different columns in IterableDataset.cast - #8453

Open
LeSingh1 wants to merge 1 commit into
huggingface:mainfrom
LeSingh1:fix-iterable-cast-column-mismatch
Open

Reject features with different columns in IterableDataset.cast#8453
LeSingh1 wants to merge 1 commit into
huggingface:mainfrom
LeSingh1:fix-iterable-cast-column-mismatch

Conversation

@LeSingh1

@LeSingh1 LeSingh1 commented Aug 9, 2026

Copy link
Copy Markdown

IterableDataset.cast() assigns the given features directly:

features = _fix_for_backward_compatible_features(features)
info = self._info.copy()
info.features = features

There is no check that the columns match the dataset's, which Dataset.cast() does perform. The worst case is an extra column: it is accepted silently and then fabricated in the data, because the declared features are applied to the examples on the fly.

ds = Dataset.from_dict({"x": [1, 2], "y": ["a", "b"]}).to_iterable_dataset()

new_features = Features({"x": Value("int64"), "y": Value("string"), "z": Value("int64")})
list(ds.cast(new_features))
# [{'x': 1, 'y': 'a', 'z': None}, {'x': 2, 'y': 'b', 'z': None}]

No error at any point — the dataset just grows a column that was never in it. The map-style call raises:

ValueError: The columns in features (['x', 'y', 'z']) must be identical as the columns in the dataset: ['x', 'y']

Missing and renamed columns are accepted too, and fail later with a CastError: Couldn't cast from deep in the Arrow cast, which doesn't say the column set is wrong.

Fix

Apply the same check and the same message as Dataset.cast() when the features are known:

if info.features is not None and sorted(features) != sorted(info.features):
    raise ValueError(...)

sorted() is used on both sides, exactly like the map-style version, so reordering the columns keeps working. Datasets whose features aren't resolved yet (info.features is None) are untouched.

Test

test_iterable_dataset_cast_with_different_columns, parametrized over a missing column, an extra column and a renamed column. All three fail on main.

cast() replaced info.features with whatever it was given, without
checking that the columns match the ones of the dataset. Passing features
with an extra column silently invented it: the column showed up in
.features and was yielded as None in every example. Passing features that
miss or rename a column was also accepted, and only failed later when
iterating, with 'CastError: Couldn't cast'.

Dataset.cast() checks this up front, so add the same check and message
here when the features are known. Datasets whose features are unknown are
unaffected, and reordering the columns is still allowed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant