Let a dataset with an Audio column be flattened - #8452
Open
LeSingh1 wants to merge 1 commit into
Open
Conversation
Dataset.flatten() fails on any dataset that holds a decodable Audio column:
ds = Dataset.from_dict({"audio": [None], "text": ["a"]},
features=Features({"audio": Audio(), "text": Value("string")}))
ds.flatten()
ValueError: Cannot flatten a decoded Audio feature.
The same dataset with an Image, Video or Pdf column flattens fine: those three
return the feature itself when it is decodable, so Features.flatten() leaves the
media column alone and only expands the other columns. Audio raised instead, so
a single audio column made the whole dataset impossible to flatten, even when
the user only wanted to expand an unrelated struct column.
Return the feature itself like the other media features do. Audio(decode=False)
still flattens to bytes/path exactly as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dataset.flatten()fails on any dataset that holds a decodableAudiocolumn:The same dataset with an
Image,VideoorPdfcolumn flattens without complaint:Features.flatten()Dataset.flatten()Image(){'audio': Image(...), 'text': ...}['audio', 'text']Video()['audio', 'text']Pdf()['audio', 'text']Audio()So a single audio column makes the whole dataset impossible to flatten — even when the user only wants to expand an unrelated struct column, and even though flattening does not decode anything.
Cause
Image.flatten,Video.flattenandPdf.flattenall share the same docstring and shape:Audio.flattenis the odd one out:Features.flattencallssubfeature.flatten()on every feature that has the method, so the raise propagates out ofFeatures.flatten()andDataset.flatten().Fix
Return the feature itself when it is decodable, exactly like the other three media features.
Audio(decode=False)still flattens to{"bytes": Value("binary"), "path": Value("string")}as before.No test asserted the old error —
grep -rn "Cannot flatten" src/ tests/matched only the line being removed — so nothing existing had to be adjusted.Tests
tests/features/test_audio.py::test_audio_feature_flattenchecksAudio().flatten(),Audio(decode=False).flatten(),Features.flatten()andDataset.flatten(). It fails onmainwith theValueErrorabove, and needs no audio backend so it runs withouttorchcodec.tests/features/,tests/test_formatting.py,tests/test_table.py,tests/test_arrow_dataset.pyandtests/test_dataset_dict.py: 1134 passed, 0 failures.