Restore the _legacy_no_dict_keys_sorting branch in Pickler._batch_setitems - #8411
Open
ebarkhordar wants to merge 1 commit into
Open
Restore the _legacy_no_dict_keys_sorting branch in Pickler._batch_setitems#8411ebarkhordar wants to merge 1 commit into
ebarkhordar wants to merge 1 commit into
Conversation
…items huggingface#7817 removed the early return that read the flag while fixing the Python 3.14 signature, leaving builder.py's 2.15.0 cache lookup computing a config_id that no longer matches what 2.15.0 wrote.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
Pickler._legacy_no_dict_keys_sortinglost its only reader in #7817. That PR rewrote_batch_setitemsto accept the argument Python 3.14 passes, and in the same hunk dropped the early return:_check_legacy_cache2still patches the flag on to rebuild the 2.15.0config_id, so that lookup now hashes with sorted dict keys and no longer matches the directory 2.15.0 wrote. The flag appears twice insrc/, as a definition and as thatpatch.objectwrite, and nothing reads it.The fix
Restore the branch, keeping the
*args/**kwargspassthrough #7817 added so the 3.14 fix is preserved on both paths.Verification
main(assert [('a', 2), ('b', 1)] == [('b', 1), ('a', 2)]) and pass with this change; the third pins the default order-insensitive behaviour and passes either way.pytest tests/test_fingerprint.py: 24 passed / 9 skipped before, 26 passed / 9 skipped after, in a clean container on Python 3.11.15 withdill0.4.1.Hasher.hash({"train": ["train.csv"], "test": ["test.csv"]})under the patched flag returns711511d8f1d9bc25, which is the valuedatasets==2.15.0actually produces for that object. Onmainit returns the sorted-key hashcfd3b51f0f8e9fd8. 4.4.0 is the first release where that regressed, matchinggit tag --contains f7c8e46ec._check_legacy_cache2.If you would rather retire the 2.14/2.15 compat path than restore it, that is the alternative I raised in the issue and I am happy to send that instead.
Fixes #8410