TypeError in _dill.py _batch_setitems with Python 3.14

[Vinello28]

Describe the bug

When using load_dataset on Python 3.14, a TypeError is raised during the fingerprinting process inside the bundled _dill.py. This happens because Python 3.14 changed the signature of the internal C-level _Pickler._batch_setitems to require 3 arguments (self, items, obj), while datasets/utils/_dill.py only passes 2 arguments.

Steps to reproduce the bug

Run any load_dataset script on Python 3.14:
```python
from datasets import load_dataset
ds = load_dataset('parquet', data_files={'train': 'data.parquet'})

Error output:

TypeError: _Pickler._batch_setitems() missing 1 required positional argument: 'obj'

Expected behavior

That this thing works (?)

Environment info

nope

[Vinello28]

Hey Mates, this works: In src/datasets/utils/_dill.py, starting around line 71, update the _batch_setitems overriding to pass the obj argument downstream.

    def _batch_setitems(self, items, obj=None):
        if self._legacy_no_dict_keys_sorting:
            return super()._batch_setitems(items, obj)
        # Ignore the order of keys in a dict
        try:
            # Faster, but fails for unorderable elements
            from datasets.fingerprint import Hasher

            items = sorted(items, key=lambda x: Hasher.hash(x[0]))
            pass
        except Exception:
            pass
        dill.Pickler._batch_setitems(self, items, obj)

[CNTRL-tek91]

I would like to take this if that's okay. It looks like the bundled _dill.py fork predates the Python 3.14 change to _Pickler._batch_setitems's signature. I can update the call site to match it and open a PR.

[CNTRL-tek91]

Actually sorry, just ignore my comment, it looks like this issue was already fixed in #7817.

[Vinello28]

no prob to me, i've just checked the fix reference and actually the CI/CD job failed for that commit. Maybe thats the reason it doesn't work yet :(

[ebarkhordar]

This was fixed by #7817 and shipped in 4.4.0. The snippet you posted, with the _legacy_no_dict_keys_sorting early return, is the pre-4.4.0 body of _batch_setitems, so the install you are running predates the fix. Since 4.4.0 the override is:

def _batch_setitems(self, items, *args, **kwargs):
    ...
    return super()._batch_setitems(items, *args, **kwargs)

Measured just now on Python 3.14.6 in a clean container, same call on a one-column local parquet file:

datasets	dill	load_dataset("parquet", data_files=...)
4.3.0	0.4.0	TypeError
4.4.0	0.4.0	3 rows
5.0.1	0.4.1	3 rows
main @ b7cb10b	0.4.1	3 rows

The 4.3.0 traceback ends at:

File "/usr/local/lib/python3.14/pickle.py", line 1058, in save_dict
    self._batch_setitems(obj.items(), obj)
TypeError: Pickler._batch_setitems() takes 2 positional arguments but 3 were given

That is the mirror of the message you quoted (yours is CPython's _Pickler._batch_setitems reached with two arguments, mine is the datasets override reached with three), and both come from the same 3.14 signature change.

On the red CI: the merge commit for #7817 does have 4 failed legs, so that observation is correct, but it does not decide this one. The 4.4.0 wheel, 5.0.1 and current main all complete the call on 3.14.6.

The report left environment info blank, so the first thing to check is what python -c "import datasets; print(datasets.__version__)" prints. If it is below 4.4.0, pip install -U datasets should be enough.

[Vinello28]

This was fixed by #7817 and shipped in 4.4.0. The snippet you posted, with the _legacy_no_dict_keys_sorting early return, is the pre-4.4.0 body of _batch_setitems, so the install you are running predates the fix. Since 4.4.0 the override is:

def _batch_setitems(self, items, *args, **kwargs):
    ...
    return super()._batch_setitems(items, *args, **kwargs)

Measured just now on Python 3.14.6 in a clean container, same call on a one-column local parquet file:

datasets	dill	load_dataset("parquet", data_files=...)
4.3.0	0.4.0	TypeError
4.4.0	0.4.0	3 rows
5.0.1	0.4.1	3 rows
main @ b7cb10b	0.4.1	3 rows

The 4.3.0 traceback ends at:

File "/usr/local/lib/python3.14/pickle.py", line 1058, in save_dict
    self._batch_setitems(obj.items(), obj)
TypeError: Pickler._batch_setitems() takes 2 positional arguments but 3 were given

That is the mirror of the message you quoted (yours is CPython's _Pickler._batch_setitems reached with two arguments, mine is the datasets override reached with three), and both come from the same 3.14 signature change.

On the red CI: the merge commit for #7817 does have 4 failed legs, so that observation is correct, but it does not decide this one. The 4.4.0 wheel, 5.0.1 and current main all complete the call on 3.14.6.

The report left environment info blank, so the first thing to check is what python -c "import datasets; print(datasets.__version__)" prints. If it is below 4.4.0, pip install -U datasets should be enough.

Thanks mate!