cythonized pydantic objects in __main__ cannot be pickled

[marco-neumann-by]

Abstract

The following code snipped fails with cloudpickle but works with stock pickle if pydantic is cythonized (either via a platform-specific wheel or by having cython installed when calling setup.py):

# bug.py
import cloudpickle
import pydantic
import pickle

class Bar(pydantic.BaseModel):
    a: int

pickle.loads(pickle.dumps(Bar(a=1))) # This works well
cloudpickle.loads(cloudpickle.dumps(Bar(a=1))) # This fails with the error below

When using the file via main:

$ python bug.py

The error message is:

_pickle.PicklingError: Can't pickle <cyfunction int_validator at 0x7fc6808f1040>: attribute lookup lambda12 on pydantic.validators failed

Note that the issue does NOT appear when a non-cythonized pydantic version is used.

Also note that the issue does NOT appear when the file is not __main__, for example:

$ python -c "import bug"

Environment

• Linux x64
• Python 3.8.6
• cloudpickle 1.6.0
• pydantic 1.7.3 w/ cython enabled

Technical Background

In contrast to pickle, cloudpickle pickles the actual class when it resides in __main__, see the following note in the README:

Among other things, cloudpickle supports pickling for lambda functions
along with functions and classes defined interactively in the
__main__ module (for instance in a script, a shell or a Jupyter notebook).

I THINK that might be the reason why this happens. What's somewhat weird is that the object in question is pydantic.validators.int_validator which CAN actually be pickled:

from pydantic.validators import int_validator
import cloudpickle
import pickle

# both work:
pickle.dumps(int_validator)
cloudpickle.dumps(int_validator)

References

This was first reported in #403 here.

[ogrisel]

Could you please edit the bug report to include the full traceback?

[ogrisel]

Also is this problem happening with the current master branch of cloudpickle?

[ogrisel]

I believe this was fixed by #409 as I cannot reproduce anymore. We still need to release though.

[lukasmasuch]

I still get the same error using the cloudpickle version from master in Python 3.8.5:

The fix from #409 only seems to target Python version < 3.7.

[kylebarron]

Edited to use cloudpickle from master

This issue should be reopened.

The difference between environments and likely why @ogrisel was unable to reproduce this is because pydantic can be installed with or without Cython support. The Cython version of Pydantic is unsurprisingly significantly faster than the pure-Python version and is also the default install (at least for platforms for which wheels exist).

Here are two examples using virtualenv that should be reproducible, using the same script as @marco-neumann-by defined initially:

# example.py
import cloudpickle
import pydantic
import pickle

class Bar(pydantic.BaseModel):
    a: int

pickle.loads(pickle.dumps(Bar(a=1))) # This works well
cloudpickle.loads(cloudpickle.dumps(Bar(a=1))) # This fails with the error below

Non-cython Pydantic

Note that the --no-binary pydantic tells pip to install without any Cython files.

virtualenv .venv
source ./.venv/bin/activate
pip install git+https://github.com/cloudpipe/cloudpickle pydantic --no-binary pydantic

Here you can tell that there are no cython files:

> ls ./.venv/lib/python3.8/site-packages/pydantic/
__init__.py           datetime_parse.py     json.py               tools.py
__pycache__           decorator.py          main.py               types.py
_hypothesis_plugin.py env_settings.py       mypy.py               typing.py
annotated_types.py    error_wrappers.py     networks.py           utils.py
class_validators.py   errors.py             parse.py              validators.py
color.py              fields.py             py.typed              version.py
dataclasses.py        generics.py           schema.py

And the example passes without issue

> python example.py
> echo $?
0

Cython-based Pydantic

Now we install pydantic without use of --no-binary pydantic.

deactivate
rm -rf .venv
virtualenv .venv
source ./.venv/bin/activate
pip install git+https://github.com/cloudpipe/cloudpickle pydantic

Now you can see that there are built C libraries included with Pydantic:

> ls ./.venv/lib/python3.8/site-packages/pydantic/
__init__.cpython-38-darwin.so           json.cpython-38-darwin.so
__init__.py                             json.py
__pycache__                             main.cpython-38-darwin.so
_hypothesis_plugin.cpython-38-darwin.so main.py
_hypothesis_plugin.py                   mypy.cpython-38-darwin.so
annotated_types.cpython-38-darwin.so    mypy.py
annotated_types.py                      networks.cpython-38-darwin.so
class_validators.cpython-38-darwin.so   networks.py
class_validators.py                     parse.cpython-38-darwin.so
color.cpython-38-darwin.so              parse.py
color.py                                py.typed
dataclasses.cpython-38-darwin.so        schema.cpython-38-darwin.so
dataclasses.py                          schema.py
datetime_parse.cpython-38-darwin.so     tools.cpython-38-darwin.so
datetime_parse.py                       tools.py
decorator.cpython-38-darwin.so          types.cpython-38-darwin.so
decorator.py                            types.py
env_settings.cpython-38-darwin.so       typing.cpython-38-darwin.so
env_settings.py                         typing.py
error_wrappers.cpython-38-darwin.so     utils.cpython-38-darwin.so
error_wrappers.py                       utils.py
errors.cpython-38-darwin.so             validators.cpython-38-darwin.so
errors.py                               validators.py
fields.cpython-38-darwin.so             version.cpython-38-darwin.so
fields.py                               version.py
generics.py

And running our example again, we can see that it fails:

> python example.py
Traceback (most recent call last):
  File "example.py", line 9, in <module>
    cloudpickle.loads(cloudpickle.dumps(Bar(a=1))) # This fails with the error below
  File "/Users/kbarron/tmp/.venv/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py", line 73, in dumps
    cp.dump(obj)
  File "/Users/kbarron/tmp/.venv/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py", line 602, in dump
    return Pickler.dump(self, obj)
_pickle.PicklingError: Can't pickle <cyfunction int_validator at 0x101cf62b0>: attribute lookup lambda12 on pydantic.validators failed

[kylebarron]

Also note that the issue does NOT appear when the file is not __main__, for example:

I can also reproduce this, however:

# example.py
import cloudpickle
import pickle
from models import Bar

pickle.loads(pickle.dumps(Bar(a=1))) # This works well
cloudpickle.loads(cloudpickle.dumps(Bar(a=1))) # This fails with the error below

# models.py
import pydantic

class Bar(pydantic.BaseModel):
    a: int

This works fine, so a quick workaround is to always define Pydantic models in a separate file.

[ericman93]

I'm still having this issue in cloudpickle 2.0.0
it is only working with non-cython Pydantic
And my Pydantic models declared in a separated file

[crclark]

@ogrisel I am also still seeing this issue in 2.0.0. The workaround in #408 (comment) works for me, but I believe this issue should be reopened.

[rjurney]

I have this issue with pydantic and pyspark.

../../opt/anaconda3/envs/graphlet/lib/python3.10/site-packages/pyspark/sql/pandas/map_ops.py:91: in mapInPandas
    udf_column = udf(*[self[col] for col in self.columns])
../../opt/anaconda3/envs/graphlet/lib/python3.10/site-packages/pyspark/sql/udf.py:276: in wrapper
    return self(*args)
../../opt/anaconda3/envs/graphlet/lib/python3.10/site-packages/pyspark/sql/udf.py:249: in __call__
    judf = self._judf
../../opt/anaconda3/envs/graphlet/lib/python3.10/site-packages/pyspark/sql/udf.py:215: in _judf
    self._judf_placeholder = self._create_judf(self.func)
../../opt/anaconda3/envs/graphlet/lib/python3.10/site-packages/pyspark/sql/udf.py:224: in _create_judf
    wrapped_func = _wrap_function(sc, func, self.returnType)
../../opt/anaconda3/envs/graphlet/lib/python3.10/site-packages/pyspark/sql/udf.py:50: in _wrap_function
    pickled_command, broadcast_vars, env, includes = _prepare_for_python_RDD(sc, command)
../../opt/anaconda3/envs/graphlet/lib/python3.10/site-packages/pyspark/rdd.py:3345: in _prepare_for_python_RDD
    pickled_command = ser.dumps(command)
../../opt/anaconda3/envs/graphlet/lib/python3.10/site-packages/pyspark/serializers.py:458: in dumps
    return cloudpickle.dumps(obj, pickle_protocol)
../../opt/anaconda3/envs/graphlet/lib/python3.10/site-packages/pyspark/cloudpickle/cloudpickle_fast.py:73: in dumps
    cp.dump(obj)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pyspark.cloudpickle.cloudpickle_fast.CloudPickler object at 0x7ff5f0410700>
obj = (<function test_graphlet_etl.<locals>.horror_to_movie at 0x7ff5d0e81480>, StructType([StructField('entity_id', StringT...ld('length', LongType(), False), StructField('gross', LongType(), False), StructField('rating', StringType(), False)]))

    def dump(self, obj):
        try:
>           return Pickler.dump(self, obj)
E           _pickle.PicklingError: Can't pickle <cyfunction str_validator at 0x7ff5b0461220>: it's not the same object as pydantic.validators.str_validator

../../opt/anaconda3/envs/graphlet/lib/python3.10/site-packages/pyspark/cloudpickle/cloudpickle_fast.py:602: PicklingError

[brettc]

I've just been bitten by this. @ogrisel, can we reopen this issue? The workaround is not an option if you are defining your objects inside a jupyter notebook.

[simon-mo]

@brettc as a workaround, you can define custom serializers to pack and unpack pydantic objects. This might help your use case.

https://github.com/ray-project/ray/blob/eed90495cedad0dc2fb6ea6d430df61e4eac24f4/python/ray/util/serialization_addons.py#L10-L35