[MRG] Fix custom reducers registered in dispatch_table handled by a member descriptor

[ogrisel]

This is a tentative fix for #259 based on something suggested by @pierreglaser in #260 (comment).

However it does not work the way I thought it would. I am opening the PR anyway to share my understanding with @pierreglaser (and other people interested in this problem). The interesting part is that using the member descriptor to set the custom dispatch_table on the pickler is actually changing the class and not the instance.

[ogrisel]

@pierreglaser feel free to close this PR if you agree that this is a dead-end. I will open an alternative PR that delays the init to the super class.

[pierreglaser]

I left a few comments, @ogrisel let me know what you think once you read them.

[pierreglaser]

CustomizablePickler.register still tries to modify self.dispatch_table directly, thus

• modifying the cloudpickle.Cloudpickler.dispatch_table class attribute
• not modifying _pickle.Pickler.dispatch_table.__get__(self)
  both because of attribute shadowing.

I suggest we update loky_dt with the reducers dict before calling self._set_dispatch_table.

[ogrisel]

Thanks for the explanations, that makes sense. Alternatively we could also setattr on top of set and that should fix the problem. Let me give it a try.

[pierreglaser]

👍

[ogrisel]

Before merging I would like to test with cpython 3.7 + pickle5 which is not tested by the CI.

[ogrisel]

I get a crash with python 3.7 + pickle5 but this is unrelated to this PR as I also observe this on master:

E               Traceback (most recent call last):
E                 File "/home/ogrisel/code/loky/loky/backend/popen_loky_posix.py", line 197, in <module>
E                   prep_data = pickle.load(from_parent)
E               ValueError: unsupported pickle protocol: 5

[pierreglaser]

I get a crash with python 3.7 + pickle5 but this is unrelated to this PR as I also observe this on master

For the record I'm giving fixing this a shot, but it's not trivial. I'll resume working on it if @ogrisel or others deem it as high priority.