Skip to content

Multiprocessing issue 'Can't pickle <function ....' #193

Open
@several27

Description

@several27

Hey,
the following code is not working in ptpython on python version 3.

from multiprocessing.pool import Pool

def do(x):
    return x 

pool = Pool(processes=16,  maxtasksperchild=100)
done_x = []
for x in pool.imap_unordered(do, [1, 2, 3, 4, 5]):
    done_x.append(x)

It produces the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/pool.py", line 695, in next
    raise value
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/pool.py", line 385, in _handle_tasks
    put(task)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/connection.py", line 206, in send
    self._send_bytes(ForkingPickler.dumps(obj))
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/reduction.py", line 50, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function do at 0x10388c268>: attribute lookup do on __main__ failed
Can't pickle <function do at 0x10388c268>: attribute lookup do on __main__ failed

I have tested the code on Python 3.5.2 on ubuntu and mac os. It is working correctly in default python's REPL, in jupyter, and by running it directly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions