python multiprocess pool模块报错pickling error

2023年12月17日 133次阅读

问题

之前在调用class内的函数用multiprocessing模块的pool函数进行多线程处理的时候报了以下下错误信息:

PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

查了下官方文档发现python默认只能pickle以下的类型:

  • None, True, and False
  • integers, floating point numbers, complex numbers
  • strings, bytes, bytearrays
  • tuples, lists, sets, and dictionaries containing only picklable objects
  • functions defined at the top level of a module (using def, not lambda)
  • built-in functions defined at the top level of a module
  • classes that are defined at the top level of a module
  • instances of such classes whose dict or the result of calling getstate() is picklable (see section –
  • Pickling Class Instances for details).

函数只能pickle在顶层定义的函数，很明显的class内的函数无法被pickle因此会报错。

import multiprocessing

def work():   # top-level 函数
    print "work!"

class Foo():
    def work(self):  # 非top-level函数
        print "work"

pool1 = multiprocessing.Pool(processes=4)
foo = Foo()
pool1.apply_async(foo.work)
pool1.close()
pool1.join()
# 此时报错

pool2 = multiprocessing.Pool(processes=4)
pool2.apply_async(work)
pool2.close()
pool2.join()
# 此时工作正常

解决方案

调用pathos包下的multiprocessing模块代替原生的multiprocessing。pathos中multiprocessing是用dill包改写过的，dill包可以将几乎所有python的类型都serialize，因此都可以被pickle。或者也可以自己用dill写一个（有点重复造轮子之嫌啊）

参考

  1. https://stackoverflow.com/questions/8804830/python-multiprocessing-pickling-error
  2. https://docs.python.org/3/library/pickle.html#what-can-be-pickled-and-unpickled
  3. https://github.com/uqfoundation/pathos
点赞