AttributeError: Can't pickle local object 'ETL_task.<locals>.task' about faktory_worker_python HOT 6 CLOSED

Try passing use_threads=True when creating a Worker instance:

w = Worker(..., use_threads=True)

from faktory_worker_python.

We are not using pickle specifically -- but it is used Python (internally) in the multiprocessing module.

Using dill would mean needing to modify multiprocessing to use it, and that is more baggage than I want to keep in this project :)

from faktory_worker_python.

After some investigation, I think the problem comes from the fact that the function that I register is wrapped by a decorator, and that the pickle library has problem pickling decorated functions. In my use case, I am registering data processing tasks that have some parts in common, and I've put all those in a decorator so that my coworkers only have to write the specific part of those tasks. It's part of my design that all the functions I register go through a decorator. That makes it quite problematic if such function cannot be pickled, and that faktory has to pickle registered functions. Any idea how I could work around this problem?

from faktory_worker_python.

Great, that just made it work; thanks for the answer. It's was already quite unclear for me why the pickle module had a problem with decorated functions, but why using threads solves the problem is even more of a mystery!

from faktory_worker_python.

I'm reading that using the dill library instead of pickle might solve the problem.

from faktory_worker_python.