aminediro / daskqueue Goto Github PK

Hi,
I have a react based webapp integrated with Django as a backend, the current challenge we are facing in file uploads like we are expecting 3-5lac file upload of average size say 5Mb each file. Currently we are implemented using celery and redis but it is not efficient to handle this much request. I wanted to know if dask queue can handle this efficiently. Can anyone please help me on this.

After multiple tests to profile the batch_submit method performance, I saw that bottleneck is the serialization in the distibuted.protocol . Basically when calling an attribute in an actor we use the RPCPool to send a send_recv call on communication layer which calls in return the protocol.dumps().

Msg size : 717
Protocol serialization : into list of 100002 protocol frames took 5.31s
Simple  cloudpickle.dumps() of message took 0.12s

This is too slow when submitting millions of tasks + I can optimize the storage layer to directly deal with serialized message.

I'll test using ZMQ for communication between client <-> QueuePool and Queue<-> workers.