qratorlabs / aio-job-queue Goto Github PK

Seems like it's time to start working on Tarantool queue implementation. I propose the following : a top-level Queue abstract class with general methods (put, get) and RedisQueue and TarantoolQueue classes in corresponding folders. RedisQueue is actually now implemented, making it compatible with abstract parent class would cost nothing

Right now we only support raw payloads. Would be nice to have options for pickle, safe_pickle, json, yaml and so on.

Original idea was to create alternative to asyncio.Queue that preserves tasks after program restart and supports acks().

Now API is too different but sometimes it would be still a good idea to have an alternative to aiothreading and ack() support in asyncio.Queue-like object.

We can implement the same API we have but with local object without communication with outside world: just a wrapper around queue.Queue (for thread safety) or similar.

This would be also very useful for tests.

serialization_method logic implemented in #11 results in data duplication left as tradeoff to maximal flexibility and interoperability of homogenous and heterogenous queues. serialization_method is both a part of Redis key (task id) an is a property of Queue (Queue._serialization_method). This data duplication could be avoided

On worker failure, i.e. when task is not acked and not failed, it should be returned to the queue after desired amount of time.