how to have a queue across two (or more) workers but still runs functions in sequence? about rq HOT 3 CLOSED

I accomplished a behavior similar to this by queuing the second job at the end of the first job. i.e. I run a script as my first job and at the end of the script I added some logic to enqueue a new job depending on the outputs that were just created.

from rq.

I ended up doing something similar to that, where I read the contents of the queue to see what jobs are running or scheduled, then add the new job to depend_on the last job in that list.

I was hoping to find a more elegant solution, though.

My code looks something like this:

        queue = 'the selected queue'
        job_id = 'some string'
        q = Queue(queue, connection=redis)
        from rq.registry import StartedJobRegistry, DeferredJobRegistry
        running_jobs = StartedJobRegistry(queue, connection=redis).get_job_ids()
        deferred_jobs = DeferredJobRegistry(queue, connection=redis).get_job_ids()
        with app.app_context():
            if len(deferred_jobs) > 0:
                last_job = deferred_jobs[-1]
                job = q.enqueue(f=args[0], job_timeout='1h', job_id=job_id, args=args[1:], depends_on=last_job)
            elif len(running_jobs) > 0:
                last_job = running_jobs[-1]
                job = q.enqueue(f=args[0], job_timeout='1h', job_id=job_id, args=args[1:], depends_on=last_job)
            else:
                job = q.enqueue(f=args[0], job_timeout='1h', job_id=job_id, args=args[1:])

from rq.

It sounds like you're using Queue as individualistic rather than a collection
[q1, q2, q3] is analogous to [high-prioritiy-queue, medium-priority-queue, low-priority-queue] within the queue namespace redis:queue:
Which is why you're seeing the behavior regarding job/function ordering

Given your use case, I would take a look at Queue(redis_queue_namespace_prefix: str). This would allow you to create a queue namespace per customer to allow for non-overlapping Jobs.
https://github.com/rq/rq/blob/a8209391fffe3bad195264e6ed22724e36ae1fb5/rq/queue.py#L69C1-L69C52

customer_foo: 'foo'
customer_bar: 'bar'
foo_q = Queue(['q1', 'q2', 'q3'], connection=redis, redis_queue_namespace_prefix=f'redis:queue:{customer_foo}:')
bar_q = Queue(['q1', 'q2', 'q3'], connection=redis, redis_queue_namespace_prefix=f'redis:queue:{customer_bar}:')
foo_q.enqueue(my_func_1)
bar_q.enqueue(my_func_1)
foo_q.enqueue(my_func_2)

Depending how you manage your workers, that might not be feasible.
Whether or not it is, If you aren't using dependencies already, I would recommend it.

Good luck!

from rq.