Planned batch support? about solid_queue HOT 6 OPEN

This is one of the features which makes us rely on sidekiq currently. I would also like to see this being implemented in solidqueue.

from solid_queue.

Thanks for your kind words, @jpcamara! I really appreciate it 😊

We have batch support in our list of possible features to add, but it's not in the immediate plans because it's a bit at odds with the simplicity we're aiming for with Solid Queue. If you'd like to try implementing it, you're, of course, welcome to do so! 😊 I'm not quite sure yet how it could look like, in a way that maintains the overall simplicity of the gem, so I'm all open to ideas!

Thanks again!

from solid_queue.

Hi @rosa! I've opened a draft PR to introduce how I think batches could be implemented in SolidQueue: #142

Looking forward to feedback on whether it's the right direction or not, when you get some time!

from solid_queue.

@jpcamara You could take a look at https://github.com/cdale77/active_job_status. It's fairly simple, if it doesn't work with SolidQueue, it shouldn't be too hard to update it to work with SolidQueue as well.

from solid_queue.

Would be amazing to have this feature implemented in solid queue. Keeping my fingers crossed for it 🤞

from solid_queue.

I've been working on inkstak/activejob-status#32 and I just noticed 5ad8727 landed in main.

It could be a relatively simple way to approach batches - a batch could just be a collection of a job statuses - with different combinations of statuses adding up to different final batch statuses.

Here's my idea - The batch status can be queued, failed, completed or working.

1. The batch is considered queued if all of the jobs are queued
2. The batch is considered failed if one of the jobs is failed
3. The batch is considered completed if all of the jobs are completed
4. The batch is considered working in all other circumstances

On one hand, I dislike the solution we're currently using (with https://github.com/inkstak/activejob-status), because it relies on the cache for functionality - without the job status being stored in the cache, the application would simply break.

On the other hand, SolidQueue::Job#status is implementation specific, and like @rosa said, it's

This is prone to race conditions and stale reads, and it's just intended
for use in Mission Control or other reporting tools, not for regular
operation.

Even though using the collection of job statuses is a simple and effective approach, I'm still not sure what would be the ideal way forward here.

from solid_queue.