Solid Queue is not retrying job about solid_queue HOT 10 CLOSED

I did try it with longer wait time like 5 and 10 seconds but let me go observe with longer wait times and confirm and report back.

Thanks @rosa

from solid_queue.

Thanks @virolea, @zainonrails!

Just one question though, does attempt: 3 means the job would run for a total of 3 times? or retried 3 times?

A total of 3 times according to Active Job's attempts parameter.

from solid_queue.

Hey @zainonrails, what's your workers and dispatchers configuration? How are you enqueuing the job?

from solid_queue.

@rosa I am enqueuing the job simply by MyJob.perform_later(id)

after adding database record in solid_queue_jobs and solid_queue_ready_executions tables it shows logs below.

[SolidQueue] Enqueued job {:queue_name=>"development_default", :active_job_id=>"b720f935-3277-403f-85a3-782794d79937", :priority=>nil, :scheduled_at=>Tue, 02 Jan 2024 00:59:10.827070000 UTC +00:00, :class_name=>"SubmitTestimonialJob", :arguments=>{"job_class"=>"SubmitTestimonialJob", "job_id"=>"b720f935-3277-403f-85a3-782794d79937", "provider_job_id"=>nil, "queue_name"=>"development_default", "priority"=>nil, "arguments"=>[49], "executions"=>0, "exception_executions"=>{}, "locale"=>"en", "timezone"=>"UTC", "enqueued_at"=>"2024-01-02T00:59:10Z"}, :concurrency_key=>nil}
05:59:10 web.1   | [ActiveJob] Enqueued SubmitTestimonialJob (Job ID: b720f935-3277-403f-85a3-782794d79937) to SolidQueue(development_default) with arguments: 49

production:
  dispatchers:
    - polling_interval: 5
      batch_size: 100
  workers:
    - queues: '*'
      threads: 2
      processes: 1
      polling_interval: 5

development:
  dispatchers:
    - polling_interval: 1
      batch_size: 10
  workers:
    - queues: "*"
      threads: 3
      processes: 1
      polling_interval: 1

from solid_queue.

Cool, and when you say "it is not working consistently" and "The behaviour seems unpredictable", what do you mean? Could you be more specific?

from solid_queue.

So, I am throwing exception in my job to see if it would pick up the retry settings or not.

After the job throws error, I quickly update the job code to remove the exception part to see if it retries it but doesn't happen unfortunately. and upon restarting the server it picks up all the jobs.

Maybe I am doing something wrong here to test this behaviour. Can you brief me on if the settings would be picked up if a worker receives an error in the job.

Should I override on_thread_error to something that would make sure the job is retried.

from solid_queue.

After the job throws error, I quickly update the job code to remove the exception part to see if it retries it but doesn't happen unfortunately. and upon restarting the server it picks up all the jobs.

Hmm... my guess is that the 3 retries happen before you get a chance to update the job:

retry_on StandardError, attempts: 3, priority: 0

This would be using the default wait time between retries, which is 3 seconds, so after roughly 9 seconds, the 3 attempts would have been done and your job would fail permanently. You should see it in the solid_queue_failed_executions table, and check its arguments, by doing something like:

$ bin/rails c
>> SolidQueue::FailedExecution.last.job

from solid_queue.

I too wanted to check if jobs were properly retried in my setup. Turns out the first retry occurences happen quite quickly as suggested by Rosa.

You can also check the failed job arguments to see if any retry was performed. ActiveJob increment the executions value.

job_id = YOUR_JOB_ID
@job = SolidQueue::Job.find(job_id)
@job.arguments["executions"] # This returns the number of times this job was executed
@job.arguments["exception_executions"] # This returns the breakdown of executions per exceptions

Here's an example for one of my jobs:

{
  "executions"=>12, 
  "exception_executions"=>{"[Exception]"=>8, "[ZeroDivisionError]"=>4}
}

from solid_queue.

@rosa @virolea

I experimented it this timw without changing the code in between, with just 1 minute to wait for retry.

The job was retried as expected and the number of executions were also updated. Just one question though, does attempt: 3 means the job would run for a total of 3 times? or retried 3 times?

What I observed is that it would run a total of that amount we set and not retry counts, that's why executions is attempts - 1

from solid_queue.

Thanks for the help everyone, closing this issue.

from solid_queue.