Simple example does not work? about sidekiq-lock HOT 13 CLOSED

ughhhhhhhhhhh that works. I have no idea why I didn't see that in the docs, I think because I was looking at tons of different lock gems that all use seconds so it was just stuck in my mind.... thank you

from sidekiq-lock.

Hey there!

If I enqueue this job once per second I see multiple jobs running at same time

Can you explain your use case? This gem doesn't prevent enqueuing multiple jobs nor running multiple jobs at the same time by sidekiq itself - so in that example multiple jobs might be running but only a single one at a time should execute for around 10 seconds - maybe this is what you're observing? If not - could you please create minimal repo that could reproduce your issue?

from sidekiq-lock.

Hey there, interesting. I knew it didn't prevent enqueueing multiple jobs, but I thought the early return when the lock fails to acquire would just cause the job to immediately get completed.

I guess that is because the lock timeout is the amount of time it waits for the lock?... I thought it was the time that the lock would eventually expire in the event of deadlocks, and the acquire! would immediately return false if it couldn't attain it (kind of like how redis-mutex would work in non-blocking mode)

My use case is... I need to send webhooks as soon as possible, which is why I have a batch job set to check every 1 second. The reason I can't enqueue the job at the time the previous work is done (pushing vs polling) is because there are multiple "transformations" the record must go through, and all those happen out of order. It was far easier to simply enqueue a poll job every 1 second, and while the job is running, any other jobs enqueued would just immediately return. I'm also not taking the "embrace concurrency" approach here with sidekiq because it's dealing with potentially 100's of thousands of records in very short timespans, and 10 webhook requests that sends 500 records in a single request is far better than 5000 separate requests.

Basically I want 1 job running at a single time, initiated every second, and any jobs that get added while one is running it just immediately returns

from sidekiq-lock.

but I thought the early return when the lock fails to acquire would just cause the job to immediately get completed.

That's exactly what should happen; in that example sidekiq would pick up the job and mark it as completed when lock can't be acquired; it really depends how you observe your workers - what's why I asked for the repo :)

I guess that is because the lock timeout is the amount of time it waits for the lock

Timeout is the time after which lock expires automatically after it's been obtained - this gem is dead-simple and does not implement any waiting time; acquire! will acquire the lock or not - it's speed is purely dependent on redis. There is no pooling involved.

Did you tried looking into redlock or maybe sidekiq-unique-jobs - maybe it would fit your use-case better ❓

from sidekiq-lock.

Hmm, so based on what you are saying it seems like what I'm doing should work. I didn't want to unique-jobs as I also use throttle and there are a lot of issues in there about conflicts. Redlock looked a little to cumbersome for my very simple use case

So are you saying my use case should work fine, or your library wouldn't work? I might be misunderstanding

This is how I start workers

bundle exec sidekiq -C config/sidekiq.yml

This is in my yml:

---
:concurrency: <%= ENV['RAILS_MAX_THREADS'] || 10 %>
:timeout: 25
:queues:
  - default

and RAILS_MAX_THREADS is set to 10

The job I put above is literally as simple as it gets, and from there I use clockwork (a clock process) running in only a single worker/thread (so no duplicates) to schedule them like this:

  every(1.second, 'BatchIntegrationEventsJob') do
    BatchIntegrationEventsJob.perform_async
  end

I could make a repo to I just figured considering the example is so simple it might be clear what I should do. When you say "observe my workers" does that answer your question above?

To look at the job queue, if thats what you mean, I added the sidekiq route

  mount Sidekiq::Web, at: '/jobs'

from sidekiq-lock.

So there was one bug in my code, in that I put the ensure on the outer method, instead of AFTER where the lock is acquired.

I updated that in my code but I still see the same problem.

class BatchIntegrationEventsJob
  include Sidekiq::Job
  include Sidekiq::Lock::Worker

  sidekiq_options({
    retry: 0,
    lock: {timeout: 5.minutes.to_i, name: self.class.to_s}
  })

  def perform
    if !lock.acquire!
      puts 'NOT ACQUIRED'
      return
    end
    begin
      puts 'ACQUIRED... sleeping'
      sleep 60
    ensure
      puts "RELEASING"
      lock.release!
    end
  end
end

If I enqueue 5 jobs at once from console:

 BatchIntegrationEventsJob.perform_async; BatchIntegrationEventsJob.perform_async;BatchIntegrationEventsJob.perform_async;BatchIntegrationEventsJob.perform_async;BatchIntegrationEventsJob.perform_async;

I see this:

ACQUIRED... sleeping
NOT ACQUIRED
2022-04-04T17:24:02.383Z pid=51944 tid=18b8 class=BatchIntegrationEventsJob jid=46cda831b290817f0c8745f1 elapsed=0.284 INFO: done
NOT ACQUIRED
NOT ACQUIRED
2022-04-04T17:24:02.383Z pid=51944 tid=18mc class=BatchIntegrationEventsJob jid=fb7340f8326ad8c2741dda44 elapsed=0.284 INFO: done
2022-04-04T17:24:02.383Z pid=51944 tid=18ag class=BatchIntegrationEventsJob jid=00afefe65c2e3b56cabc6b03 elapsed=0.285 INFO: done
NOT ACQUIRED

If I wait 3 seconds and enqueue 5 more I see this:

ACQUIRED... sleeping
NOT ACQUIRED
2022-04-04T17:24:10.217Z pid=51944 tid=1898 class=BatchIntegrationEventsJob jid=fc09784f9fe32cbdb20ac18a elapsed=0.012 INFO: done
NOT ACQUIRED
2022-04-04T17:24:10.217Z pid=51944 tid=19lc class=BatchIntegrationEventsJob jid=ca6263006bf9b3cf9454dde1 elapsed=0.014 INFO: done
NOT ACQUIRED
2022-04-04T17:24:10.218Z pid=51944 tid=19jc class=BatchIntegrationEventsJob jid=b131eb79f9bc7a5da54de0a9 elapsed=0.013 INFO: done
NOT ACQUIRED
2022-04-04T17:24:10.218Z pid=51944 tid=19i8 class=BatchIntegrationEventsJob jid=178473ec2ce3485c44fbc87a elapsed=0.013 INFO: done

The second ACQUIRED... sleeping is peculiar to me

from sidekiq-lock.

Hey, sorry for the lag in response - I'm currently pretty occupied; can you please create a repo so I can reproduce this? (gemfile with lock, worker code mentioned above and steps how you're running worker exactly)

The only way such thing could happen (lock acquired twice) would be a collision on random lock value, which is very unlikely to happen 🤔

from sidekiq-lock.

https://github.com/9mm/lock

# console tab 1
bundle install
redis-server

# console tab 2
bundle exec sidekiq -C config/sidekiq.yml

# console tab 3
rails console

# inside rails console...
BatchJob.perform_async; BatchJob.perform_async; BatchJob.perform_async; BatchJob.perform_async; BatchJob.perform_async;

# wait 5 seconds
BatchJob.perform_async; BatchJob.perform_async; BatchJob.perform_async; BatchJob.perform_async; BatchJob.perform_async;

# wait 5 seconds
BatchJob.perform_async; BatchJob.perform_async; BatchJob.perform_async; BatchJob.perform_async; BatchJob.perform_async;

You will see multiple log lines of ACQUIRED... sleeping even though the first job is still sleeping (60 seconds) so no lock should be acquired. They should all say NOT ACQUIRED

Relevant files:

https://github.com/9mm/lock/blob/master/app/sidekiq/batch_job.rb
https://github.com/9mm/lock/blob/master/config/sidekiq.yml

from sidekiq-lock.

Thank you for providing the repo! This week is kinda hectic, I can't promise I will be able to take a look at it within next few days, but it's on my radar. Hopefully we will be able to get to the bottom of this soon enough :)

from sidekiq-lock.

Awesome, thank you!

from sidekiq-lock.

Ugh, I handled that case as your typical open source maintainer that responds after 4 years the issue was initially opened 😅 I feel kinda bad about it, but was pretty occupied recently (still am honestly speaking) - just a note that I didn't totally forget about it

from sidekiq-lock.

Hhaha, I understand. I look forward to whenever you can assist 😎

from sidekiq-lock.

Ugh, thank you for your patience @9mm <3 - I had a look at the repo (huuuge kudos for providing it) and I think I see now where the problem lies, I kinda forgot how my own gem works ;) Per readme

timeout - specified expire time, in milliseconds

Rails method 5.minutes.to_i will return number of seconds, can you try to change that to: 5.minutes.to_i * 1000 and let me know if now everything behaves as expected? 🤞

from sidekiq-lock.