rwojsznis / sidekiq-lock Goto Github PK

I didn't had time to test it with new sidekiq, creating this issue so I won't forget to do this this week(end) ;].

Something like.

include Sidekiq::Lock::Worker
sidekiq_options lock: [{ timeout: nil, name: 'lock-worker1' }, { timeout: nil, name: 'lock-worker2' }]

Hello,

Gem sidekiq-lock and sidekiq-unique-jobs use the same option
sidekiq_options lock: :until_executing
sidekiq_options lock: { timeout: 1000, name: 'project_status_lock' }
which causes an error:
NoMethodError: undefined method to_sym for {:timeout=>10000, :name=>"project_status_lock"}:Hash Did you mean? to_s, to_set /usr/local/bundle/gems/sidekiq-unique-jobs-6.0.13/lib/sidekiq_unique_jobs/options_with_fallback.rb:52:in lock_class'
because sidekiq-unique-jobs does not expect Hash as lock option value which need to be defined as hash in sidekiq-lock.

So far, I have not found a way to make these two gems work together.

lock - as a method name - should be configurable globally via sidekiq.configure_... call (don't want to do it per worker, it seems like a overkill).

Hi,

Thank you for making this gem.

I spent a little time getting it to work with ActiveJob (very small change at the end). I'd be happy to put up a PR for it, but I'm not sure how you would like to do the check. Right now my change exists as a small monkeypatch:

module Sidekiq
  module Lock
    class Middleware
      def call(_worker, msg, _queue)
        klass = msg.fetch('wrapped').constantize
        options = klass.get_sidekiq_options['lock']

        setup_lock(options, msg['args']) unless options.nil?

        yield
      end
    end
  end
end

The issue is that the job class for ActiveJob is always a ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper, rather than the worker class directly, meaning that when it tried to get_sidekiq_options['lock'], they were unavailable. This patch merely gets to the correct class where the options actually exist.

The jobs themselves can be used as documented (make sure to include Sidekiq::Lock::Worker in the job class that needs locking functionality)

Is this behavior desirable? If so, how would you prefer that the conditional check be made?

I wonder if I can use multiple locks? The main reason - to be able to limit how many same job workers can go in parallel. So say I want to limit 1 job to be run concurrently with only 2 worker threads - can I somehow set 2 locks and then acquire first or second?

When trying to use dynamic locks based on passed in parameters to the worker

timeout: proc { |user_id, timeout| timeout * 2 },

I end up with an error before the worker's perform method can be called:

 WARN: undefined method `*' for nil:NilClass

But it seems to work fine if I keep the timeout value equal to a constant and just use the proc on the lock.

sidekiq_options lock: {
      timeout: 1200000,
      name:    proc { |message_id, timeout| "lock:attachmentmsgid:#{message_id}" }
    }

def perform(message_id, timeout = 1200000)
    Rails.logger = Sidekiq::Logging.logger
    if lock.acquire!
    begin

I feel like im going crazy, what is wrong with this code?

[1] pry(#<CampaignBatchTestJob>)> lock.acquire!
TypeError: Unsupported command argument type: TrueClass

class CampaignBatchTestJob

  include Sidekiq::Job
  include Sidekiq::Lock::Worker

  sidekiq_options queue: "default", retry: 0, lock: {
    name: proc {|id| "campaign_batch_test_job_#{id}"},
    timeout: 10.minutes.to_i * 1000, # millseconds
  }

  def perform(id, duping = false)
    binding.pry
    return unless lock.acquire!

    sidekiq (7.2.0)
      concurrent-ruby (< 2)
      connection_pool (>= 2.3.0)
      rack (>= 2.2.4)
      redis-client (>= 0.14.0)
    sidekiq-lock (0.6.0)
      redis (>= 3.0.5)
      sidekiq (>= 2.14.0)

Hi,

I just added the gem and I'm getting, NameError: undefined local variable or method `lock'
I see the gem in the gemfile.lock list.
I added the following to my worker

include Sidekiq::Lock::Worker
sidekiq_options lock: { timeout: 1000, name: 'lock-worker' }

Any idea why it may not be working?

Hello,

Is there a way to make an infinite timeout? I have some tasks that can take hours, I don't want the lock to be released until it's finished :/

Also, maybe it wouldn't hurt to emphasize that you have to lock.release! after your worker is done otherwise new workers won't aquire the lock until timeout is reached.

I think Sidekiq lock should still allow workers to be run inline. Locking behavior should be the same

via Worker.new.perform(args)

As of right now lock is nil.

I'm implementing this functionality now. Hopefully you wont mind it being merged back in

Thanks

I'm using this as such:

class BatchIntegrationEventsJob
  include Sidekiq::Job
  include Sidekiq::Lock::Worker

  sidekiq_options({
    retry: 0,
    lock: {timeout: 30.seconds.to_i, name: self.class.to_s}
  })

  def perform
    return unless lock.acquire!
    sleep 10
  ensure
    lock.release!
  end
end

If I enqueue this job once per second I see multiple jobs running at same time, what am I doing wrong?

It would be cool to have some kind of test-helper if you are testing your sidekiq jobs inline (in one way or another).

I define a test job

class Job
  include Sidekiq::Worker
  include Sidekiq::Lock::Worker

  sidekiq_options queue: :test, retry: true, unique: :until_executing, lock: {
    timeout: 1.hour,
    name:    'index_lock',
    value:   proc { |klass, id, options= {}| "#{klass}:#{id}" }
  }

  def perform(klass, id, options = {})
    a = lock.acquire!
    puts a

    if a
      begin
        puts "process #{klass}, #{id}, #{options}"
        sleep 5.minutes
      ensure
        puts "release lock #{klass}:#{id}"
        lock.release!
      end
    else
      Job.perform_in 30.seconds, klass, id, options
    end
  ensure
    # GC.start
  end
end

Then I run

Job.perform_async 'a', 1
Job.perform_async 'a', 2
Job.perform_async 'a', 3
Job.perform_async 'a', 4

The log show

true
process a, 1, {}
true
process a, 1, {}
false
true
process a, 1, {}
true
process a, 1, {}

Error

Gem Load Error is: undefined method `configure_server' for Sidekiq:Module

raises on rails startup. I think it's because gem missing require "sidekiq" line.