Note: This is a complete piece of software, it should work across all future sidekiq & ruby versions.
Redis-based simple locking mechanism for sidekiq. Uses SET command introduced in Redis 2.6.16.
It can be handy if you push a lot of jobs into the queue(s), but you don't want to execute specific jobs at the same
time - it provides a lock method that you can use in whatever way you want.
This gem requires at least:
- redis 2.6.12
- sidekiq 6
Add this line to your application's Gemfile:
gem 'sidekiq-lock'And then execute:
$ bundleSidekiq-lock is a middleware/module combination, let me go through my thought process here :).
In your worker class include Sidekiq::Lock::Worker module and provide lock attribute inside sidekiq_options,
for example:
class Worker
include Sidekiq::Worker
include Sidekiq::Lock::Worker
# static lock that expires after one second
sidekiq_options lock: { timeout: 1000, name: 'lock-worker' }
def perform
# ...
end
endWhat will happen is:
-
middleware will setup a
Sidekiq::Lock::RedisLockobject underThread.current[Sidekiq::Lock::THREAD_KEY](it should work in most use cases without any problems - but it's configurable, more below) - assuming you providedlockoptions, otherwise it will do nothing, just execute your worker's code -
Sidekiq::Lock::Workermodule provides alockmethod that just simply points to that thread variable, just as a convenience
So now in your worker class you can call (whenever you need):
lock.acquire!- will try to acquire the lock, if returns false on failure (that means some other process / thread took the lock first)lock.acquired?- set totruewhen lock is successfully acquiredlock.release!- deletes the lock (only if it's: acquired by current thread and not already expired)
sidekiq_options lock will accept static values or Proc that will be called on argument(s) passed to perform method.
- timeout - specified expire time, in milliseconds
- name - name of the redis key that will be used as lock name
- value - (optional) value of the lock, if not provided it's set to random hex
Dynamic lock example:
class Worker
include Sidekiq::Worker
include Sidekiq::Lock::Worker
sidekiq_options lock: {
timeout: proc { |user_id, timeout| timeout * 2 },
name: proc { |user_id, timeout| "lock:peruser:#{user_id}" },
value: proc { |user_id, timeout| "#{user_id}" }
}
def perform(user_id, timeout)
# ...
# do some work
# only at this point I want to acquire the lock
if lock.acquire!
begin
# I can do the work
# ...
ensure
# You probably want to manually release lock after work is done
# This method can be safely called even if lock wasn't acquired
# by current worker (thread). For more references see RedisLock class
lock.release!
end
else
# reschedule, raise an error or do whatever you want
end
end
endJust be sure to provide valid redis key as a lock name.
You can change lock to something else (globally) in sidekiq server configuration:
Sidekiq.configure_server do |config|
config.lock_method = :redis_lock
endIf you would like to change default behavior of storing lock instance in Thread.current for whatever reason you
can do that as well via server configuration:
# Any thread-safe class that implements .fetch and .store methods will do
class CustomStorage
def fetch
# returns stored lock instance
end
def store(lock_instance)
# store lock
end
end
Sidekiq.configure_server do |config|
config.lock_container = CustomStorage.new
endAs you know middleware is not invoked when testing jobs inline, you can require in your test/spec helper file
sidekiq/lock/testing/inline to include two methods that will help you setting / clearing up lock manually:
set_sidekiq_lock(worker_class, payload)- note: payload should be an array of worker argumentsclear_sidekiq_lock
- Fork it
- Create your feature branch (
git checkout -b my-new-feature) - Commit your changes (
git commit -am 'Add some feature') - Push to the branch (
git push origin my-new-feature) - Create new Pull Request
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent