mike-marcacci / node-redlock Goto Github PK
View Code? Open in Web Editor NEWA node.js redlock implementation for distributed, highly-available redis locks
License: MIT License
A node.js redlock implementation for distributed, highly-available redis locks
License: MIT License
Is it s, ms, or relative to ttl?
Best regards,
Alex
node-relock version: 4.0.0
ioredis version: 4.11.2
Hi--
I'm currently trying to implement node-redlock
into my application. When I run it locally, it works like a charm, doing exactly what I expect it to do.
However, when I put it into my QA environment (Amazon Elasticache cluster with 3 nodes), the lock
promise never resolves, no error is thrown, and even the clientError
handler does not fire. Any thoughts about what might be the root cause of this?
Here's my code
async isLocked(params) {
const {id} = params;
const resource = `my-resource-${id}`;
const ttl = 5000;
try {
await redlock.lock(resource, ttl);
// never returns
return false;
} catch (e) {
console.log(e);
return true;
}
}
I appreciate your help--please let me know if you need additional information. Thanks!
node-relock version: 4.1.0
ioredis version: 4.14.1
Hi, I'm trying to use node-redlock with redis-cluster. When locking multi resources, I got a "LockError" exception. It exceeds the 10 attempts to lock.
Here's my code
let client = new redis.Cluster([
{
ip: '127.0.0.1',
port: '7001'
},
{
ip: '127.0.0.1',
port: '7002'
},
{
ip: '127.0.0.1',
port: '7003'
}
])
let lk = new redlock([client])
let l = null
l = await lk.lock(['lk1', 'lk2'], 1000) //failure
//l = await lk.lock(['lk'], 1000) //success
let res = await client.get('foo')
(node:28707) UnhandledPromiseRejectionWarning: LockError: Exceeded 10 attempts to lock the resource "lk1,lk2".
at /home/sx/projects/nodejs/njproj1/node_modules/redlock/redlock.js:411:20
at tryCatcher (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/util.js:16:23)
at Promise.errorAdapter [as _rejectionHandler0] (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/nodeify.js:35:34)
at Promise._settlePromise (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/promise.js:601:21)
at Promise._settlePromise0 (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/promise.js:649:10)
at Promise._settlePromises (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/promise.js:725:18)
at _drainQueueStep (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/async.js:93:12)
at _drainQueue (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/async.js:86:9)
at Async._drainQueues (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/async.js:102:5)
at Immediate.Async.drainQueues (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/async.js:15:14)
at runCallback (timers.js:794:20)
at tryOnImmediate (timers.js:752:5)
at processImmediate [as _immediateCallback] (timers.js:729:5)
seems promising... 👍
Have you tested this with ioredis
on Redis 3.0 Cluster?
Hi @mike-marcacci,
I tried to understand the redlock performance under high contention, still don't really understand. So I would be grateful if you could help me understand. I m developing a real-time collaborative writing like Google Word. In that system we have a number of NodeJS servers, and when a new user comes to the system (connected to one of the NodeJS servers), we assign unique authorship color.
During that time, no matter the nodejs server, need to access the same few Redis entries to determine color. During that particular time, these Redis entries have to be RedLocked from other servers.
So let's say, if 1000 users connect to different NodeJS servers at the same time, with the Redlock on the same Redis entries, do you have some idea on Redlock performance? (each connection takes ~14 milliseconds). It is also very difficult to test concurrency :(
Thank you so much, really appreciate it.
There is an option string_numbers
in Redis client (ref: https://github.com/NodeRedis/node_redis#options-object-properties) to return number values as strings.
Node-redlock is incompatible with this option because it expects the response to be always a number: https://github.com/mike-marcacci/node-redlock/blob/master/redlock.js#L203. In result, setting string_numbers
to true
causes an error in Redlock.unlock(): LockError: Unable to fully release the lock on resource
.
Hey, man, would it be better if i can obtain multi resource by one lock . Think about this use case :
In a shopping application , when a user place order with multi sku and count, it is a good idea to use redlock to lock the sku and judge is there enough inventory to sell.
locker.lock(resource, ttl).then(_ => {
// this code never called
})
// the string identifier for the resource you want to lock
var resource = 'locks:account:322456';
remove this
message:
'ERR Error running script (call to f_862f31bf38e9b63ce28d6e9e5143551bb34ef981): Wrong number of args calling Redis command From Lua script ',
command:
{ name: 'eval',
args:
[ 'return redis.call("set", KEYS[1], ARGV[1], "NX", "PX", ARGV[2])',
'1',
'shop_wv_test1',
'f9a2445c012a1d2f2f38291a434c0731',
'3000' ] } } undefined
2019-03-14 15:11:18,217 ERROR 1370 nodejs.LockError: Exceeded 10 attempts to lock the resource "shop_wv_test1".
at /Users/wv/work/qianmi/api/shop/packages/shop-bff/node_modules/redlock/redlock.js:343:20
at runCallback (timers.js:694:18)
at tryOnImmediate (timers.js:665:5)
at processImmediate (timers.js:647:5)
From previous event:
at Redlock._lock (/Users/wv/work/qianmi/api/shop/packages/shop-bff/node_modules/redlock/redlock.js:282:9)
at Redlock.lock (/Users/wv/work/qianmi/api/shop/packages/shop-bff/node_modules/redlock/redlock.js:139:14)
at Object.task (/Users/wv/work/qianmi/api/shop/packages/shop-bff/app/schedule/goods-name-index-all.ts:47:23)
name: "LockError"
message: "Exceeded 10 attempts to lock the resource "shop_wv_test1"."
Is there a way to pass around the lock object, or instantiate a new lock from some secret? My use case is to have a browser-based client use a server running this package to proxy locking/unlocking.
First of all, let me congratulate you for the way you handle your issues, your answers are complete and thoughtful.
Now, to my issue:
I'm seeing a relatively small (1-3% out of around 100.000 tries) of LockError
when trying to lock a resource. It exceeds the 10 attempts to lock.
What are the possible causes for a lock to fail?
From what I could gather, I am not attempting to lock the same resource twice (there are no other calls to lock a resource with the same tag LOCK:STATUS:PAYLOAD:${id}
)
Is there something I could do about it? I know it's hard without sharing any more code.
Thanks in advance.
node-redlock is not working with async-redis, sample code
// ...do something here...
will never execute.
Hi,
First - even though I haven't used this module yet, I want to thank for the work and support you're giving here . I really think it is great!
Now, I have this scenario where I have a scheduler in my app that execute scheduled tasks (or cron jobs). This scheduler is part of the app (I know, bad design, but that's what I currently have), so once I start the instance, I also get the scheduler running.
In one-node scenario this works fine. Now I want to scale up, and as a quick-fix, I though to use redlock, though I am not sure I can really get the right "guarantees".
I have read issues discussing similar requirements/concerns (e.g. #15, #19, and #21 which is similar) but still not 100% sure how I should implement this. Mainly around retries and errors that are reported.
My initial thought is to use 0 retries, that why, given a task T1 to be executed (at some time t),
But I think this will not guarantee that the task will be executed.
Failure to get the lock (if I understand correctly) could mean that either some other worker got the lock before (and W1 should not do the work), or W1 has some error (e.g. didn't get quorum because it is in a "smaller" side of a partition), and should retry.
Is there a way to implement a "Execute Exactly Once" logic with RedLock alone or maybe I should resort to using some additional state and allow retries?
An example of the above would be to save an execution log/state that once the worker got a lock, it will check against and return if there's a log entry indicating the work was executed.
Note that in this way, the critical section would include the work, and the update of the log. In such implementation, a case of many workers competing for a task with several retries means 1 worker will do the actual work and N-1 workers will try to acquire the lock until they succeed and do nothing once they have the lock.
Thanks (again, and in advance! ;-))
On a simple cluster with 3 nodes created using the redis-trib.rb script, I get an "(error) MOVED" as a client error when I try to lock a resource.
Is this intended? How should these redirects be handled?
Including node's standard "events" library broke support for node v0.10.x, which we should probably still support.
See comments here.
pub.js
const Redis = require('ioredis');
const env = {
host: process.env.SOCKET_IO_ADAPTER_HOST || 'localhost',
port: ~~process.env.SOCKET_IO_ADAPTER_PORT || 6379,
};
const redis = new Redis(env);
const key = 'channel';
setInterval(async () => {
await redis.set(key, 'ok');
await redis.expire(key, 1);
}, 2000);
test.js
const Redis = require('ioredis');
const Redlock = require('redlock');
const env = {
host: process.env.SOCKET_IO_ADAPTER_HOST || 'localhost',
port: ~~process.env.SOCKET_IO_ADAPTER_PORT || 6379,
};
const redis = new Redis(env);
const redlock = new Redlock([redis]);
(async () => {
const lock = await redlock.lock('locks:000', 2000);
console.log('ok here.');
await lock.unlock();
redis.psubscribe('__keyevent@*__:expired');
redis.on('pmessage', async (p, c, m) => {
try {
// it throws: Exceeded 10 attempts to lock the resource "locks:001".
const lock = await redlock.lock('locks:001', 2000);
console.log(p, c, m);
await lock.unlock();
} catch (e) {
console.error('e', e.message);
}
})
})();
run node pub
and node test
in deferent terminals.
Hello.
Briefly - how would I detect situation where I cannot get lock because of all Redis instanes gone (eg. db connection error on all instances).
More details:
try {
lock = await redlock.lock("x", 1000);
} catch (err) {
// Db connection problem (what I want to log and response with appropriate error code/message)
// OR
// Alredy locked by other instance of my app and I want to ignore it (eg. I will respond with 202 code to signal that task already in progress)
}
Is there a way I can distinguish between type of errors, eg:
Is it possible that you will introduce LockError.code property and expose (export) some LOCK_ERROR_CODE_... consts?
Regards,
Wojtek
After upgrading to v4, I started getting this error LockError: Unable to fully release the lock on resource
.
Redis server v=5.0.7
. (appears to have the same problems in newer versions as well)
I rolled back to version 3.1.2 for now.
This could be related with #39
Hi,
I need to mock redlock to write unit testcases. npm redis-mock is not working for me. any suggestions?
Is there no function which directly unlocks on the basis of resourceId instead of the current function which needs an instance of Lock to unlock it.
It'd be awesome if you supported exponential backoff when retrying.
I want to know how many times it retried until get the lock, so is it possible to expose it to lock instance
Add a new method "tryLock", which doesn't keep waiting for the lock
I've 2 web-servers witha load balancer. I have a cron which I want to run on only one server. Can I use redlock for this?
When I used redlock like this:
var resource = 'redisKey';
function getData(){
redlock.lock(resource, ttl).then(function(lock) {
// logic of function
});
return lock.unlock()
.catch(function(err) {
console.error(err);
});
});
};
Here, my getData function(which is part of cron for which I need lock) is working fine, but I'm not able to check if my redisKey gets SET or not. I tried commenting lock.unlock()
but couldn't test it.
Any suggestions how can I test if my key it set by using redlock this way?
I have seen the following sequence of events and am wondering if I am using redlock incorrectly:
Function A's attempt to extend the lock never resolves.
Is there possibly a bug in the extend code that causes the lock to be released instead of extended?
I am using only 1 redis instance.
I've just read about the redlock algorithm and can't figure out is it a correct case of usage. Let's say I have a few workers and an observable which spies on some database. I use timeout to randomly pick a worker:
const redlock = new Redlock([...], { retryCount: 0 });
...
sub.on('message', (channel, message) => setTimeout(async () => {
try {
var lock = await redlock.lock('db1' /* use parsed message in the real app */, 1000);
} catch (e) {
console.log(e);
}
if (lock) {
const watcher = new DatabaseObservable('db1').subscribe(
message => {
console.log(m); // something has changed in the database
},
err => {
lock.unlock(); // the connection to the database was interrupted
},
() => {
lock.unlock(); // the connection to the database was closed normally
}
);
try {
while (1) {
await sleep(800); // hmm... the lock here could become dead
lock = await lock.extend(1000);
}
} catch (e) { ... }
}
}, Math.random()));
sub.subscribe('watch')
When I need to create a new database observable I do redis.publish('watch', 'db1')
. For each database there can be only one observable at the same time. The lock unlocks once the connection to the database is interrupted or closed.
Redis.publish('watch', 'db1')
unless the key is locked? Something like:// in the server process around a websocket connection
setInterval(() => {
if ( /* unlocked */) {
redis.publish('watch', 'db1')
}
}, 1000);
Hi, thanks for making this available. I have a question regarding this comment in the configuration section of README.
// you should have one client for each redis node
// in your cluster
I am running a 9 node redis cluster with 3 shards and each shard supported by one primary and 2 slaves hosted in Amazon Elastic cache. For the normal use of Redis in my node based app cluster I just use a single load balancer provided DNS name to access the cluster for various redis operations. I do have redis-cli access to each of the individual nodes and therefore able to supply individual clients for redlock.
To use this cluster with redlock, do I need to create clients for all 9 nodes or is it OK to just supply a separate client for each of the 3 shard masters (or any other 3 nodes).
I just had an issue in which we had accidentally pointed to a specific node instead of the replica set URL, and the replica set changed the primary, so we started getting ERROR: READ-ONLY
errors. However the errors we saw from redlock
were just Exceeded X attempts to lock...
It would be nice to have a debug mode or some other method of logging for if err
is not null
in the loop
function. I'm happy to submit a PR - what do you think is the best approach?
Hi,
I am trying to do something with Redlock (which it may not be designed for):
redlock
.lock(lockResource, lockTtl)
.then(lock => {
// do stuff
})
.catch(lockErr => {
// lockErr instanceOf redlock.LockError?
if (technical/network error) {
// still do stuff
} else {
// when the lock is created by some other process (but infrastructure is healthy)
// do nothing
}
});
I want to know if the lock acquisition failed because someone else locked it or due to technical issues because in case of technical or network errors I want to perform the job without the lock.
What is the best way to accomplish this?
Best regards,
Alex
Thank you for this awesome module!
Is there a way to unlock / close a lock you don't have a reference to anymore but have the resource name.
For instance, what if lock a resource for a long time (15 minutes like this):
function test(resource){
redlock.lock(resource, 900000).then(function(lock) {
// ...do something here...
// but no unlock
});
}
Then later on, how could I unlock the resource? My use case is that I have conditions to check one at a time and locking them one-by-one. If all conditions are fulfilled, I have to unlock all the gathered locks. I would prefer to avoid accumulating references to the lock
s as my iterations are executed recursively 😕 .
How to list the active locks? Tried "KEYS *" and it doesn't return the lock.
Just in case the lock is locked with a very big TTL and node crashed without unlock. Is it possible to unlock it via cli?
Thanks!
I have read https://redis.io/topics/distlock , still not clear what kind of locking it is. Could u shed me some light? Is it read lock/write lock or absolute lock where other party can do nth but wait?
The documentation seems to imply that extend()
will extend the current timeout by X ms. So I'd think this would extend the lock to 15sec:
redlock.lock('locks:account:322456', 5000, function(err, lock) {
lock.extend(5000, function(err, lock){
lock.extend(5000, function(err, lock){
// 15 secs or 5 secs?
});
});
});
But extend()
appears to only overwrite the ttl, so it's only 5sec in the above example. This should be clarified in the docs, or extend()
should be renamed ttl()
since it's just resetting the ttl to a new value.
Would be nice to have a quit or close method to ensure clients used exclusively for redlock are closed when done.
var redisClient = redis.createClient();
var redlock = new Redlock({
driftFactor: 0.01,
retryCount: 225,
retryDelay: 250
}, redisClient);
redlock.lock(LOCK_PUBLIC_LIST, LOCK_TIME, function(error, lock) {
if (error) {
console.log(error);
}
});
Will produce the error: TypeError: undefined is not a function
If create client is made with a public facing IPv4 and port, it works every time
@mike-marcacci
I have a local redis cluster: from 7001 to 7006
There is no problem with the normal operation of this cluster (such as set)
When I use single machine (6379), Cluster single node (7001), ioredis.cluster as parameters to pass into redlock, it is normal
But according to your demo, using an array of nodes will report an error: ReplyError: MOVED 15784 127.0.0.1:7003
However, 127.0.0.1:7003 does generate set information
Below is my code, I hope you can see and give me some Suggestions, thank you!
let conf = [{
"host": "127.0.0.1",
"port": 6379
}, {
"host": "127.0.0.1",
"port": 7001
},
{
"host": "127.0.0.1",
"port": 7002
},
{
"host": "127.0.0.1",
"port": 7003
},
{
"host": "127.0.0.1",
"port": 7004
},
{
"host": "127.0.0.1",
"port": 7005
},
{
"host": "127.0.0.1",
"port": 7006
}
];
let ioredis = require('ioredis');
let redlock = require('redlock');
// fork
let redis = [new ioredis(conf[1])];
// array
let redisList = conf.map(e => new ioredis(e));
// cluster
let redisCluster = [new ioredis.Cluster(conf)];
// redlock
let __R = new redlock(redisList, {
driftFactor: 0.01, // time in ms
retryCount: 10,
retryDelay: 200, // time in ms
retryJitter: 200 // time in ms
});
// error
__R.on('clientError', function(err) {
console.error(err);
debugger
});
// lock
__R.lock("LOCK-KEYS", 1000).then((lock) => {
debugger
return lock.unlock()
.then(() => {
debugger
})
.catch((err) => {
debugger
});
});
I have Master server and several Clusters connected to it to process data. Master is locking resource and I want one cluster to unlock it once it's finished.
So, is there a way to lock resource on Master and unlock it on Worker / Cluster? Or is there a way to restore lock
object by resource name?
I need to handle my queue items differently depending on which type of error is thrown. For example, if I get a retry error when lock exists (#12), I'd just like to remove the queue item completely. Currently I'm just matching on the error string (https://github.com/mike-marcacci/node-redlock/blob/master/redlock.js#L309), but it would be great if each error had a different name or code associated w/ it so I could be more explicit.
If that sounds good I can PR, but wondering if you have thoughts on how they should be distinguished (name vs. code, type of code, etc.)
We’re using fakeredis
, but all other Redis simulators that I can find have the same limitation: they don’t support EVAL
.
It looks like redlock is using EVAL
to let the user replace the lock/unlock/extend commands. Would it be possible to use non-EVAL
methods in the case where the user doesn’t choose to customize those commands? For example in logs I see this:
1497911517.567063 [0 127.0.0.1:58782] "eval" "return redis.call(\"set\", KEYS[1], ARGV[1], \"NX\", \"PX\", ARGV[2])" "1" "my-app:redlock:dev:testing" "4f827527f08e4379b39cec4e460b6d16" "3000"
That could be replaced by a plain SET
command, which would work in fakeredis
.
Referenced in: https://github.com/hdachev/fakeredis/issues/38
Is there a way to check if any key is locked at the moment?
Something like
redlock.locked(key).then(results => {
console.log(result); //true -> locked, false -> not locked
})
So client code can differentiate failing to acquire the lock from other errors (like redis connectivity issues). Ability to do stuff like this would be nice:
if (err instanceof redlock.LockError) {
// …
}
Hey,
just had to search the code to figure out what are the ttl time units.
so I think it will be easier if it will just be written in the readme.
I'm going to be updating this library over the next day or so to add support for promises (while continuing to support callbacks, of course).
It is currently not possible to ask redlock
not to retry after failure without something like this:
var redlock = new Redlock([redis], {retryCount: new Number(0)});
I understand that retrying failed lock is an important part of distlock
algorithm, but this option might be worth adding for more simple scenarios, like 1 lock per user account to prevent user from doing things like accidental double HTTP POST on page refresh.
For sentinel usage, can I pass a single ioredis client initialized with sentinel option?
For cluster usage, can I pass a single ioredis Cluster client?
pass all nodes
method in documents looks ugly to me!
Hi,
Great work !
What do you think about adding an optional argument to lock function instead of setting a random value like :
const lock = await redlock.lock('lock:123', 'theGuyOrServiceLocking', 1000);
It could be useful in order to log/send an error message indicating which account is locking resource for example.
Thanks,
I have 3 independent service service,such as serviceA, serviceB, serviceC. When I close serviceA, I find that the callback have not result when use the api Redlock.prototype.lock in callback style. But serviceB, serviceC is runnig, the api Redlock.prototype.lock can not get the expect result
Redis Client
ioredis
Sample Code
const test = async () => {
const Redis = require('ioredis')
const Redlock = require('redlock')
const client = new Redis('localhost:6379')
const redLock = new Redlock([client])
try {
await client.setex('foo', 6000, JSON.stringify({ name: 'Bob' }))
const lock = await redLock.lock('foo', 2000)
} catch (e) {
console.log(e)
}
}
test()
Error
LockError: Exceeded 10 attempts to lock the resource "foo"
I dug into the code to inspect what was going on and it looks like the call to acquire the lock never returns a response (ie: OK
) although there were no errors. This means the check to increment votes fails and we never get the lock. Any thoughts on this or am I way off?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.