mike-marcacci / node-redlock Goto Github PK

For sentinel usage, can I pass a single ioredis client initialized with sentinel option?

For cluster usage, can I pass a single ioredis Cluster client?

pass all nodes method in documents looks ugly to me!

Hey, man, would it be better if i can obtain multi resource by one lock . Think about this use case :
In a shopping application , when a user place order with multi sku and count, it is a good idea to use redlock to lock the sku and judge is there enough inventory to sell.

I want to know how many times it retried until get the lock, so is it possible to expose it to lock instance

First of all, let me congratulate you for the way you handle your issues, your answers are complete and thoughtful.

Now, to my issue:
I'm seeing a relatively small (1-3% out of around 100.000 tries) of LockError when trying to lock a resource. It exceeds the 10 attempts to lock.

What are the possible causes for a lock to fail?
From what I could gather, I am not attempting to lock the same resource twice (there are no other calls to lock a resource with the same tag LOCK:STATUS:PAYLOAD:${id})

Is there something I could do about it? I know it's hard without sharing any more code.

Thanks in advance.

I'm going to be updating this library over the next day or so to add support for promises (while continuing to support callbacks, of course).

// the string identifier for the resource you want to lock
var resource = 'locks:account:322456';

1. Can somebody explains me the naming convention here? 'locks'? Are there other prefixes? And any significance?
2. Is it read lock? write lock? or what kind of lock?

Is it s, ms, or relative to ttl?

Best regards,

Alex

remove this

I have seen the following sequence of events and am wondering if I am using redlock incorrectly:

1. Function A requests a lock
2. Function B requests a lock (almost the same time)
3. Function A acquires lock.
4. Function A attempts to extend the lock.
5. Function B receives lock.
6. Function B attempts to extend the lock.
7. Deadlock.

Function A's attempt to extend the lock never resolves.

Is there possibly a bug in the extend code that causes the lock to be released instead of extended?

I am using only 1 redis instance.

On a simple cluster with 3 nodes created using the redis-trib.rb script, I get an "(error) MOVED" as a client error when I try to lock a resource.

Is this intended? How should these redirects be handled?

Redis Client
ioredis

Sample Code

const test = async () => {
  const Redis = require('ioredis')
  const Redlock = require('redlock')
  const client = new Redis('localhost:6379')
  const redLock = new Redlock([client])
  try {
    await client.setex('foo', 6000, JSON.stringify({ name: 'Bob' }))
    const lock = await redLock.lock('foo', 2000)
  } catch (e) {
    console.log(e)
  }
}

test()

Error

LockError: Exceeded 10 attempts to lock the resource "foo"

I dug into the code to inspect what was going on and it looks like the call to acquire the lock never returns a response (ie: OK) although there were no errors. This means the check to increment votes fails and we never get the lock. Any thoughts on this or am I way off?

node-relock version: 4.0.0
ioredis version: 4.11.2

Hi--

I'm currently trying to implement node-redlock into my application. When I run it locally, it works like a charm, doing exactly what I expect it to do.

However, when I put it into my QA environment (Amazon Elasticache cluster with 3 nodes), the lock promise never resolves, no error is thrown, and even the clientError handler does not fire. Any thoughts about what might be the root cause of this?

Here's my code

async isLocked(params) {
    const {id} = params;
    const resource = `my-resource-${id}`;
    const ttl = 5000;

    try {
        await redlock.lock(resource, ttl);
        // never returns
        return false;
    } catch (e) {
        console.log(e);
        return true;
    }
}

I appreciate your help--please let me know if you need additional information. Thanks!

Hi,

I need to mock redlock to write unit testcases. npm redis-mock is not working for me. any suggestions?

Hey,
just had to search the code to figure out what are the ttl time units.
so I think it will be easier if it will just be written in the readme.

Hi,

I am trying to do something with Redlock (which it may not be designed for):

redlock
   .lock(lockResource, lockTtl)
          .then(lock => {
              // do stuff
          })
          .catch(lockErr => {
             // lockErr instanceOf redlock.LockError?
             if (technical/network error) {
               // still do stuff
             } else {
               // when the lock is created by some other process (but infrastructure is healthy)
               // do nothing
             }
          });

I want to know if the lock acquisition failed because someone else locked it or due to technical issues because in case of technical or network errors I want to perform the job without the lock.
What is the best way to accomplish this?

Best regards,

Alex

Including node's standard "events" library broke support for node v0.10.x, which we should probably still support.

See comments here.

Hi,

First - even though I haven't used this module yet, I want to thank for the work and support you're giving here . I really think it is great!

Now, I have this scenario where I have a scheduler in my app that execute scheduled tasks (or cron jobs). This scheduler is part of the app (I know, bad design, but that's what I currently have), so once I start the instance, I also get the scheduler running.

In one-node scenario this works fine. Now I want to scale up, and as a quick-fix, I though to use redlock, though I am not sure I can really get the right "guarantees".

I have read issues discussing similar requirements/concerns (e.g. #15, #19, and #21 which is similar) but still not 100% sure how I should implement this. Mainly around retries and errors that are reported.

My initial thought is to use 0 retries, that why, given a task T1 to be executed (at some time t),

• If the worker, W1, managed to lock - it execute the T1
• If the worker, W1, didn't managed to lock, it doesn't do anything (0-retries)

But I think this will not guarantee that the task will be executed.

Failure to get the lock (if I understand correctly) could mean that either some other worker got the lock before (and W1 should not do the work), or W1 has some error (e.g. didn't get quorum because it is in a "smaller" side of a partition), and should retry.

Is there a way to implement a "Execute Exactly Once" logic with RedLock alone or maybe I should resort to using some additional state and allow retries?

An example of the above would be to save an execution log/state that once the worker got a lock, it will check against and return if there's a log entry indicating the work was executed.

Note that in this way, the critical section would include the work, and the update of the log. In such implementation, a case of many workers competing for a task with several retries means 1 worker will do the actual work and N-1 workers will try to acquire the lock until they succeed and do nothing once they have the lock.

Thanks (again, and in advance! ;-))

I have 3 independent service service,such as serviceA, serviceB, serviceC. When I close serviceA, I find that the callback have not result when use the api Redlock.prototype.lock in callback style. But serviceB, serviceC is runnig, the api Redlock.prototype.lock can not get the expect result

seems promising... 👍

Have you tested this with ioredis on Redis 3.0 Cluster?

I have Master server and several Clusters connected to it to process data. Master is locking resource and I want one cluster to unlock it once it's finished.
So, is there a way to lock resource on Master and unlock it on Worker / Cluster? Or is there a way to restore lock object by resource name?

Is there a way to pass around the lock object, or instantiate a new lock from some secret? My use case is to have a browser-based client use a server running this package to proxy locking/unlocking.

@mike-marcacci
I have a local redis cluster: from 7001 to 7006

There is no problem with the normal operation of this cluster (such as set)

When I use single machine (6379), Cluster single node (7001), ioredis.cluster as parameters to pass into redlock, it is normal

But according to your demo, using an array of nodes will report an error: ReplyError: MOVED 15784 127.0.0.1:7003

However, 127.0.0.1:7003 does generate set information

Below is my code, I hope you can see and give me some Suggestions, thank you!

let conf = [{
        "host": "127.0.0.1",
        "port": 6379
    }, {
        "host": "127.0.0.1",
        "port": 7001
    },
    {
        "host": "127.0.0.1",
        "port": 7002
    },
    {
        "host": "127.0.0.1",
        "port": 7003
    },
    {
        "host": "127.0.0.1",
        "port": 7004
    },
    {
        "host": "127.0.0.1",
        "port": 7005
    },
    {
        "host": "127.0.0.1",
        "port": 7006
    }
];

let ioredis = require('ioredis');
let redlock = require('redlock');

// fork
let redis = [new ioredis(conf[1])];
// array
let redisList = conf.map(e => new ioredis(e));
// cluster
let redisCluster = [new ioredis.Cluster(conf)];
// redlock
let __R = new redlock(redisList, {
    driftFactor: 0.01, // time in ms
    retryCount: 10,
    retryDelay: 200, // time in ms
    retryJitter: 200 // time in ms
});
// error
__R.on('clientError', function(err) {
    console.error(err);
    debugger
});
// lock
__R.lock("LOCK-KEYS", 1000).then((lock) => {
    debugger
    return lock.unlock()
        .then(() => {
            debugger
        })
        .catch((err) => {
            debugger
        });
});

Hi @mike-marcacci,

I tried to understand the redlock performance under high contention, still don't really understand. So I would be grateful if you could help me understand. I m developing a real-time collaborative writing like Google Word. In that system we have a number of NodeJS servers, and when a new user comes to the system (connected to one of the NodeJS servers), we assign unique authorship color.

During that time, no matter the nodejs server, need to access the same few Redis entries to determine color. During that particular time, these Redis entries have to be RedLocked from other servers.

So let's say, if 1000 users connect to different NodeJS servers at the same time, with the Redlock on the same Redis entries, do you have some idea on Redlock performance? (each connection takes ~14 milliseconds). It is also very difficult to test concurrency :(

Thank you so much, really appreciate it.

node-redlock is not working with async-redis, sample code

// ...do something here...

will never execute.

ref

Add a new method "tryLock", which doesn't keep waiting for the lock

It is currently not possible to ask redlock not to retry after failure without something like this:

  var redlock = new Redlock([redis], {retryCount: new Number(0)});

I understand that retrying failed lock is an important part of distlock algorithm, but this option might be worth adding for more simple scenarios, like 1 lock per user account to prevent user from doing things like accidental double HTTP POST on page refresh.

So client code can differentiate failing to acquire the lock from other errors (like redis connectivity issues). Ability to do stuff like this would be nice:

if (err instanceof redlock.LockError) {
  // …
}

pub.js

const Redis = require('ioredis');
const env = {
  host: process.env.SOCKET_IO_ADAPTER_HOST || 'localhost',
  port: ~~process.env.SOCKET_IO_ADAPTER_PORT || 6379,
};

const redis = new Redis(env);
const key = 'channel';

setInterval(async () => {
  await redis.set(key, 'ok');
  await redis.expire(key, 1);
}, 2000);

test.js

const Redis = require('ioredis');
const Redlock = require('redlock');
const env = {
  host: process.env.SOCKET_IO_ADAPTER_HOST || 'localhost',
  port: ~~process.env.SOCKET_IO_ADAPTER_PORT || 6379,
};

const redis = new Redis(env);
const redlock = new Redlock([redis]);

(async () => {
  const lock = await redlock.lock('locks:000', 2000);
  console.log('ok here.');
  await lock.unlock();
  redis.psubscribe('__keyevent@*__:expired');
  redis.on('pmessage', async (p, c, m) => {
    try {
      // it throws: Exceeded 10 attempts to lock the resource "locks:001".
      const lock = await redlock.lock('locks:001', 2000);
      console.log(p, c, m);
      await lock.unlock();
    } catch (e) {
      console.error('e', e.message);
    }
  })
})();

run node pub and node test in deferent terminals.

It'd be awesome if you supported exponential backoff when retrying.

Is there no function which directly unlocks on the basis of resourceId instead of the current function which needs an instance of Lock to unlock it.

We’re using fakeredis, but all other Redis simulators that I can find have the same limitation: they don’t support EVAL.

It looks like redlock is using EVAL to let the user replace the lock/unlock/extend commands. Would it be possible to use non-EVAL methods in the case where the user doesn’t choose to customize those commands? For example in logs I see this:

1497911517.567063 [0 127.0.0.1:58782] "eval" "return redis.call(\"set\", KEYS[1], ARGV[1], \"NX\", \"PX\", ARGV[2])" "1" "my-app:redlock:dev:testing" "4f827527f08e4379b39cec4e460b6d16" "3000"

That could be replaced by a plain SET command, which would work in fakeredis.

Referenced in: https://github.com/hdachev/fakeredis/issues/38

After upgrading to v4, I started getting this error LockError: Unable to fully release the lock on resource.

Redis server v=5.0.7. (appears to have the same problems in newer versions as well)

I rolled back to version 3.1.2 for now.

This could be related with #39

I have read https://redis.io/topics/distlock , still not clear what kind of locking it is. Could u shed me some light? Is it read lock/write lock or absolute lock where other party can do nth but wait?

I need to handle my queue items differently depending on which type of error is thrown. For example, if I get a retry error when lock exists (#12), I'd just like to remove the queue item completely. Currently I'm just matching on the error string (https://github.com/mike-marcacci/node-redlock/blob/master/redlock.js#L309), but it would be great if each error had a different name or code associated w/ it so I could be more explicit.

If that sounds good I can PR, but wondering if you have thoughts on how they should be distinguished (name vs. code, type of code, etc.)

The documentation seems to imply that extend() will extend the current timeout by X ms. So I'd think this would extend the lock to 15sec:

redlock.lock('locks:account:322456', 5000, function(err, lock) {
  lock.extend(5000, function(err, lock){
    lock.extend(5000, function(err, lock){
      // 15 secs or 5 secs?
    });
  });
});

But extend() appears to only overwrite the ttl, so it's only 5sec in the above example. This should be clarified in the docs, or extend() should be renamed ttl() since it's just resetting the ttl to a new value.

Hello.
Briefly - how would I detect situation where I cannot get lock because of all Redis instanes gone (eg. db connection error on all instances).

More details:

try {
    lock = await redlock.lock("x", 1000);
} catch (err) {
    // Db connection problem (what I want to log and response with appropriate error code/message)
    //   OR
    // Alredy locked by other instance of my app and I want to ignore it (eg. I will respond with 202 code to signal that task already in progress)
}

Is there a way I can distinguish between type of errors, eg:

1. Unable to acquire/extend lock because of db connection error.
2. Unable to acquire lock because resource already locked.
3. Unable to extend lock because it already expired.

Is it possible that you will introduce LockError.code property and expose (export) some LOCK_ERROR_CODE_... consts?

Regards,
Wojtek

var redisClient = redis.createClient();
    var redlock = new Redlock({
        driftFactor: 0.01,
        retryCount: 225,
        retryDelay: 250
    }, redisClient);

redlock.lock(LOCK_PUBLIC_LIST, LOCK_TIME, function(error, lock) {
    if (error) {
        console.log(error);
    }
});

Will produce the error: TypeError: undefined is not a function

If create client is made with a public facing IPv4 and port, it works every time

Is there a way to check if any key is locked at the moment?

Something like

redlock.locked(key).then(results => {
    console.log(result); //true -> locked, false -> not locked
})

Thank you for this awesome module!

Is there a way to unlock / close a lock you don't have a reference to anymore but have the resource name.
For instance, what if lock a resource for a long time (15 minutes like this):

function test(resource){
  redlock.lock(resource, 900000).then(function(lock) {
    // ...do something here...
    // but no unlock
  });
}

Then later on, how could I unlock the resource? My use case is that I have conditions to check one at a time and locking them one-by-one. If all conditions are fulfilled, I have to unlock all the gathered locks. I would prefer to avoid accumulating references to the locks as my iterations are executed recursively 😕 .

I've 2 web-servers witha load balancer. I have a cron which I want to run on only one server. Can I use redlock for this?

When I used redlock like this:

var resource = 'redisKey';
function getData(){

    redlock.lock(resource, ttl).then(function(lock) {
                  // logic of function
            });
     return lock.unlock()
        .catch(function(err) {
                   console.error(err);
     });
  });

};

Here, my getData function(which is part of cron for which I need lock) is working fine, but I'm not able to check if my redisKey gets SET or not. I tried commenting lock.unlock() but couldn't test it.
Any suggestions how can I test if my key it set by using redlock this way?

Hi,

Great work !

What do you think about adding an optional argument to lock function instead of setting a random value like :
const lock = await redlock.lock('lock:123', 'theGuyOrServiceLocking', 1000);

It could be useful in order to log/send an error message indicating which account is locking resource for example.

Thanks,

There is an option string_numbers in Redis client (ref: https://github.com/NodeRedis/node_redis#options-object-properties) to return number values as strings.

Node-redlock is incompatible with this option because it expects the response to be always a number: https://github.com/mike-marcacci/node-redlock/blob/master/redlock.js#L203. In result, setting string_numbers to true causes an error in Redlock.unlock(): LockError: Unable to fully release the lock on resource.

I've just read about the redlock algorithm and can't figure out is it a correct case of usage. Let's say I have a few workers and an observable which spies on some database. I use timeout to randomly pick a worker:

const redlock = new Redlock([...], { retryCount: 0 });

...

sub.on('message', (channel, message) => setTimeout(async () => {
  try {
    var lock = await redlock.lock('db1' /* use parsed message in the real app */, 1000);
  } catch (e) {
    console.log(e);
  }

  if (lock) {
    const watcher = new DatabaseObservable('db1').subscribe(
      message => {
        console.log(m); // something has changed in the database
      },

      err => {
        lock.unlock(); // the connection to the database was interrupted
      },

      () => {
        lock.unlock(); // the connection to the database was closed normally 
      }
    );

    try {
      while (1) {
        await sleep(800); // hmm... the lock here could become dead
        lock = await lock.extend(1000);
      }
    } catch (e) { ... }
  }
}, Math.random()));

sub.subscribe('watch')

When I need to create a new database observable I do redis.publish('watch', 'db1'). For each database there can be only one observable at the same time. The lock unlocks once the connection to the database is interrupted or closed.

1. Does Redlock fit my case?
2. What is the right way here to extend locks?
3. Connections to the databases could be interrupted. So I want to recover them. Can I set an interval which repeats Redis.publish('watch', 'db1') unless the key is locked? Something like:

// in the server process around a websocket connection

setInterval(() => {
  if ( /* unlocked */) {
    redis.publish('watch', 'db1')
  }
}, 1000);

Would be nice to have a quit or close method to ensure clients used exclusively for redlock are closed when done.

message:
'ERR Error running script (call to f_862f31bf38e9b63ce28d6e9e5143551bb34ef981): Wrong number of args calling Redis command From Lua script ',
command:
{ name: 'eval',
args:
[ 'return redis.call("set", KEYS[1], ARGV[1], "NX", "PX", ARGV[2])',
'1',
'shop_wv_test1',
'f9a2445c012a1d2f2f38291a434c0731',
'3000' ] } } undefined
2019-03-14 15:11:18,217 ERROR 1370 nodejs.LockError: Exceeded 10 attempts to lock the resource "shop_wv_test1".
at /Users/wv/work/qianmi/api/shop/packages/shop-bff/node_modules/redlock/redlock.js:343:20
at runCallback (timers.js:694:18)
at tryOnImmediate (timers.js:665:5)
at processImmediate (timers.js:647:5)
From previous event:
at Redlock._lock (/Users/wv/work/qianmi/api/shop/packages/shop-bff/node_modules/redlock/redlock.js:282:9)
at Redlock.lock (/Users/wv/work/qianmi/api/shop/packages/shop-bff/node_modules/redlock/redlock.js:139:14)
at Object.task (/Users/wv/work/qianmi/api/shop/packages/shop-bff/app/schedule/goods-name-index-all.ts:47:23)
name: "LockError"
message: "Exceeded 10 attempts to lock the resource "shop_wv_test1"."

I just had an issue in which we had accidentally pointed to a specific node instead of the replica set URL, and the replica set changed the primary, so we started getting ERROR: READ-ONLY errors. However the errors we saw from redlock were just Exceeded X attempts to lock...

It would be nice to have a debug mode or some other method of logging for if err is not null in the loop function. I'm happy to submit a PR - what do you think is the best approach?

node-relock version: 4.1.0
ioredis version: 4.14.1

Hi, I'm trying to use node-redlock with redis-cluster. When locking multi resources, I got a "LockError" exception. It exceeds the 10 attempts to lock.

Here's my code

  let client = new redis.Cluster([
        {
            ip: '127.0.0.1',
            port: '7001'
        },
        {
            ip: '127.0.0.1',
            port: '7002'
        },
        {
            ip: '127.0.0.1',
            port: '7003'
        }
    ])
    let lk = new redlock([client])
    let l = null
    l = await lk.lock(['lk1', 'lk2'], 1000) //failure
    //l = await lk.lock(['lk'], 1000)  //success
    let res = await client.get('foo')

(node:28707) UnhandledPromiseRejectionWarning: LockError: Exceeded 10 attempts to lock the resource "lk1,lk2".
    at /home/sx/projects/nodejs/njproj1/node_modules/redlock/redlock.js:411:20
    at tryCatcher (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/util.js:16:23)
    at Promise.errorAdapter [as _rejectionHandler0] (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/nodeify.js:35:34)
    at Promise._settlePromise (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/promise.js:601:21)
    at Promise._settlePromise0 (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/promise.js:725:18)
    at _drainQueueStep (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues (/home/sx/projects/nodejs/njproj1/node_modules/bluebird/js/release/async.js:15:14)
    at runCallback (timers.js:794:20)
    at tryOnImmediate (timers.js:752:5)
    at processImmediate [as _immediateCallback] (timers.js:729:5)

Hi, thanks for making this available. I have a question regarding this comment in the configuration section of README.

// you should have one client for each redis node
// in your cluster

I am running a 9 node redis cluster with 3 shards and each shard supported by one primary and 2 slaves hosted in Amazon Elastic cache. For the normal use of Redis in my node based app cluster I just use a single load balancer provided DNS name to access the cluster for various redis operations. I do have redis-cli access to each of the individual nodes and therefore able to supply individual clients for redlock.

To use this cluster with redlock, do I need to create clients for all 9 nodes or is it OK to just supply a separate client for each of the 3 shard masters (or any other 3 nodes).

1. How to list the active locks? Tried "KEYS *" and it doesn't return the lock.

2. Just in case the lock is locked with a very big TTL and node crashed without unlock. Is it possible to unlock it via cli?

Thanks!