invertase / cluster-key-slot Goto Github PK

View Code? Open in Web Editor NEW

17.0 11.0 11.0 52 KB

Generates CRC hashes for strings - for use by Node Redis clients to determine key slots.

License: Other

JavaScript 100.00%

cluster-key-slot's Introduction

Redis Key Slot Calculator

A high performance redis cluster key slot calculator for node redis clients e.g. node_redis, ioredis and redis-clustr.

This also handles key tags such as somekey{actualTag}.

Install

Install with NPM:

npm install cluster-key-slot --save

Usage

const calculateSlot = require('cluster-key-slot');
const calculateMultipleSlots = require('cluster-key-slot').generateMulti;

// ...

// a single slot number
const slot = calculateSlot('test:key:{butOnlyThis}redis');
// Buffer is also supported
const anotherSlot = calculateSlot(Buffer.from([0x7b, 0x7d, 0x2a]));

// multiple keys - multi returns a single key slot number, returns -1 if any
// of the keys does not match the base slot number (base is defaulted to first keys slot)
// This is useful to quickly determine a singe slot for multi keys operations.
const slotForRedisMulti = calculateMultipleSlots([
  'test:key:{butOnlyThis}redis',
  'something:key45:{butOnlyThis}hello',
  'example:key46:{butOnlyThis}foobar',
]);

Benchmarks

OLD in these benchmarks refers to the ioredis crc calc and many of the other calculators that use Buffer.

node -v                                                                                                                                                                                                ✔  16.38G RAM  10:29:07
v10.15.3

NEW tags x 721,445 ops/sec ±0.44% (90 runs sampled)
OLD tags x 566,777 ops/sec ±0.97% (96 runs sampled)
NEW without tags x 2,054,845 ops/sec ±1.77% (92 runs sampled)
OLD without tags x 865,839 ops/sec ±0.43% (96 runs sampled)
NEW without tags singular x 6,354,097 ops/sec ±1.25% (94 runs sampled)
OLD without tags singular x 4,012,250 ops/sec ±0.96% (94 runs sampled)
NEW tags (Buffer) x 552,346 ops/sec ±1.35% (92 runs sampled)

cluster-key-slot's People

Contributors

Stargazers

Watchers

Forkers

wthsths noderedis global-localhost global19 global19-atlassian-net isabella232 adedayoominiyi uki00a vitaly-z streamich sonthanhdan

cluster-key-slot's Issues

Add typescript types

Hi,

Its difficult to use this module in a TypeScript project as it has no typings available.

Would you consider adding some?

TS2309 error on index.d.ts

To address the ES Lint rule @typescript-eslint/no-var-requires, I updated my usage of cluster-key-slot:

- const calculateSlot = require('cluster-key-slot');
+ import calculateSlot from 'cluster-key-slot';

tsc complains the changed coded has TS2309 error:

> tsc 
node_modules/cluster-key-slot/index.d.ts:9:3 - error TS2309: An export assignment cannot be used in a module with other exported elements.

9   export = calculate;
    ~~~~~~~~~~~~~~~~~~~

My tsconfig.json:

{
  "compilerOptions": {
    "module": "commonjs",
    "target": "es6",
    "allowJs": true,
    "noImplicitAny": true,
    "esModuleInterop": true,
    "strict": true,
    "moduleResolution": "node",
    "sourceMap": true,
    "noEmit": true,
    "experimentalDecorators": true,
    "jsx": "preserve"
  },
  "files": ["node_modules/jest-expect-message/types/index.d.ts"],
  "include": ["src"]
}

"cluster-key-slot": "1.1.2"
"typescript: "5.0.4"

Sorry for not having a minimum reproduction.

Question :will this repo be invluded in invertase swag

Disagreement between redis an cluster-key-slot on keyslot for fancy character, patch included

The character in question is:

http://www.fileformat.info/info/unicode/char/1f480/index.htm

I have run into other strange characters that are a problem as well. It looks like the string to utf8 array conversion is not working correctly somewhere. redis calculated a different hash slot and returns MOVED 9284, the client tries again at the wrong slot still. . . this repeats until the client gives up.

$ cat slot-bug.js 
var Redis = require('ioredis');
var cluster = new Redis.Cluster([{
    port: 30001,
    host: '127.0.0.1'
}]);
cluster.set('\uD83D\uDC80', 'SKULL');
cluster.get('\uD83D\uDC80', function(err, res) {
    if (err) {
        console.log('failed to get result');
    } else {
        console.log('Got result: ' + res);
    }
    cluster.disconnect();
});

$ node slot-bug.js 
failed to get result
Unhandled rejection Error: Too many Cluster redirections. Last error: ReplyError: MOVED 9284 127.0.0.1:30002
    at Cluster.handleError (/home/dkimdon/temp/slot-bug/node_modules/ioredis/lib/cluster/index.js:554:30)
    at Command.command.reject (/home/dkimdon/temp/slot-bug/node_modules/ioredis/lib/cluster/index.js:444:13)
    at Redis.exports.returnError (/home/dkimdon/temp/slot-bug/node_modules/ioredis/lib/redis/parser.js:75:18)
    at JavascriptReplyParser.returnError (/home/dkimdon/temp/slot-bug/node_modules/ioredis/lib/redis/parser.js:25:13)
    at JavascriptReplyParser.run (/home/dkimdon/temp/slot-bug/node_modules/redis-parser/lib/javascript.js:135:18)
    at JavascriptReplyParser.execute (/home/dkimdon/temp/slot-bug/node_modules/redis-parser/lib/javascript.js:112:10)
    at Socket.<anonymous> (/home/dkimdon/temp/slot-bug/node_modules/ioredis/lib/redis/event_handler.js:107:22)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at readableAddChunk (_stream_readable.js:176:18)
    at Socket.Readable.push (_stream_readable.js:134:10)
    at TCP.onread (net.js:548:20)

This fixes it for me, I didn't see where to change toUTF8Array() so I used the native node API:

$ cat  fix-keyslot.patch 
--- ./node_modules/cluster-key-slot/lib/index.js.orig	2017-03-30 22:52:59.403073650 -0700
+++ ./node_modules/cluster-key-slot/lib/index.js	2017-03-30 22:55:48.311166599 -0700
@@ -130,7 +130,7 @@
   var start = -1;
   var result = 0;
   var resultHash = 0;
-  var utf8 = toUTF8Array(str);
+  var utf8 = Buffer.from(str);
   var len = utf8.length;
 
   while (i < len) {
$ patch -p0  < fix-keyslot.patch 
patching file ./node_modules/cluster-key-slot/lib/index.js
$ node slot-bug.js 
Got result: SKULL
$

Looks like redis thinks the slot should be 9284:

127.0.0.1:30001> cluster keyslot 💀
(integer) 9284
127.0.0.1:30001>

Adding this to the cluster-key-slot tests:

$ git diff
diff --git a/test/hash.spec.js b/test/hash.spec.js
index 9d07289..7c18291 100644
--- a/test/hash.spec.js
+++ b/test/hash.spec.js
@@ -21,6 +21,7 @@ var tests = {
   '漢字': 14191,
   '汉字': 16196,
   '호텔': 4350,
+  '\uD83D\uDC80': 9284, // SKULL, UTF-8 (hex) 0xF0 0x9F 0x92 0x80 (f09f9280)
   '\uD800\uDC00': 11620 // surrogate pair
 };

Unpatched version of cluster-key-slot fails the test:

$ npm test
 1) single hash: generate() generate a correct hash from string:

     AssertionError: 💀 - generated invalid hash: 550
     + expected - actual

     -550
     +9284
     
     at assertHash (test/hash.spec.js:42:10)
     at Array.forEach (native)
     at Context.<anonymous> (test/hash.spec.js:47:24)
. . .

Patched version works:

$ patch ./lib/index.js < ~/temp/slot-bug/fix-keyslot.patch 
patching file ./lib/index.js
$ npm test

> [email protected] test /home/dkimdon/src/cluster-key-slot
> node ./node_modules/istanbul/lib/cli.js cover --preserve-comments ./node_modules/mocha/bin/_mocha -- -R spec



  single hash: generate()
    ✓ generate a correct hash from string

  multiple hashes: generateMulti()
    ✓ generate a correct hash from multiple strings
    ✓ returns -1 if any of the keys generates a different hash slot than the rest


  3 passing (7ms)

. . .

io.redis example

Hello,

I'm sorry to ask but how would you use it with io.redis ? I'm not sure to understand how to implement it .

THanks

tsc fails with an error

when using tsc and cluster-key-slot as a dependency the type check fails with an error

Error: node_modules/cluster-key-slot/index.d.ts(9,3): error TS2309: An export assignment cannot be used in a module with other exported elements.

any fix for that?

Invalid SPDX identifier for license

APACHE-2.0 is not a valid SPDX license identifier as it is all upper-case. Apache-2.0 would be correct. Our CI/CD pipelines are failing because of this.

I can see that the fix is actually merged into master branch, but not released. Will there be a patch release soon?