jaredwray / keyv Goto Github PK

We've had quite a few issue asking for requests that are outside the scope of Keyv (#28, #34, #12). We should add a section in the readme that clarifies that Keyv is only a storage adapter, extra functionality should be built on top of Keyv.

Also add a section to the readme showcasing packages built on Keyv:

• https://github.com/Moeriki/memoized-keyv
• https://github.com/lukechilds/cacheable-request
• Got
• Rill
• Others?

I'm gonna code a new storage adapter for levelDB, using LevelUP. 👍

Running npm audit gave me this:

  Moderate        SQL Injection
  Package         sql
  Patched in      No patch available
  Dependency of   @keyv/mysql [dev]
  Path            @keyv/mysql > @keyv/sql > sql
  More info       https://nodesecurity.io/advisories/662

  Moderate        SQL Injection
  Package         sql
  Patched in      No patch available
  Dependency of   @keyv/postgres [dev]
  Path            @keyv/postgres > @keyv/sql > sql
  More info       https://nodesecurity.io/advisories/662

  Moderate        SQL Injection
  Package         sql
  Patched in      No patch available
  Dependency of   @keyv/sqlite [dev]
  Path            @keyv/sqlite > @keyv/sql > sql
  More info       https://nodesecurity.io/advisories/662

  Low             Prototype Pollution
  Package         lodash
  Patched in      >=4.17.5
  Dependency of   @keyv/mysql [dev]
  Path            @keyv/mysql > @keyv/sql > sql > lodash
  More info       https://nodesecurity.io/advisories/577

  Low             Prototype Pollution
  Package         lodash
  Patched in      >=4.17.5
  Dependency of   @keyv/postgres [dev]
  Path            @keyv/postgres > @keyv/sql > sql > lodash
  More info       https://nodesecurity.io/advisories/577

  Low             Prototype Pollution
  Package         lodash
  Patched in      >=4.17.5
  Dependency of   @keyv/sqlite [dev]
  Path            @keyv/sqlite > @keyv/sql > sql > lodash
  More info       https://nodesecurity.io/advisories/577

Should we be worried about this?

Hi,

Great work with keyv!

Do you mind if I help you rewrite it to typescript?

Also I believe support for custom serializers would be really beneficial to serialize such objects like Maps, Sets or any custom class.
What do you think?

The idea is provide more than one layer support as fallback.

Similar to cache L1 and L2 approach.
https://people.csail.mit.edu/sanchez/papers/2016.model.hpca.pdf

Just an example aproximation using redis and memory implementation.

get operations:

• first try to get the value from redis instance.
• it it fails, try to find the value in the memory implementation.
• under memory hit, move the key to redis.

set operations:

• if redis have a free slot, use it.
• if redis doesn't have a free slot but the slot it's available on memory, use it.
• otherwise, remove the oldest reference entry at memory implementation.

Version 1.0.0 of keyv-sqlite just got published.

This version is covered by your current version range and after updating it in your project the build failed.

As keyv-sqlite is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this 💪

Your Greenkeeper Bot 🌴

Version 1.0.2 of keyv-mongo just got published.

This version is covered by your current version range and after updating it in your project the build failed.

As keyv-mongo is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this 💪

Your Greenkeeper Bot 🌴

Pass an array of keys to be deleted in a single operation.

await keyv.delete(['key-1', 'key-2', 'key-3']);

Would be nice to have for performant deletion of multiple keys. However not sure this is gonna be feasible to implement without breaking Map compatibility.

ideal for avoid use mongo/redis on development, just plain json for temporal data.

Test storage adapters allow process to exit.

Otherwise the process may hang if the storage adapter keeps a connection open.

We should add something like this in the constructor:

process.on('exit', () => {
    this.opts.store.closeConnection && this.opts.store.closeConnection();
});

Related: https://github.com/lukechilds/keyv-redis/issues/12

Version 10 of Node.js (code name Dubnium) has been released! 🎊

To see what happens to your code in Node.js 10, Greenkeeper has created a branch with the following changes:

• Added the new Node.js version to your .travis.yml

If you’re interested in upgrading this repo to Node.js 10, you can open a PR with these changes. Please note that this issue is just intended as a friendly reminder and the PR as a possible starting point for getting your code running on Node.js 10.

Your Greenkeeper Bot 🌴

Trying to use keyv and @keyv/mongo works with npm (duh), pnpm, but not if the project is a pnpm workspace. The error is "Error: Cannot find module '@keyv/mongo'".

Repro repo at https://github.com/dandv/pnpm-monorepo-keyv

Any idea what might cause this?

(this goes for both Keyv and cacheable-request as they're both heavily related)

Keyv got way more popular than I imagined it would. 3 million downloads a month on npm and according to GitHub insights, over 60k repos depend on Keyv 😬.

It was literally just built as part of a cache PR for Got, but my plans for it grew and now it seems like lots of people are using it for lots of different things.

Unfortunately this means with this amount of users, the issues/PRs come in at a rate quicker than I can deal with them.

I’ve also had an extremely busy year with paid work (which unfortunately takes priority over OSS so I can afford to live). On top of that, I’ve had a few personal/relationship/financial/mental health issues that I’ve been dealing with this year too. As a consequence, OSS has been pushed on the back burner a bit. I haven’t invested anywhere near the amount of time I’d of liked to into OSS (especially Keyv) this year.

I keep telling myself I wrote this for free and I don’t owe anyone anything, but people have written products they depend on to earn a living based on this code so I do feel obligated to try and keep this project well maintained, which I feel I’m failing at.

There have also been contributors who have been very understanding of my lack of response (which I really appreciate) and some of these open PRs are 99% good to to be merged and just need a tiny bit of feedback which I just haven’t found the time to do. So I feel pretty bad about that. Sorry guys, you know who you are.

Anyway, the purpose of this issue was a call for help to see if anyone is interested in helping me maintain Keyv. I’ve seen a few new faces recently that have made some pretty high-quality contributions.

Just to mention some: @sindresorhus @szmarczak @roccomuso @e0ipso @UltCombo @FabricioMatteMassive @dandv @kornelski

Thanks a lot for your help, it's been greatly appreciated. If any of you (or anyone else reading this) is interested in becoming a maintainer of Keyv, let me know and I'll send you an invite.

Just to clarify I'm not abandoning Keyv/cacheable-request or any of my other OSS projects. I'm just struggling to keep up with the maintenance overhead alongside IRL stuff and these are the two most demanding.

I've still got a todo list of about a gazillion things I want to improve with Keyv, and fully intend to do them. I just have bigger priorities I need to fit it around right now.

So yeah, thanks for all the contributions you guys have made so far. If you're interested in becoming a maintainer, hit me up. If not, time is precious, I totally understand.

Cheers,
Luke ✌️

Hi,

as keyv is a very strong proposition for implementing a cache, are there any plans to :

• support setting a maximum size, which would trigger eviction of oldest keys
• support a simple cleanup of expired keys

or may be those have been ruled for simplicity ? in that case, it feels like at least a way to iterate on keys would allow for a wrapper to implement periodic cleanup. If mimicking Map, this could mean implementing keys() as returning an Iterable

The trouble being it needs to be implmented in all stores.

Anyway, your work is very elegant and promising.

Similar to #41 but with .get() instead of .delete()

Pass an array of keys to get in a single operation.

await keyv.get(['key-1', 'key-2', 'key-3']);

Add .keys(), .values() .entries() methods similar to the Map spec but instead they should return a Promise that resolves to an iterable.

Something like:

const keyv = new Keyv();
await keyv.set('foo', 'bar');
await keyv.set('fizz', 'buzz');

const keys = await keyv.keys();
for (const key of keys) {
  console.log(key);
}
// 'foo'
// 'fizz'

const values = await keyv.values();
for (const value of values) {
  console.log(value);
}
// 'bar'
// 'buzz'

const entries = await keyv.entries();
for (const [key, value] of entries) {
  console.log(`${key} => ${value}`);
}
// 'foo => bar'
// 'fizz => buzz'

I would want to use this library in tools that run in both browser and server.

As far as I can tell, there are only two things blocking this:

1. This line https://github.com/lukechilds/keyv/blob/de2e7a2ffc4a5007569b62a31956d7b692a011e7/src/index.js#L18
2. The dependence on node's events

Replacing events doesn't have to be a breaking change - the full EventEmitter polyfills are pretty large, but eventemitter3 isn't too bad. I am partial to better-emitter, which is even smaller, but does have a different API.

Dropping the dynamic require would have to be a breaking change, but I think it would be a good step - I think having to maintain a list of adaptor libraries isn't worth the complexity. I have more trust in an API that expects me to instantiate and pass in my adaptor.

I like the simplicity of this library, thanks for making it 👍

I just created a project to add DynamoDB as a store. I'd like to have official support for it. I noticed that the Keyv factory has stores hard-coded. Would you consider granting "official" status to https://github.com/e0ipso/keyv-dynamodb?

I would also (#48) like the ability to not use a namespace. The use case is easier debugging without having to mentally parse out the keyv: prefix from every value. Plus, with MongoDB, collections are a cleaner and simpler way to achieve namespacing.

There is discussion in several issues about providing some ACID guarantees when iterating over a store's keys/values/entries and concurrently modifying them:

• #40
• #43
• #74

I thought it'd be better to have a discussion for these in one place, rather than scattered across multiple issues.

Quick note on terminology: I think the main concern in the iterables discussions is isolation:

Isolation ensures that concurrent execution of transactions leaves the database in the same state that would have been obtained if the transactions were executed sequentially.

Though the other properties are desirable, I think they're out of scope here as they're more likely to be the responsibility of the backend, whereas I believe Keyv itself can at least provide a fallback implementation of isolation for backends that don't provide it natively.

There are several different ways to handle this, including not handling it (i.e. let the user pick up the pieces) and not allowing it. I'd like to propose 2 solutions:

1. don't enforce isolation by default i.e. in situations where it's not needed
2. provide a way to ensure isolation in situations where it's desired

I believe not providing any isolation guarantees (1) should be the default since the issue only arises in cases where the store is being mutated while it's being iterated. Client code which is not mutating the store shouldn't need to do anything special to iterate over its contents.

For 2), one solution could be a transactional interface i.e. a block (function) in which iteration and mutations are performed. The mutations are deferred until the block is complete and then committed e.g.:

keyv.begin(async (it) => {
    for await (key of it.keys()) {
        if (key === 'foo') {
            it.delete(key) // deferred
        }
    }
})

This is easy enough to implement with a Proxy e.g.:

function deferMutations (keyv, enqueue) {
    const mutators = {
        clear () {
            enqueue('clear')
            return Promise.resolve(true)
        },

        delete (key) {
            enqueue('delete', key)
            return keyv.has(key)
        },

        set (key, value) {
            enqueue('set', key, value)
            return Promise.resolve(true)
        }
    }

    return new Proxy(keyv, {
        get (target, key, receiver) {
            return mutators[key] || Reflect.get(target, key, receiver)
        }
    })
}

class Keyv {
    begin (block) {
        const keyv = this
        const queue = []

        let active = true

        const enqueue = (op, ...args) => {
            if (active) {
                queue.push(() => keyv[op](...args))
            } else {
                throw new InactiveMutatorError(op, ...)
            }
        }

        // defer modifications until after the block has completed
        const delegate = deferMutations(keyv, enqueue)

        return block(delegate).then(async (commit) => {
            active = false

            if (commit !== false) {
                for (const deferred of queue) {
                    await deferred()
                }
            }
        })
    }
}

Note: the block can abort the transaction by explicitly returning false:

keyv.begin(async (it) => {
    for await (key of it.keys()) {
        if (key === 'foo') {
            it.delete(key)
        } else if (key === 'bar') {
            return false // rollback
        }
    }
})

Transactions can also be nested:

keyv.begin(async (it) => {
    for await (key of it.keys()) {
        if (key === 'foo') {
            it.begin(async (nested) => {
                nested.set(...)
            })
        }
    }
})

Hi,
I'm writing the storage adapter for Etcd. Travis doesn't have an etcd-service available.
How would you manage to test it in CI?

I'd like to propose a solution for a common pattern. At least for me.

Usage

const cachedRequest = keyv.memoize(request);

cachedRequest('http://example.com').then(() => { /* result */ });
// later
cachedRequest('http://example.com').then(() => { /* cached result */ });

Implementation

/**
 * @param {function}        func
 * @param {object}          [opts]
 * @param {function}        [opts.resolver=identity]
 * @param {function|number} [opts.ttl]
 * @return {Promise}
 */
Keyv.prototype.memoize = function memoize(func, { resolver, ttl }) {
  return (...args) => {
    const key = resolver ? resolver(...args) : args[0];
    return this.get(key).then((storedValue) => {
      if (storedValue !== undefined) {
        return storedValue;
      }
      return Promise.resolve(func(...args)).then((value) => {
        if (value !== undefined) {
          return this.set(key, value, typeof ttl === 'function' ? ttl(value) : value)
            .then(() => value)
          ;
        }
        return value;
      });
    });
  };
};

Something like:

const keyv = new Keyv();
await keyv.set('foo', 'bar');

await keyv.get('foo');
// 'bar'
await keyv.get('foo', { raw: true });
// { value: 'bar', expires: null }

Hi!

We are planning to use keyv in metro as a multi-layered cache system. The goal is to let people mix and match different cache layers, by using the Map compatible interface (something we wanted even before finding keyv!).

In order to be layered, we need to be able to broadcast key/value combinations from one layer to the rest (essentially, a successful get in one layer has to be set in all the rest of the layers). However, keyv only supports one single adapter per instance.

While the desired approach can be done either as an adapter that proxies to multiple stores, or as a layer built on top of many keyv instances, I was curious to know if you would like to have such behavior built inside keyv. The current API would not change, but it would allow to pass multiple stores (my idea is to add stores, and keep store as backward-compatible, which would just re-wrap as stores = [store]). Stores would then sequentially be called until we find a positive result.

I'd also like to add a @keyv/memcached, as well as a @keyv/map (which essentially would be module.exports = Map, but it would let combine as well the memory interface by default). For all of that maybe it'd make sense to have a GH keyv organization itself (just suggesting).

We'd like also add Flow typing to it (I've seen #22 for TS support), so I guess that one would be OK as well :)

Let me know what you think! 🙂

Version 1.0.0 of keyv-mongo just got published.

This version is covered by your current version range and after updating it in your project the build failed.

As keyv-mongo is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this 💪

Your Greenkeeper Bot 🌴

If Keyv (and official storage adapters) were to expose more internals, this would allow for much richer functionality to be built on top of Keyv.

Mainly:

• Optionally return the raw DB object (#39)
• Get all keys method (#40)
• Delete multiple keys in a single operation (#41)
• Get multiple keys in a single operation (#78)
• Check if key exists (#50 )

Main thing that comes to mind is a DB migrator (needs access to expiry dates) and keyv-shrink which is currently a fork because the desired functionality couldn't be implemented on top of Keyv.

With the three listed options implemented, keyv-shrink could be implemented efficiently on top of Keyv. All updates to Keyv and Keyv storage adapters would automatically work with keyv-shrink. Example of an implementation could be something like this:

const Keyv = require('keyv');

class KeyvShrink extends Keyv {
  shrink() {
    ...
  }
}

@MySidesTheyAreGone What are your thoughts on this? I'm tied up on client project right now but I'm interested in adding this soon.

Great list of modules here. I think it would be a good idea to switch to scoped modules with a public org.

There are several benefits here:

• There is no chance that someone could steal a package name.
• Users can more trust just packages without worrying about squaters/malware.
• You can more easily give other developers access/control of individual repo's and manage it all from one org.

Version 1.6.1 of @keyv/test-suite just got published.

This version is covered by your current version range and after updating it in your project the build failed.

As @keyv/test-suite is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this 💪

Your Greenkeeper Bot 🌴

Currently JSONB.stringify/JSONB.parse is used for serializing/deserializing to support a wide range JavaScript Types.

Allowing users to set a custom a custom serializer function will allow for cleaner usage with custom data structures and enable performance improvements for people who don't need all the type support JSONB provides.

Syntax should look something like:

const Keyv = require('keyv');
const { serialize, deserialize } = require('my-serializer');

const keyv = new Keyv({ serialize, deserialize });

Previous discussion in #22

Both mongodb and redis support a way increment variables directly. It would be nice to have something similar here. Specifically I am wanting to use keyv to implement a rate limiter but without increment it's impossible to do so.

May be out of scope for the module but I thought it'd be cool.

Also this module is legit. Thanks for making it :).

Is there a adapter that uses the build in Node-IPC for storing data in memory.
I'm searching for something like memored but with namespaces.

Am I missing something or there is no .has() api method? (similar to Map)

What I need is to distinguish between situation when there's no key in cache (has would return false) or there is a key in cache, but it's value is undefined. If I use .get(key) for this - both will return undefined.

Sorry for the awkward title.

If I get something and then manipulate it and the storage is a simple in-memory Map, could the manipulation (e.g. changing an object's property) affect the cached value?

Thanks!

Hey @lukechilds, thanks for spend yout time creating this, it's awesome!

As crazy idea, I'm thinking if could be possible support a p2p connector.

Specifically I'm thinking into:

• https://github.com/amark/gun
• https://github.com/mafintosh/hyperdb

But not sure if exists better standard libraries, maybe I'm missing something.

What do you think?

The default store is new Map(). It has no(?) TTL support. But we could do something like:

if (store instanceof Map) {
	store.timers = {};
	store._set = store.set;
	store.set = (key, value, ttl) => {
		clearTimeout(store.timers[key]);
		store.set(key, value);
		store.timers[key] = setTimeout(() => {
			delete store.timers[key];
			store.delete(key);
		}, ttl);
	};
}

Will enable autocompletion in supported text editors/IDEs.

Previous discussion in #22

To prevent definitions going out of sync we should either include some sort of test to validate the definitions or alternatively they be generated from our source (annotated comments?).

Version 1.6.0 of @keyv/test-suite just got published.

This version is covered by your current version range and after updating it in your project the build failed.

As @keyv/test-suite is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this 💪

Your Greenkeeper Bot 🌴

Would this be within scope?

I'm importing https://github.com/sindresorhus/got/ into my project and using webpack to build. Webpack doesn't like requiring of expressions:

WARNING in ./node_modules/keyv/src/index.js
18:14-40 Critical dependency: the request of a dependency is an expression
 @ ./node_modules/keyv/src/index.js
 @ ./node_modules/cacheable-request/src/index.js
 @ ./node_modules/got/index.js
 @ ./lib/util.js
 @ ./functions/live-stream.js

I'm not sure what the best way to fix this up with would be

Is there a way to ask the adapter to create the table if it doesn't exist?

I'm have plan to custom Keyv, let me expalain. I'm sure that Keyv is flexible and fast key value store. Let support Keyv with create, authenticate, delete, backup, restore and migrate database. Once can add, update, and remove user. In Mongodb using collection. They can create, insert, update, read, find spesific data, rename, and delete collection. Let support it in Keyv and customize to get better performance like etcd.

Thanks

Version 1.0.1 of keyv-mongo just got published.

This version is covered by your current version range and after updating it in your project the build failed.

As keyv-mongo is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this 💪

Your Greenkeeper Bot 🌴

Version 1.3.8 of @keyv/redis just got published.

This version is covered by your current version range and after updating it in your project the build failed.

As @keyv/redis is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this 💪

Your Greenkeeper Bot 🌴

Any chances of using mysql.createPool for the DB connection?

When we use keyv-mongo for cache backend for cacheable-request, MongoDB does not allow files bigger than 16 MB.

Can I implement mongodb gridfs to achieve the same? @lukechilds any thoughts about how should the API be?

In the error handling: Please add a "," in
keyv.on('error', err => {...});

When having an import/export feature, it would be easy to migrate from one database to another.

Hi, any particular reasons why mongojs was preferred to the official MongoDB driver for Node? I'd like to take a stab at fixing the failing tests and the exit issue. I've worked with the official driver package, but not with mongojs.

I find myself needing an all function for uses like a leaderboard for my users, right now I'm having to bypass Keyv and select from sqlite manually, which for a database wrapper isn't appreciated.

Can we get <Keyv>.all() added? Or possibly some kind of filtering too?

Version 1.0.0 of keyv-postgres just got published.

This version is covered by your current version range and after updating it in your project the build failed.

As keyv-postgres is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this 💪

Your Greenkeeper Bot 🌴