Git Product home page Git Product logo

community's Introduction

community

Discussion, support and common information for projects in the community.

level badge Test Donate

Table of Contents

Click to expand

FAQ

What is Level?

Level is a collection of Node.js modules for creating transparent databases. A solid set of primitives enable powerful databases to be built in userland. They can be embedded or networked, persistent or transient - in short, tailored to your needs.

At the heart of Level are key-value databases that follow the characteristics of LevelDB. They support binary keys and values, batched atomic writes and bi-directional iterators that read from a snapshot in time. Entries are sorted lexicographically by keys which, when combined with ranged iterators, makes for a powerful query mechanism. Level combines idiomatic JavaScript interfaces like async iterators with Node.js interfaces like streams, events and buffers. It offers a rich set of data types through encodings and can split a database into evented sections called sublevels.

The underlying storage can be easily swapped by selecting a different database implementation, all sharing a common API and the same characteristics. Together they target a wide range of runtime environments: Node.js and Electron on Linux, Mac OS, Windows and FreeBSD, including ARM platforms like Raspberry Pi and Android, as well as Chrome, Firefox, Edge, Safari, iOS Safari and Chrome for Android.

Where do I start?

The level module is the recommended way to get started. It offers a persistent database that works in both Node.js and browsers, backed by LevelDB and IndexedDB respectively. Many alternatives are available. For example, memory-level is an in-memory database backed by a red-black tree. Visit Level/awesome to discover more modules.

What is abstract-level?

If you are new to Level, there is a quick answer: abstract-level is the new core of Level on top of which several databases are (or will be) implemented. Read on if you're already familiar with Level modules (before 2022) and have used level, levelup, abstract-leveldown, encoding-down or deferred-leveldown.

Back in 2012, levelup offered a Node.js binding for Google's LevelDB. Authored by Rod Vagg, levelup exposed the features of LevelDB in a Node.js-friendly way. It had streams, binary support, encodings... all the goodies. Later on, the binding was moved to leveldown, so that other stores could be swapped in while retaining the friendly API of levelup.

This is when "up" vs "down" naming was born, where databases followed the formula of "level = levelup + leveldown". For example, level-mem was a convenience package that bundled levelup with memdown. The abstract-leveldown module offered a lower-level abstraction for the "down" part, to encapsulate common logic between "down" stores. Many such stores were written, replacing LevelDB with IndexedDB, RocksDB, in-memory red-black trees, relational databases and more.

Around 2017, further parts were extracted from levelup and moved to single-purpose modules. This effectively introduced the concept of "layers", where an implementation of abstract-leveldown wasn't necessarily a storage for levelup but could also wrap another abstract-leveldown implementation. For example, levelup encoding logic was extracted to encoding-down. This changed the database formula to "level = levelup + encoding-down + leveldown". Or in other words: "levelup + layer + layer".

This highly modular architecture led to clean code, where each module had a single responsibility. By this time, the overall API had settled and matured, some contributors moved on to other exciting things and the primary remaining effort was maintenance. This posed new challenges. We worked on test suites, added automated browser tests, code coverage and database manifests.

Yet, releases too often required canary testing in dependents. It was hard to predict the effect of a change. In addition, documentation became fragmented and some modules actually suffered from the high modularity, having to peel off layers to customize behavior. At the same time, we could see that typical usage of a Level database still involved encodings and the other goodies that the original levelup had.

Enter abstract-level. This module merges levelup, encoding-down and abstract-leveldown into a single codebase. Instead of implementing behaviors "vertically" in layers, it is done per database method. Performance-wise abstract-level is on par with the old modules. GC pressure is lower because methods allocate less callback functions. Custom (userland) database methods also benefit from the new architecture, because they can reuse utility methods included in abstract-level rather than a layer having to detect and wrap custom methods.

Lastly, abstract-level comes with new features, some of which were not possible to implement before. Among them: Uint8Array support, built-in sublevels, atomically committing data to multiple sublevels, and reading multiple or all entries from an iterator in one call.

How do I upgrade to abstract-level?

We've put together several upgrade guides for different modules. For example, if you're currently using level@7 and no other modules (ignoring transitive dependencies) then it will suffice to read the upgrade guide of level@8.

Naming-wise, databases generally use an npm package name in the form of *-level while utilities and plugins are called level-*. This replaces the down versus up naming scheme. Similarly, while it was previously helpful for documentation to distinguish between "database" and its "underlying store", now you will mostly just encounter the term "database".

To upgrade, please consult the following table. If you use a combination of the modules listed here, each must be upgraded to its abstract-level equivalent.

Old module New module Named export 3 Upgrade guide
level <= 7 level >= 8 Level level@8
abstract-leveldown abstract-level AbstractLevel abstract-level@1
levelup n/a n/a Depends 2
level or levelup with streams level-read-stream EntryStream level-read-stream@1
leveldown classic-level ClassicLevel classic-level@1
level-mem memory-level MemoryLevel memory-level@1
memdown memory-level MemoryLevel memory-level@1
level-js browser-level BrowserLevel browser-level@1
level-rocksdb rocks-level RocksLevel Not yet available
rocksdb rocks-level RocksLevel Not yet available
multileveldown many-level ManyLevelGuest many-level@1
level-party rave-level RaveLevel Not yet available
subleveldown1 n/a n/a abstract-level@1
deferred-leveldown1 n/a n/a abstract-level@1
encoding-down1 n/a n/a abstract-level@1
level-errors1 n/a n/a abstract-level@1
level-packager n/a n/a n/a
level-supports <= 2 level-supports >= 3 supports n/a
level-codec 4 level-transcoder Transcoder level-transcoder@1
level-test n/a n/a Not yet available
  1. Functionality is now included in abstract-level.
  2. If the module that you're wrapping with levelup is listed here then refer to that module's upgrade guide, else see abstract-level@1.
  3. Most new modules use named exports, for example const { ClassicLevel } = require('classic-level') instead of const leveldown = require('leveldown').
  4. Encodings that follow the level-codec interface (without level-codec as a dependency) can still be used.

Where can I get support?

If you need help - technical, philosophical or other - feel free to open an issue in community or a more specific repository. We don't (yet) use GitHub Discussions, at least until discussions get the ability to close them.

You will generally find someone willing to help. Good questions get better and quicker answers. We do not offer paid support. All time is volunteered.

Where can I follow progress?

Most if not all activity happens on GitHub. See our project board to find out what we're working on. Any timelines there are just a rough indication of priority. We cannot guarantee that feature X or Y will actually be released on the given dates.

Subscribe to individual repositories to follow their progress. All releases are accompanied by a changelog and a GitHub Release, which gives you the option to only subscribe to new releases.

People

Collaborators

Collaborator emeriti

Contributors

Is your name missing? Send us a pull request!

API

This repository also used to hold a small amount of metadata on past and present contributors. They can be accessed from code by:

console.log(require('level-community'))

This metadata is no longer maintained and the npm package will be deprecated at some point. Contributors are instead documented in this README under People.

Contributing

Level/community is an OPEN Open Source Project. This means that:

Individuals making significant and valuable contributions are given commit-access to the project to contribute as they see fit. This project is more like an open wiki than a standard guarded open source project.

See the Contribution Guide for more details.

Donate

Support us with a monthly donation on Open Collective and help us continue our work.

License

MIT

community's People

Contributors

artskydj avatar chesles avatar dependabot[bot] avatar dominictarr avatar greenkeeper[bot] avatar heapwolf avatar huan avatar jcrugzz avatar juliangruber avatar kesla avatar mafintosh avatar max-mapper avatar mcollina avatar no9 avatar obastemur avatar pgte avatar ralphtheninja avatar raynos avatar vweevers avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

heapwolf no9 dominictarr davidmarkclements tbuchok jgravois digideskio gavofyork ralphtheninja isabella232 elpapo23 roborambo branarakic mustafateke535 jiskra33

community's Issues

GitHub topics?

Does anyone make use of this? Should we make an effort to have the same set of topics on every repo?

Standard status

  • level-fstream
  • lazy-open
  • level-ttl
  • abstract-leveldown
  • awesome
  • codec
  • concat-iterator
  • deferred-leveldown
  • electron-demo
  • encoding-down
  • errors
  • iterator-stream
  • level
  • level-browserify
  • leveldown
  • leveldown-hyper
  • level-hyper
  • level-js
  • level-rocksdb
  • level-test
  • levelup
  • level-ws
  • mem
  • memdown
  • packager
  • rocksdb
  • subleveldown
  • leveldown-mobile
  • level-mobile
  • level-basho
  • level-lmdb

Ongoing and Future Work

This is a summary on what has been happening lately, what we're working on right now and what the future holds. If you want to see anything in particular done right now or cool things for the future, feel free to post suggestions and/or edit this message.

Ongoing

  • Run tests in Sauce Labs using airtap where applicable
  • Implement coverage across the board using coveralls.

Future

  • Add canary tests (Level/abstract-leveldown#184)
  • IDEA: Try to find good use cases for where we can make use of Property Based Testing. The idea is to try to build a test system that helps us find more bugs (see cryptpad with notes from PBT meetup)
  • Fix up the homepage running at leveljs.org. We have really been slacking off here and we could do some really cool stuff with tutorials, write blog posts on the project as a whole (this issue could be a post)
  • Bring back basho and lmdb into separate *downs (like rocksdb)? The easiest way is most likely to start from current state of leveldown and pull in the deps/ again from scratch.
  • Write more benchmarks and run them continuously

Achieved

  • Merge level(up) functionality into abstract-leveldown (#58, abstract-level)
  • Refactor (the dependencies of) subleveldown
  • Implement manifests (#83)
  • Make sure we have a set of publishers for all modules (#17)
  • Increase transparency with a public backlog: https://github.com/orgs/Level/projects/2
  • Homogenize and polish documentation (#29)
  • Port native modules to N-API (leveldown-hyper not yet done)
  • Refactor the internals of leveldown and friends, fixing resource cleanup and segfaults along the way (leveldown-hyper not yet done)
  • Complete and maintain level/awesome showing what's there and the status of related projects when it comes to dependencies etc
  • Implement seek() in more abstract-leveldown implementations
  • Drop key types other than strings and buffers from memdown and level-js (Level/memdown#186 (comment))
  • Address security concerns from community (#43)

Write tutorials for real world use cases

Beginners would feel way more welcome and we had more shareable stuff if we had some well written tutorials for realworld usecases, showing the leveldb way and showcasing plugins.

Initial ideas:

  • the redis twitter tutorial ported to leveldb
  • creating indexes
  • map reduce (should be for map-reduce beginners)
  • simple persistent realtime data with level-scuttlebutt
  • philosophical rant on databases that are monoliths
  • accessing a db from multiple processes
  • writing a levelUp plugin

More website-/app-y ideas would be great though

Remark warnings on no social profile cont. 

lms@x260 ~/src/level/leveldown-hyper (prepare-2.0.0)
$ npm run remark 

> [email protected] remark /home/lms/src/level/leveldown-hyper
> remark README.md CONTRIBUTORS.md CHANGELOG.md UPGRADING.md -o

CHANGELOG.md
  1:1  info     skipping: no contributors heading found                                require-heading  remark-git-contributors

CONTRIBUTORS.md
  1:1  warning  no social profile for [email protected]                                 social           remark-git-contributors
  1:1  warning  no social profile for [email protected]                       social           remark-git-contributors
  1:1  warning  no social profile for [email protected]                  social           remark-git-contributors
  1:1  warning  no social profile for [email protected]                       social           remark-git-contributors
  1:1  warning  no social profile for [email protected]                                 social           remark-git-contributors
  1:1  warning  no social profile for [email protected]                             social           remark-git-contributors
  1:1  warning  no social profile for [email protected]                                 social           remark-git-contributors
  1:1  warning  no social profile for [email protected]                                 social           remark-git-contributors
  1:1  warning  no social profile for [email protected]                                social           remark-git-contributors
  1:1  warning  no social profile for [email protected]                             social           remark-git-contributors
  1:1  warning  no social profile for [email protected]                               social           remark-git-contributors
  1:1  warning  no social profile for [email protected]                         social           remark-git-contributors
  1:1  warning  no social profile for [email protected]                                   social           remark-git-contributors
  1:1  warning  no social profile for [email protected]                                 social           remark-git-contributors
  1:1  warning  no social profile for [email protected]                             social           remark-git-contributors
  1:1  warning  no social profile for [email protected]  social           remark-git-contributors
  1:1  warning  no social profile for [email protected]                                  social           remark-git-contributors

README.md
  1:1  info     skipping: no contributors heading found                                require-heading  remark-git-contributors

UPGRADING.md
  1:1  info     skipping: no contributors heading found                                require-heading  remark-git-contributors

20 messages (⚠ 17 warnings)

Add dependency-check to all repositories

  • abstract-leveldown
  • awesome
  • codec
  • community
  • concat-iterator
  • database
  • deferred-leveldown
  • electron-demo
  • encoding-down
  • errors
  • iterator-stream
  • lazy-open
  • level
  • level-browserify
  • leveldb.org
  • leveldown
  • leveldown-hyper
  • level-hyper
  • level-js
  • level-rocksdb
  • level-test
  • level-ttl
  • levelup
  • level-ws
  • mem
  • memdown
  • packager
  • rocksdb
  • subleveldown

Beef up benchmark suite

I'd like to drop the SQLite and Leveled bencharks, as we know we're faster than SQLite and also Leveled, as soon as #95 is done.

Then, add bencharks for all levelup features so we can test streams, encodings and more.

Are you cool with this?

Address security concerns from community

In light of the recent event-stream incident, we (@ralphtheninja and I) want to take action to reduce the attack surface of packages maintained in Level.

Level has been and will remain an OPEN Open Source Project. While we recognize the risk of giving people owner rights, it has been vital to the open, transparent and dare I say loving nature of Level. We might add some policy, if it really benefits security. Keep in mind that too much policy can scare off contributors, put a burden on maintainers and provide a false sense of security, hiding real issues that are out of our control under a layer of bureaucracy that in addition impedes individual freedom.

Trust is essential in OSS and we want to be wary of knee-jerk reactions to incidents like event-stream.

That said, we are thinking about what we can do and open to any suggestions. After an initial brainstorm we came up with 3 actionable items and wished to move further discussion to GitHub for community input and transparency.

1. Reduce npm owners

  • Our npm packages have more owners than needed for continued maintenance. Go through the list of current owners and ask if they really want/need publish rights: #17.

2. Reduce GitHub organization owners

  • We have quite a few inactive members. We will ask them if they can be removed and list removed members as Collaborator Emeriti in a suitable place.

3. Archive unmaintained projects

Archival consists of:

  • Pinning dependencies (note: transient dependencies are out of our control)
  • Releasing a final patch version
  • Deprecating the package
  • Removing extraneous npm owners
  • Archiving the GitHub repository.

Candidates for archival:

Please edit the above list or leave a comment if you think one of these should not be archived.

Add and remove npm owners

I've avoided @mentioning owners who are not (currently) active here, except when they are the sole owners or if already mentioned in this thread or a previous edit. For packages that are (to be) archived, we're more aggressive with removing owners.

replication

@gedw99: "I am building a 3D cad modelling system and tons of json data I need to
store on the servers in many data centers.
I run offline using indexdb and so need to also sync.

Originally I used pouchdb and couxhdb.

But I want to replace all of it with level dB."

  • what's the merge strategy?
  • will it be master-only?
  • how is your topology?

LevelDB & Node.js 'real world' use cases

I've read the docs on LevelDB and some topics on the LevelDB Google Group & StackOverflow, I understand for what it was built.

What I want to know is what are some of your use cases and on what scenario do you believe LevelDB & Node.js is a good fit.

I am not very experienced with DBs but I would like to learn and there's not too much info on LevelDB & Node.js

Thank you :)

plugin injection points

So, we have had quite a bit of discussion, and tried various approaches to implementing plugins in levelup

https://github.com/rvagg/node-levelup/issues/search?q=plugins

quick summation of what has happened so far, I started experimenting with a crude monkey patching based approach, but ran into trouble with handling ranges - each plugin had manage which ranges it affected, which was tricky. I later refactored this to create a subsection of the database, with level-sublevel. This is a great improvement, because it allows you to extend a range within leveldb as if it's a whole db.

@rvagg has also experimented with exposing various integration points into levelup, https://github.com/rvagg/node-levelup/issues/92

personally, I am highly in favor of combining these two, and even merging sublevel into levelup, or at least, adding integration points to levelup so that level-sublevel does not have to monkey patch it.

the question is: what is the list of integration points that we need?

  • prehooks (intercept a mutation [batch, put, del])
  • posthooks (intercept a mutation callback)
  • encoding/key-encoding
  • setup (register special jobs that run first, after the database opens,
  • asynchronously delay mutations. **

** maybe. The ability to get the current values for keys before performing a mutation.
this would be useful for validation, and merging concurrent updates.

I have a plugin for this, but it hasn't been updated to work with level-sublevel yet. This differs from level-hooks, which only provides
a sync api.

A setup integration point will be useful for saving metadata about the database in the database, and maybe stuff like summaries about the current overall state - whether a schema change migration is complete, etc.

Any other suggestions?

Implement read-snapshots

options.snapshot = NULL; on an iterator will give a consistent snapshot view of the data while reading. db->GetSnapshot(); will create a snapshot and return a version that can be set against options.snapshot. The drawback of the latter is that you have to explicitly db->ReleaseSnapshot() to clean up versions you've created.

At least the NULL case would be handy to start with.

Create SauceLabs OSS accounts

As I wrote here:

It turns out that you can have only 1 subaccount, and it isn't public. Not all that useful.
So we'll need separate Sauce Labs accounts for each repo. We can however reuse our gmail address, with aliases. E.g. for the memdown account we'll use leveldb.org+memdown@, for abstract-leveldown we use leveldb.org+abstract.leveldown@, etc.

Might be easier to do this in one go. For:

  • memdown (previously used the main gmail address)
  • level-test (Level/level-test#62)
    • Create account
      • Username: level-test
      • Email: leveldb.org+level.test@
    • Activate OSS
    • Setup Travis and airtap
  • abstract-leveldown (Level/abstract-leveldown#123)
    • Create account
      • Username: abstract-leveldown
      • Email: leveldb.org+abstract.leveldown@
    • Activate OSS
    • Setup Travis and airtap
  • level-packager
    • Create account
      • Username: level-packager
      • Email: leveldb.org+level.packager@
    • Activate OSS
    • Setup Travis and airtap
  • levelup
    • Create account
      • Username: levelup
      • Email: leveldb.org+levelup@
    • Activate OSS
    • Setup Travis and airtap
  • level-js (Level/level-js#44)
    • Create account
      • Username: level-js
      • Email: leveldb.org+level.js@
    • Activate OSS
    • Setup Travis and airtap

And at a later time, maybe also:

  • level-browserify
  • level-mem
  • encoding-down
  • deferred-leveldown
  • level-codec
  • level-errors
  • level-iterator-stream
  • level-ws

What to do with the 'leveldb' package in npm

The current owner of the leveldb package in npm has agreed to let us take it over, we just need to figure out what we want to put there: my8bird/node-leveldb#52. Note that the current leveldb package isn't very happy with newer releases of Node, and certainly not 0.11 which is a major pain for any native addon to support.

The only caveat is that we just bump the major version so the existing code can be left intact.

Some (non-exhaustive) options:

  • Publish the current level package as leveldb
  • Publish leveldown as leveldb
  • Create something new that wraps leveldown to make something that behaves closer to the LevelDB API (I'm not sure what this would look like, LevelDOWN probably is that thing already)

Thoughts?

One additional thought from me is that as far as I know, what the current leveldb package does that we don't support is:

  • Explicit snapshots
  • Synchronous operations

While I'm not particularly in favour of adding the latter, I think we really should get to work on getting snapshotting in, it's the big missing thing and while it's not something we have people begging for, it might open up some interesting new experiments!

Code coverage (nyc + coveralls)

  • abstract-leveldown
    • Enabled nyc and coveralls
    • Tests running at 100%
  • codec
    • Enabled nyc and coveralls
    • Tests running at 100%
  • concat-iterator
    • Enabled nyc and coveralls
    • Tests running at 100%
  • deferred-leveldown
    • Enabled nyc and coveralls
    • Tests running at 100%
  • electron-demo
    • Enabled nyc and coveralls
    • Tests running at 100%
  • encoding-down
    • Enabled nyc and coveralls
    • Tests running at 100%
  • errors
    • Enabled nyc and coveralls
    • Tests running at 100%
  • iterator-stream
    • Enabled nyc and coveralls
    • Tests running at 100%
  • level
    • Enabled nyc and coveralls
    • Tests running at 100%
  • leveldown
    • Enabled nyc and coveralls
    • Tests running at 100%
  • leveldown-hyper
    • Enabled nyc and coveralls
    • Tests running at 100%
  • level-hyper
    • Enabled nyc and coveralls
    • Tests running at 100%
  • level-js
    • Enabled nyc and coveralls
    • Tests running at 100%
  • level-rocksdb
    • Enabled nyc and coveralls
    • Tests running at 100%
  • level-test
    • Enabled nyc and coveralls
    • Tests running at 100%
  • level-ttl
    • Enabled nyc and coveralls
    • Tests running at 100%
  • levelup
    • Enabled nyc and coveralls
    • Tests running at 100%
  • multileveldown
    • Enabled nyc and coveralls
    • Tests running at 100%
  • level-ws
    • Enabled nyc and coveralls
    • Tests running at 100%
  • mem
    • Enabled nyc and coveralls
    • Tests running at 100%
  • memdown
    • Enabled nyc and coveralls
    • Tests running at 100%
  • packager
    • Enabled nyc and coveralls
    • Tests running at 100%
  • rocksdb
    • Enabled nyc and coveralls
    • Tests running at 100%
  • subleveldown
    • Enabled nyc and coveralls
    • Tests running at 100%
  • level-browserify (archived)
    • Enabled nyc and coveralls
    • Tests running at 100%
  • lazy-open (archived)
    • Enabled nyc and coveralls
    • Tests running at 100%
  • level-lmdb (to be archived)
    • Enabled nyc and coveralls
    • Tests running at 100%
  • leveldown-mobile (to be archived)
    • Enabled nyc and coveralls
    • Tests running at 100%
  • level-mobile (to be archived)
    • Enabled nyc and coveralls
    • Tests running at 100%
  • level-fstream (to be archived)
    • Enabled nyc and coveralls
    • Tests running at 100%
  • level-basho (archived)
    • Enabled nyc and coveralls
    • Tests running at 100%

Action required: Greenkeeper could not be activated 🚨

🚨 You need to enable Continuous Integration on all branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because we are using your CI build statuses to figure out when to notify you about breaking changes.

Since we did not receive a CI status on the greenkeeper/initial branch, we assume that you still need to configure it.

If you have already set up a CI for this repository, you might need to check your configuration. Make sure it will run on all new branches. If you don’t want it to run on every branch, you can whitelist branches starting with greenkeeper/.

We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.

Once you have installed CI on this repository, you’ll need to re-trigger Greenkeeper’s initial Pull Request. To do this, please delete the greenkeeper/initial branch in this repository, and then remove and re-add this repository to the Greenkeeper integration’s white list on Github. You'll find this list on your repo or organiszation’s settings page, under Installed GitHub Apps.

Migrate to lerna

With monorepo would be easier to maintain dependencies and fix related issues.

badges?

Which badges do you want to use for level? (It's fine to say that you don't want any at all 😀) Anything in particular that you like more or is a must have? Which badges suck and are useless to you?

I personally like badges a lot, but it's also a fine line between when they are just too many and don't give any "valuable" information.

My approach to plugins

Posting in a new issue instead of #68 or #80 because this is about a specific implementation that I'd like comment on.

The LevelUP pluggability branch contains an extension to 0.6.x that uses externr to expose some basic extension points (more are possible of course). I've put an example in that branch here that places prefixes on keys (and removes them on the way out), and also prevents reading of certain other keys, purely to provide an example.

The externr approach is quite different from the level-hooks approach so I don't imagine this to be non-controversial. My hope is that the two approaches can coexist and perhaps leverage off each other.

While externr is pretty fast for the 'noop' case where you don't have a plugin operating on a given extension point, my guess is that it'll undo the performance gains in #90 (which haven't made it into a release yet fyi).

The way LevelUP extensions work is with a "use" call or option. There is a global levelup.use(plugin) that you can register plugins with any LevelUP instance created after that point. There is a "use" property on the options argument to levelup() when you're making a new instance, the property can point to a single plugin or an array of plugins. That instance will then use those plugins. Each instance will also expose a .use() method that can take single or arrays of plugins, so you could add plugins after an instance is created.

Plugins are simply objects whose keys are the extension points it wishes to inject itself in to. The values are functions that do the work. See the example linked to above to see what I mean.

The LevelDOWN plugins branch contains additions the implement a basic plugin system for the native layer by way of providing a Plugin class that can be extended. So far it has only one extension point, an Init(database) method that passes a newly created LevelDOWN database instance (i.e., when you call leveldown(location)). Plugins can then do what they like on the database object, mostly just adding methods or replacing existing ones I suspect. But I imagine also being able to offer a mechanism to insert a particular LevelDB comparator if you need your db sorted in a particular way. Or even more advanced, a replacement LevelDB filter if you have something more efficient than the default bloom filter.

Currently the way you put a plugin in LevelDOWN is with a global leveldown._registerPlugin(location) call, where location is the path to a .node object file it can dlopen(). Once loaded, the plugin is placed into a list of plugins and when needed the list is iterated over and each plugin has the appropriate method invoked (currently just Init() on instance creation).

Extending LevelDOWN is quite tricky, I'm not aware of any other native build offering plugins. So there are a few challenges. Currently there's an npm issue that's preventing it from being a seamless thing.

My working example is a range delete which I decided could be implemented outside of LevelUP/LevelDOWN and offered as a plugin. @Raynos has level-delete-range but when you have to do individual deletes on callbacks from an iterator it's majorly inefficient; you really want to be doing bulk deletes in one go, native & async.

Enter Downer RangeDel. I've exposed a .use() function on the exports so you have to explicitly run that to make it inject itself into the nearest leveldown it can find (hopefully there's just one, peerDependencies should help with that). When a LevelDOWN instance is created, it attaches a .rangeDel() method to the object. The plugin is able to reuse a lot of the existing LevelDOWN code for iterators so the .rangeDel() method has an almost identical options signature to a .readStream() in LevelUP. Doing the actual delete is a simple 5-line job but it's efficient and done in one go with no callback until it's completed.

Then, to make that available to LevelUP, I have Upper RangeDel. It has downer-rangedel as a dependency so you only need to load it alongside levelup to get going. I've also exposed a .use() method there so you have to explicitly invoke it too. It'll inject itself globally into LevelUP so it'll run in every LevelUP instance you create (but, you could opt out of global and levelup({ use: require('upper-rangedel') }) for an individual instance.

The code for this should be a bit easier to understand cause it's all JS. The plugin simply extends the "constructor" of each LevelUP instance and passes the call down to this._db.rangeDel() where it expects that Downer RangeDel has put a method. I've also added some arguments checking and mirroed the deferred-open functionality in the rest of LevelUP so you could do something like: levelup('foo.db').rangeDel() and it should work.

To show it in action, I have an example of Upper RangeDel in work. It uses "dev" tagged levelup and leveldown releases in npm as this is not available in current "latest" releases.

package.json

{
  "name": "example",
  "version": "0.0.0",
  "main": "index.js",
  "dependencies": {
    "levelup": "~0.7.0-b02",
    "upper-rangedel": "0.0.1"
  }
}

index.js

require('upper-rangedel').use()

var levelup = require('levelup')
  , db = levelup('/tmp/foo.db')
  , data = [
        { type: 'put', key: 'α', value: 'alpha' }
      , { type: 'put', key: 'β', value: 'beta' }
      , { type: 'put', key: 'γ', value: 'gamma' }
      , { type: 'put', key: 'δ', value: 'delta' }
      , { type: 'put', key: 'ε', value: 'epsilon' }
    ]
  , printdb = function (callback) {
      db.readStream()
        .on('data', console.log)
        .on('close', callback)
        .on('error', callback)
    }

db.batch(data, function (err) {
  if (err) throw err
  console.log('INITIAL DATABASE CONTENTS:')
  printdb(function (err) {
    if (err) throw err
    db.rangeDel({ start: 'β', limit: 3 }, function (err) {
      if (err) throw err
      console.log('\nDATABASE CONTENTS AFTER rangeDel({ start: \'β\', limit: 3 }):')
      printdb(function (err) {
        if (err) throw err
        console.log('\nDone')
      })
    })
  })
})

Output

INITIAL DATABASE CONTENTS:
{ key: 'α', value: 'alpha' }
{ key: 'β', value: 'beta' }
{ key: 'γ', value: 'gamma' }
{ key: 'δ', value: 'delta' }
{ key: 'ε', value: 'epsilon' }

DATABASE CONTENTS AFTER rangeDel({ start: 'β', limit: 3 }):
{ key: 'α', value: 'alpha' }
{ key: 'ε', value: 'epsilon' }

Done

You can run this now, but you'll have to do a second npm install after the first one finishes with a failure; this is something we'll need to overcome in npm or with some crazy hackery with gyp or npm pre/postinstalls.

This obviously needs more polish and thought before its production ready, but I also want to make sure I have some kind of agreement on the approach before I push ahead. So I need your thoughts!

What about sublevels?

I think we need to invest some time in either level-sublevel or subleveldown, get one of them up to par with latest levelup, abstract-leveldown and encoding-down, maybe deprecate the other.

Thoughts @Level/owners?

Implement db#clear across the board

Background: Level/abstract-leveldown#24 (comment).

Phase 1: get it working

Phase 2: optimize

On the side

WebAssembly build of LevelDB

It would be nice to have webassembly build of leveldown. There are many cases when platform independent build is more preferable than top performance.

Example: wrapped Google's woff2 https://github.com/fontello/wawoff2. ~ 2x slower than native, but no need to recompile, download and so on. Convenient.

Ideas for a query language

This is an idea for a level module, but am posting it here because it's too early in the morning/late at night to implement and this will notify exactly the right people.

level is pretty useful as a straight key value store, but sooner or later you need to query something else,
so you add an index. A lot of leveldb modules are about indexes. For example, level-seach which just indexes everything.

The trouble with all these indexes, is to manage them all - which index do you need for which query and when do you add a new index? there is a design trade off here.

It would be so much easier if you could just not think about indexes at all, just have filters,
just forEach (scan) over all the data and return only what you want. totally functional and elegant,
just not performant.

But what if you could use that interface, and it would transparently use the right indexes when possible? (or even better, generate them when necessary)

Lets say we have the npm registry, and we want to retrieve all the modules written by @substack,
even though substack has written many modules, it is still less than 1% of the registry, so a full scan
would be 99% wasted effort. Lets say that the average registry document is 5k, that is 95000*0.99*5000=470mb of unnecessary reads. 475mb total.

But lets say that we indexed all the maintainers? for simplicity lets say that the average length of both module names and user names is 8 characters. Say we create an index like {username}:{modulename} and index this for every module, 95000*(8+8+1) (1 for the separator) = 1.6mb of index data. but now we just need to read through 1% of the index (since they are ordered by username we can just jump straight to substack) 950*17 = 15200 that is only 15k of unnecessary data.
So, now we only read 15200 + 950*5000 = 4.7mb

Using this index we increase storage requirements by 1.6 mb but increase read perf by 100x.

But what about a more complex query? suppose we want to find modules written by substack that depend (directly) on async? now, a simple way to do this would be scan through the modules written by substack and check whether async is in the deps. Lets say there was a time when substack used async, in 9 modules. 1% of the time. the scan costs us 4.7 mb, like before, but we discarded 99% of it.

Okay, so lets create a filter for this. udm:{username}:{dep}:{module} where module depends on dep
this index is more expensive because it has needs another 9 characters, plus we need a tag to distinguish it from the other indexes. 3+1+3*(8+1) = 29 so this index would need 2.75mb.
how much data would it save us? now we could directly read out the 9 substack modules that use async, 29*9+ 9*5000 = 45k this is 1000 times less! but here is the problem: how do we know we want to perform this query? what if we want to know all the users that depend on optimist? we'd also need a dum:{dep}:{username}:{module} index - and so on, we want our database to be really really fast,
so lets just index every pair of properties twice!

Lets say there are 20 properties on average, how many combinations of 2 properties?
(20!/(2!*(20-2)!) which cancels down to (20*19)/2 = 275), times 2 because we need 2 indexes,
gives us: 550*29 = 15950, that is 3 times the size of the document. if there where 30 properties it would be 40*39*29 = 45240. doubling the size of the document increases the size of the index 3 times,
now 9 times the size of the original document.

Now, at some point, it's better to do a scan than have another index, because the indexes take up too much space, and optimize queries that you never use. Clearly this is dependant on what sort of queries you do in practice, possibly you could have a system that automagically creates indexes when you start to use a query more. maybe even could it know when you have an expensive query?

But is there a nice way that you can gracefully transition between those?
decouple the query interface from the indexes, so you can add and remove indexes
without changing the queries at all. And measuring the efficiency of the queries,
so that you know when new indexes are needed.

module system

rvagg:

I don't feel strongly either way about this. I'm not a strict minimalist but like Dominic I tend to use batch() programatically so a chaining API is less helpful (but I may use it).

What we really need is a proper plugin system for Node so a project like LevelUP can load optional plugins that may be installed in the NODE_MODULES path(s) and do things like expose the main LevelUP.prototype to plugins that may want to augment it. Then we could say something on our README like: "If you npm install levelup-batch, LevelUP will make the chaining batch() API available. And levelup-batch can monkey-patch LevelUP.prototype.batch to provide chaining.

I need something like this for Ender too so maybe I'll actually build a generic plugin helper system some day..

rvagg:

Re plugins, Grunt and DocPad do something similar already but certainly not in a modular way that you can pull out and re-use, we need something that smaller projects like LevelUP can easily include. We should collaborate on it so we get the API right!

Snapshot API

What about this API for snapshots

db#snapshot()

Create a new snapshot.

snapshot#get(key, value[, opts])

Read a single value from a snapshot.

snapshot#create{Read,Key,Value}Stream

Read multiple values from a snapshot.

snapshot#dispose()

Delete the snapshot.

Maybe also:

snapshot#{del,createWriteStream,...}

Just forwarded to db#*

Usage

For a consistend read stream this would be:

var snapshot() = db.snapshot();
snapshot.createReadStream()
  .pipe(...)
  .on('close', shapshot.dispose.bind(snapshot));

Use undefined instead of error for non-existing records

This is a request for discussion about the semantics of db.get() when a key doesn't exist. Currently, an error with a non-enumerable type key is provided for missing keys:

db.get('does-not-exist', function (err, value) {
  console.log('err=', err);
  console.log('err.type=', err.type);
})

produces this output:

err= {}
err.type= NotFoundError

The error is not easy to inspect at a glance (a separate issue) and if you want to check for non-existence you've got to remember that the err.type for non-existence is NotFoundError.

Would it make more sense to just use undefined for the value and not go through the err parameter? undefined is already an invalid value to use for storage so it wouldn't conflict with anything unless there are some edge cases for custom valueEncodings I'm not aware of.

Requirements for manifests

I felt the need to defragment various threads:

Prior art:

Requirements:

  • Manifests must be objects (established in Level/levelup#279)
  • Declare high-level features as booleans:
    • snapshot guarantees (use case: tests, consumers that require consistency)
    • exclusive access (use case: live streams, prehooks) (see Level/levelup#279)
    • binary keys (use case: tests, deciding on a network transport)
    • permanence (use case: packager tests)
    • seeking (use case: tests, multiget, skip scans)
  • Declare additional methods and properties (that are not part of the abstract API)
    • Examples:
      • approximateSize() (use case: defer/proxy/expose)
      • sublevels (use case: exposing them to clients)
    • Declare name, return type, sync/callback/promise
  • Nice to have: use manifests the other way around, to declare features that a plugin wants

Open questions:

  • Should manifests extend manifests from underlying downs? Note that underlying downs can be swapped at runtime.
  • In some cases, feature support depends on the runtime environment too. E.g. not all browsers support binary keys in level-js. Is that something we want to expose?

State of documentation

The state of README.md (style updated with level badge etc), CHANGELOG.md and UPGRADING.md.

EDIT (@ralphtheninja): Updated with LICENSE.md/CONTRIBUTORS.md/README.md which essentially means "There is a LICENSE.md file and a CONTRIBUTORS.md file, where LICENSE.md links to CONTRIBUTORS.md and README.md links to LICENSE.md according to new and simplified format in https://github.com/Level/level-js/tree/ce8d77c89f38e444b6951e890aa3b5e72a221aaf#license"

EDIT (@vweevers): Added tasks to remove contributors from package.json and to remove copyright headers from code.

Below is a summary of all repositories. Some of these are not actively maintained and some might be more or less irrelevant for other reasons, e.g. maybe we don't need UPGRADING.md for electron-demo etc.

Please comment and/or edit this post if you think that something should be archived or need special care.

  • abstract-leveldown
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • awesome
    • README.md (needs to be re-generated after level-js and level-browserify)
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • codec
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • concat-iterator
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • community
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • deferred-leveldown
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • electron-demo
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • encoding-down
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • errors
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • iterator-stream
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • level
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • leveldb.org
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • leveldown
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • leveldown-hyper
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • level-hyper
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • level-js
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • level-test
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • level-rocksdb
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • level-ttl
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • levelup
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • level-ws
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • mem
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • memdown
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • packager
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • rocksdb
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • subleveldown
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • lazy-open (archived)
    • README.md
    • CHANGELOG.md
    • ~~ UPGRADING.md~~
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • level-lmdb (to be archived)
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • level-mobile (to be archived)
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • leveldown-mobile (to be archived)
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • level-fstream (to be archived)
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • level-basho (archived)
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • level-browserify (archived)
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • typings (archived)
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code
  • database (archived)
    • README.md
    • CHANGELOG.md
    • UPGRADING.md
    • LICENSE.md/CONTRIBUTORS.md/README.md
    • Remove contributors from package.json
    • Remove copyright headers from code

plugin extension points: merge level-hooks / level-sublevel

I'm not suggesting we do this right away, but I am suggesting this is something worth considering.

Also, I don't want to merge this, unless there is a consensus that this is an easy and effective way to build plugins.

I'd also intend to refactor the code style, etc, to match levelup.

I think that given these base features, and maybe one or two more,
a wide array of plugins could be developed.

plugins v2 (sectioned db, with hooks)

Basically, instead of hooking new behaviour into the main database, you can create sub sections:

SubLevel(db)
var fooDb = db.sublevel('foo', '~')

fooDb is an object with the levelup api, except when you do an fooDb.put('bar') the key is prefixed with ~foo~ so that it's separated from the other keys in the main db. This is great, because if you want to build a section of the database with special behaviour in it, you can create a subsection, and extend the behaviour in anyway you like -- but it will only affect that section! So, you can monkeypatch it - whatever, you won't introduce bugs into other parts of the program.

Most of the plugins I built needed some sort of interception, where a value is inserted into one range, which triggers an action which inserts something into a different section. To get reliably consistent data this needs to be atomic.

so, you can tie set hooks on a subsection, that trigger an insert into another subsection.

when a key is inserted into the main db, index that write with a timestamp, saved into another subsection.

var SubLevel = require('level-sublevel'); SubLevel(db)
var sub = db.sublevel('SEQ')

db.pre(function (ch, add) {
  add({
    key: ''+Date.now(), 
    value: ch.key, 
    type: 'put'
  }, sub) //NOTE pass the destination db to add
          //and the value will end up in that subsection!
})

db.put('key', 'VALUE', function (err) {

  //read all the records inserted by the hook!
  sub.createReadStream()
    .on('data', console.log)
})

db.pre(function hook (ch, add) {...}) registers a function to be called whenever a key is inserted into that section. ch is a change, like a row argument to db.batch: {key: k, value: v, type: 'put' || 'del'}.

If the hook function calls add(ch) ch is added to the batch (a regular put/del is turned into a batch) but, if a subsection is passed in add(ch, sub) then that put will be added to that subsection.

Compare subsection code for queue/trigger https://github.com/dominictarr/level-sublevel/blob/e2d27cc8e8356cde6ecf4d50c980c2ba93d87b95/examples/queue.js
with the old code -
https://github.com/dominictarr/level-queue/blob/master/index.js
and https://github.com/dominictarr/level-trigger/blob/18d0a1daa21aab1cbc1d0f7ff3690b91c1e0291d/index.js

the new version is only ~ 60 lines, down from about ~ 200, also it's possible to use multiple different queue/trigger libs within the same db. And also, there is no tricky code that refers to ranges or prefixes in the subsection based code!

in summary,

  • create subsections, add any features to your subsection.
  • use pre(fun) to trigger atomic inserts into your subsection.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.