level / levelup Goto Github PK

Node version 0.8.14

No problem installing on OSX, but when I tried to put it on the server I got the following error:

gyp ERR! configure error
gyp ERR! stack Error: gyp failed with exit code: 1
gyp ERR! stack at ChildProcess.onCpExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:350:16)
gyp ERR! stack at ChildProcess.EventEmitter.emit (events.js:99:17)
gyp ERR! stack at Process._handle.onexit (child_process.js:678:10)
gyp ERR! System Linux 2.6.39.1-x86_64-linode19
gyp ERR! command "node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /root/wotf/node_modules/levelup
gyp ERR! node -v v0.8.14
gyp ERR! node-gyp -v v0.7.1
gyp ERR! not ok
npm ERR! [email protected] install: node-gyp rebuild
npm ERR! sh "-c" "node-gyp rebuild" failed with 1
npm ERR!
npm ERR! Failed at the [email protected] install script.

am using windows 7 64 bit.

npm http GET https://registry.npmjs.org/levelup
npm http 200 https://registry.npmjs.org/levelup
npm http GET https://registry.npmjs.org/levelup/-/levelup-0.5.1.tgz
npm http 200 https://registry.npmjs.org/levelup/-/levelup-0.5.1.tgz
npm http GET https://registry.npmjs.org/bufferstream/0.5.1
npm http GET https://registry.npmjs.org/errno/0.0.3
npm http GET https://registry.npmjs.org/concat-stream/0.0.9
npm http GET https://registry.npmjs.org/bindings/1.0.0
npm http 200 https://registry.npmjs.org/concat-stream/0.0.9
npm http GET https://registry.npmjs.org/concat-stream/-/concat-stream-0.0.9.tgz
npm http 200 https://registry.npmjs.org/bufferstream/0.5.1
npm http GET https://registry.npmjs.org/bufferstream/-/bufferstream-0.5.1.tgz
npm http 200 https://registry.npmjs.org/bindings/1.0.0
npm http GET https://registry.npmjs.org/bindings/-/bindings-1.0.0.tgz
npm http 200 https://registry.npmjs.org/errno/0.0.3
npm http GET https://registry.npmjs.org/errno/-/errno-0.0.3.tgz
npm http 200 https://registry.npmjs.org/bufferstream/-/bufferstream-0.5.1.tgz
npm http 200 https://registry.npmjs.org/bindings/-/bindings-1.0.0.tgz
npm http 200 https://registry.npmjs.org/errno/-/errno-0.0.3.tgz
npm http 200 https://registry.npmjs.org/concat-stream/-/concat-stream-0.0.9.tgz
npm http GET https://registry.npmjs.org/buffertools/1.1.0
npm http 200 https://registry.npmjs.org/buffertools/1.1.0
npm http GET https://registry.npmjs.org/buffertools/-/buffertools-1.1.0.tgz
npm http 200 https://registry.npmjs.org/buffertools/-/buffertools-1.1.0.tgz

[email protected] install /Users/local/ZOHOCORP/koteswara-0347/dbtest/node_modules/levelup/node_modules/bufferstream/node_modules/buffertools
node-gyp rebuild

gyp http GET http://nodejs.org/dist/v0.8.16/node-v0.8.16.tar.gz
gyp WARN install got an error, rolling back install
gyp ERR! configure error
gyp ERR! stack Error: connect ETIMEDOUT
gyp ERR! stack at errnoException (net.js:770:11)
gyp ERR! stack at Object.afterConnect as oncomplete
gyp ERR! System Darwin 11.4.2
gyp ERR! command "node" "/usr/local/bin/node-gyp" "rebuild"
gyp ERR! cwd /Users/local/ZOHOCORP/koteswara-0347/dbtest/node_modules/levelup/node_modules/bufferstream/node_modules/buffertools
gyp ERR! node -v v0.8.16
gyp ERR! node-gyp -v v0.6.1
gyp ERR! not ok
npm ERR! [email protected] install: node-gyp rebuild
npm ERR! sh "-c" "node-gyp rebuild" failed with 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the buffertools package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node-gyp rebuild
npm ERR! You can get their info via:
npm ERR! npm owner ls buffertools
npm ERR! There is likely additional logging output above.

npm ERR! System Darwin 11.4.2
npm ERR! command "node" "/usr/local/bin/npm" "install" "levelup" "--save"
npm ERR! cwd /Users/local/ZOHOCORP/koteswara-0347/dbtest
npm ERR! node -v v0.8.16
npm ERR! npm -v 1.1.69
npm ERR! code ELIFECYCLE
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! /Users/local/ZOHOCORP/koteswara-0347/dbtest/npm-debug.log
npm ERR! not ok code 0

currently, put and del callback the keys they where called with,
this is problematic for hooking into these, because a put could become a batch,
(and batch only calls back cb(err) or cb())

I can't see much use it calling back the args, especially since the callback is likely to already have access to them via closure scope.

I using the callback signature of batch for the put and get.

The gain is to make hooks simpler.

your thoughts everyone?

We should implement this lambda architecture on top of levelup as an example of how easy stuff is.

Things like:

• query
• delete range

can all be modules that are a single function and take a levelup db instance and do some stuff.

@rvagg would you prefer a smaller core with an emphasis on userland modules that do more complicated things with levelup or would you prefer a larger core?

Hooks on put and del that allow you to turn put and del into a batch.

If your usage of levelup is complicated you may want to intercept del and put and also store a marker in the database saying "some kind of job needs to run"

@dominictarr needs this for map-reduce.

The main reason for this is that you cant use before:put or after:put since if the process crashes then either you have the data and no marker or the marker and no data in the database.

You want to convert the put or del into an atomic batch operation.

options.filter_policy = NewBloomFilterPolicy(10);

Should be helpful for performance but it'd be nice to have some benchmarks in place so it can be demonstrated. From LevelDB:

We recommend that applications whose working set does not fit in memory and that do a lot of random reads set a filter policy.

It would be nice to document a list of abstractions people have already build.

For example I'm looking for the simplest way to have multiple indexes for levelup.

But I don't think it's been written yet.

It would also be nice to have a place where we document high level patterns or have a general community wishlist for things that would be cool.

It would be cool to have a browser editor for leveldb,
It could also run on a indexeddb shim, so you could make a hello world site,
where you test all the stuff without installing anything!

It could also use events, and live streams to keep update stuff in realtime.

discussion, please!

(Was "Windows build fails", ed:RV)

Actual NPM error log:
https://gist.github.com/3730932

I just found something about compiling leveldb in Windows:
https://groups.google.com/d/msg/leveldb/VuECZMnsob4/tROLqJq_JcEJ

And here are the aditional packages that are needed:
http://www.boost.org/users/download

And some more information from official source:
http://code.google.com/p/leveldb/source/browse/WINDOWS?name=windows

The following code should give a helpful error about missing a name like this

events.js:68
        throw arguments[1]; // Unhandled 'error' event
                       ^
OpenError: #name cannot be `null` or `undefined`

Code

var levelup = require('levelup');
levelup('./testdb', function(err, db){
        db.approximateSize(function(err, size){
                if(err)
                        console.log(err);
                console.log(size);
        });
});

However we get this instead

/Users/sandfox/code/testme/node_modules/levelup/lib/levelup.js:323
      this.emit('error', err)
           ^
TypeError: Object #<Object> has no method 'emit'
    at LevelUP.approximateSize (/Users/sandfox/code/testme/node_modules/levelup/lib/levelup.js:323:12)
    at LevelUP.approximateSize (/Users/sandfox/code/testme/node_modules/levelup/lib/levelup.js:318:12)
    at /Users/sandfox/code/testme/test-error.js:5:5
    at LevelUP.open (/Users/sandfox/code/testme/node_modules/levelup/lib/levelup.js:86:11)

I've narrowed it down to scope issue and have hacked up a fix. My javascript skills aren't that great so feel free to bat down or suggest improvements.
Also I haven't made any proper test cases apart from the above

Imagine being able to do a range query

var stream = db.query({ start: start, end: end })

// get a never ending stream of all data in between that range

What this means is it will listen on put events and check whether its in the range and emit that on the stream.

You lose any garantuees of the result of the query being in order but it is a live / tailable query.

Having a function for

db.del({
    start: "a"
    , end: "b"
})

Would be useful.

Use-case here is to have mechanism to purge old data mainly

db.del({
    start: "name:"
    , end: "name:someTs"
})

Would you care to add github reference to your package.json (helps discovery via npm)?
I can make PR if you like.

According to the leveldb documentation http://leveldb.googlecode.com/svn/trunk/doc/index.html

There is a feature where you can pass in a comparitor function that defines the sorting of keys.

Does anyone have any plans to implement this?

alias existing versions but rename to match Node core

levelup prints errors on the console, like:

NotFoundError: Key not found in database [foo]
    at /Users/julian/git/multilevel/node_modules/levelup/lib/levelup.js:160:15

We should either remove that or provide opts.silent to turn it off.

A quick look around the source didn't tell me where the error is outputted.

Specifically 0 causes this error. Not sure what other values do.

var lu = require('levelup')
lu('test', {createIfMissing:true}, function(err, db) {
    var s = db.readStream({start: 0}).on('data', console.log);
});

node crashes with this error:

terminate called throwing an exceptionAbort trap: 6

Using {start: '0'} gives expected results.

Emit events for changes: put, batch & delete; a building block for replication.
Could probably do with open and close events too.

• ReadOptions::verify_checksums for read operations, which, by default, don't verify checksums.
• Options::paranoid_checks when opening a db will do aggressive checksum verification

Hi!

I've experienced increasing memory usage when using the readStream extensively, to the point where I suspect that levelup have a memory leak.

Running the code below gives me eventually a FATAL ERROR: JS Allocation failed - process out of memory.

var levelup = require('levelup');

var db = levelup('levelup-leak-db');

db.put('foo', 'bar');

function read() {
    db.readStream().once('end', read);
}
read();

setInterval(function() {
    console.log(process.memoryUsage());
}, 1000);

See complete diff from 0.4.4: 0.4.4...master

There's a persistent C++ problem that keeps on showing up but I haven't been able to make it occur since my last round of tweaks but I'm guessing that's just coincidental. A bit of work has backed up because I wanted to try and track it down before a release but I've yet to find anything!

I've published this current one to npm with the dev tag so you can install it as npm install levelup@dev and also with npm install [email protected] (I put out a dev of 0.5.0 yesterday, I should have made it 0.5.0-a or something but now an actual stable 0.5 will probably have to be 0.5.1).

Let me know if you have any reason to object to an 0.5 release, otherwise I'll publish it tomorrow.

Trying to install levelup via npm (actually was trying to install pouchdb but this dependency always fails).

NPM log https://gist.github.com/4132175

This is on - OSX 10.8.2, node v0.8.14

(as an FYI, I downloaded it via git and 'npm install' in the directory - compiles fine so looks like a broken version on NPM rather than the repo)

perhaps as a separate tool, that can be run on the cmdline & also called programatically?

Candidates to go

• encoding: "json"
• db.keyStream()
• db.valueStream()
• db.writeStream()

Hopefully writeStream() can just be a module.

Other possibilities:

• db.readStream() to be replaced with a very lightweight cursor implementation
• all encoding logic
• db.isOpen() and db.isClosed() if you care read db.state
• operation specific encoding. What is the use-case for this?

When I get a readStream({reverse: true}), followed by a plain readStream(), I get the documents out in reverse order again, unless I specify reverse: false. start and end behave similarly, the last value for each option acting as the default value for the next call.

Example:

> db.readStream().on('data', console.log)
{ key: '0', value: "blah" }
{ key: '1', value: "foo" }
{ key: '2', value: "bar" }
{ key: '3', value: "..." }

# ok, reverse the stream
> db.readStream({reverse: true}).on('data', console.log)
{ key: '3', value: "..." }
{ key: '2', value: "bar" }
{ key: '1', value: "foo" }
{ key: '0', value: "blah" }

# stream is still reversed
> db.readStream().on('data', console.log)
{ key: '3', value: "..." }
{ key: '2', value: "bar" }
{ key: '1', value: "foo" }
{ key: '0', value: "blah" }

# back to normal with reverse: false
> db.readStream({reverse: false}).on('data', console.log)
{ key: '0', value: "blah" }
{ key: '1', value: "foo" }
{ key: '2', value: "bar" }
{ key: '3', value: "..." }

https://gist.github.com/4105078

There may be some tweaks that can be done

This would be useful for say, or maybe retriving the latest result.
This feature could be used to make leveldb a mvcc database.

It'd be nice if the api supported shorthand. 🐙

{ 'put': 'key', value: output }

I could do a pull request for this if you like the idea.

I tried using both put and the WriteStream, but looking at the data (*.sst files) with less it doesn't appear to be compressed. It's a binary, because of how keys and values are separated, but that's all. I tried both bigger HTML files and custom text.

How would you integrate them?

Thinking about ways to create subdbs and cleanly partition dbs.

So, the idea is that you could create subdbs with namespaces like this:

var hiDb = db.createSub('hello')
//and then
hiDb.get('hello', function (){...})
//etc

To be really useful, I want this to work with hooks.
here is what I think it will look like:

say, I want to increment a sequence number for each thing key that is changed.

seqDb = db.createSub('sequence')
hiDb.pre(function (ch, add) {
  add({key: timestamp(), ch.key, type: 'put'}, seqDb)
})

here, adding seqDb to the add call tells add to add it with the prefix for that db.

note: this is a breaking change for the hooks api.

options.compression = leveldb::kNoCompression;

worth also documenting that this is not recommended as Snappy is very snappy and it's not run over uncompressible data anyway.

stream.write({ type: "del", key: "foo" })

should work.

Also maybe stream.write([{ type: "del", key: "foo" }, { type: "put", key: "foo2", value: "hello!" }])

This one is just a personal annoyance, not a great priority. Bufferstream is used solely for fstream compatibility, which is not likely to be a much-used feature. In turn, bufferstream in turn requires buffer-tools which requires compiling. All it's used for is to turn an existing, immutable, Buffer into a Stream, nothing fancy! There's native stuff to worry about without pulling in an additional compilable module so I'd prefer it if we could remove this dependency.

Before an upgrade, we need to build a functional test across an actual database created with 1.5.0 (compressed and uncompressed would be nice), just to make sure that LevelDB upgrades don't impact on the usability of databases created with older versions.

provide a set of key ranges and get back an approximation of space used on the fs to store the data

options.cache = leveldb::NewLRUCache(100 * 1048576);  // 100MB cache

Will turn on an uncompressed cache of frequently used blocks.

options.fill_cache = false;

During a read operation will prevent the read items from displacing items in the cache, it's probably worth turning this on by default for readStream()'s without start & end options but also as an option on any read operation, stream and get()

Emit a before put and after put event (same for del and batch)

Use case is to have an UI side effect when someone puts data into the database without waiting for file IO. It reduces observed latency in your application and makes it snappy

I wouldn't mind thoughts from other people on the API and if it can be improved. The goal of LevelUP is not to expose the exact LevelDB API but to expose something that's Node.js friendly, whether that's the LevelDB API or something else.

All operations must be async, I don't want any sync I/O at all, or even options for it. But this creates a bit of possible awkwardness around create/open:

var db = require('levelup').createDatabase('./mydb', { /* opts */ })
db.open(function (err) {
  // start doing stuff on `db`
})

The alternative might be something like this:

require('levelup').open('./mydb', { /* opts */ }, function (db, err) {
  // start doing stuff on `db
})

But I don't particularly not being able to have a reference to the database until it's opened, that feels a little too leaky.

Thoughts from outside my head would be appreciated!

/cc @maxogden

Current defaults are: { createIfMissing : false, errorIfExists : false } but I'm always changing this to { createIfMissing : true, errorIfExists : false }.

Any objections to changing to this for 0.5.0?

Your eventemitter seems to have the standard limit of 10 listeners...

put causes an emitter leak. You could say, just use batch (but it has the same problem). In many programs, put or batch will be used over time and intermittently, most likely exceeding max emitters.

var levelup = require('levelup');
var level = levelup('./test', { createIfMissing: true, errorIfExists: false });

for (var i = 0; i < 11; i++) {

  var key = 'i' + Math.random()*99;

  level.put(key, key, function (err) {
    console.log(key);
  });
}

(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
    at EventEmitter.addListener (events.js:178:15)
    at EventEmitter.once (events.js:199:8)
    at EventEmitter.LevelUP.put (/Users/paolo/workroot/git/wayla/Wayla-Redux/node_modules/levelup/lib/levelup.js:155:21)
    at Object.<anonymous> (/Users/paolo/workroot/git/wayla/Wayla-Redux/server/eeleak.js:9:9)
    at Module._compile (module.js:454:26)
    at Object.Module._extensions..js (module.js:472:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.runMain (module.js:497:10)
    at process._tickCallback (node.js:325:13)

for (var i = 0; i < 11; i++) {

  var key = 'i' + Math.random()*99;

  level.batch([{ type: 'put', key: key, value: key }], function (err) {
    console.log(key);
  });
}

they cause a segfault if let through, stop them at the interface layer, or perhaps even in the native layer just to be sure.

I want to make a small patch that is causing problems when using level-hooks as the database is opening.

The trouble is that each operation (get, put, del, batch) defers if the database is currently opening. However, if I have patched that operation, it will get called twice.

https://github.com/rvagg/node-levelup/blob/master/lib/levelup.js#L158

Instead, if the database is opening, it should not call the public method again, instead it should defer the call to the leveldown binding

https://github.com/rvagg/node-levelup/blob/master/lib/levelup.js#L174

will put in a pull request shortly!

Let's say I want encoding like

var db = levelup("uri", {
    toEncoding: function (raw) {
        return new Type(JSON.parse(raw))
    }
    , toBuffer: function (type) {
        delete type._id
        return JSON.stringify(type)
    }
})

function Type(data) { ... }

I basically want to wrap all objects coming out of the db in a certain encoding and clean up all objects I put into the db with a certain cleanup logic.

darn streams

related to dominictarr/level-scuttlebutt#2 & dominictarr/level-map#1

I'm on this, but this._status is getting out of hand