venemo / node-lmdb Goto Github PK

While there are many great examples, there isn't automated test coverage. Many of the examples could be converted into tests that could be run with continuous integration.

All the examples show lots of warnings like this

(node) v8::ObjectTemplate::Set() with non-primitive values is deprecated
(node) and will stop working in the next major release.

and follow huge stack traces. When node 6.x will get support?

Create a wrapper for mdb_env_info.

When running in multi-process mode I am getting the error MDB_READERS_FULL.

Looking through the code env.cpp seems to default it to 1

    // Parse the maxDbs option
    rc = applyUint32Setting<unsigned>(&mdb_env_set_maxreaders, ew->env, options, 1, "maxReaders");

however, mdb.c seems to default it to 126

    /** Number of slots in the reader table.
     *  This value was chosen somewhat arbitrarily. 126 readers plus a
     *  couple mutexes fit exactly into 8KB on my development machine.
     *  Applications should set the table size using #mdb_env_set_maxreaders().
     */
#define DEFAULT_READERS 126

also, adding maxReaders:126 to the env.options seems to fix it.

Is there any reason why keys cannot be stored in UTF8 format rather than UTF16?

Under the lmdb hood they're just a bunch of bytes and as most keys tend to be Latin using 2 bytes per char seems pretty wasteful.

If you have 30M+ keys (which we do) it can add up to much larger databases which need to be paged in and out of memory.

Create a wrapper for mdb_env_copy.

I've been messing around with "String::NewExternal(new CustomExternalStringResource(&data))" versus "Buffer::New((char_)data.mv_data, data.mv_size, [](char*, void*) -> void { /_ Don't need to do anything here, because the data belongs to LMDB anyway */ }, NULL)->handle_" and come to the conclusion that with a 4KB data then the former is about four times faster. It's as if Buffer::New() is not doing the zero copy... Have you done any performance tests? -- Simon

Hey,
I've discovered a memory leak in the lmdb.Cursor and updated the cursor example to reproduce the leak outside of my application.
After that I've used Instruments Memory-Leak Template and attached it to the running node process to digg a bit deeper and saw the obvious and continuously growing amount of CustomExternalStringResource …
I hope the following screenshots can help to fix this leak and make this lmdb module ready for long term and heavy usage execution.

cheers

In the readme it says to build with:

node-gyp configure
make -C build -j4

But this really should be:

node-gyp configure
node-gyp build

What are the -C and -j4 flags doing?

Cannot get the LMDB to compile. here are my server details:

We are running:
CentOS 6.5 x86_64
gcc version 4.4.7
node 0.10.28

and here is the failed build log

root@dev01-ident node-lmdb]# node-gyp build
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | linux | x64
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
make: Entering directory `/root/node-lmdb/build'
 CC(target) Release/obj.target/node-lmdb/libraries/liblmdb/mdb.o
cc1: warning: command line option "-std=c++0x" is valid for C++/ObjC++ but not for C
cc1: warning: command line option "-fvisibility-inlines-hidden" is valid for C++/ObjC++ but not for C
 CC(target) Release/obj.target/node-lmdb/libraries/liblmdb/midl.o
cc1: warning: command line option "-std=c++0x" is valid for C++/ObjC++ but not for C
cc1: warning: command line option "-fvisibility-inlines-hidden" is valid for C++/ObjC++ but not for C
 CXX(target) Release/obj.target/node-lmdb/src/node-lmdb.o
 CXX(target) Release/obj.target/node-lmdb/src/env.o
../src/env.cpp: In constructor ‘EnvWrap::EnvWrap()’:
../src/env.cpp:42: error: ‘nullptr’ was not declared in this scope
../src/env.cpp: In static member function ‘static v8::Handle<v8::Value> EnvWrap::open(const v8::Arguments&)’:
../src/env.cpp:146: error: ‘nullptr’ was not declared in this scope
../src/env.cpp: In static member function ‘static v8::Handle<v8::Value> EnvWrap::close(const v8::Arguments&)’:
../src/env.cpp:166: error: ‘nullptr’ was not declared in this scope
../src/env.cpp: In static member function ‘static v8::Handle<v8::Value> EnvWrap::sync(const v8::Arguments&)’:
../src/env.cpp:209: error: expected primary-expression before ‘[’ token
../src/env.cpp:209: error: expected primary-expression before ‘]’ token
../src/env.cpp:209: error: expected primary-expression before ‘*’ token
../src/env.cpp:209: error: ‘request’ was not declared in this scope
../src/env.cpp:209: error: expected unqualified-id before ‘void’
../src/env.cpp:213: error: expected primary-expression before ‘[’ token
../src/env.cpp:213: error: expected primary-expression before ‘]’ token
../src/env.cpp:213: error: expected primary-expression before ‘*’ token
../src/env.cpp:213: error: expected primary-expression before ‘int’
../src/env.cpp:213: error: expected unqualified-id before ‘void’
make: *** [Release/obj.target/node-lmdb/src/env.o] Error 1
make: Leaving directory `/root/node-lmdb/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/lib/node_modules/node-gyp/lib/build.js:267:23)
gyp ERR! stack     at ChildProcess.EventEmitter.emit (events.js:98:17)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (child_process.js:807:12)
gyp ERR! System Linux 2.6.32-431.17.1.el6.x86_64
gyp ERR! command "node" "/usr/bin/node-gyp" "build"
gyp ERR! cwd /root/node-lmdb
gyp ERR! node -v v0.10.28
gyp ERR! node-gyp -v v0.13.1
gyp ERR! not ok

I ran it by my server host technician, and he did not fair any better.

I am trying to do multiple concurrent reads on my environment, however if I try to open a second transaction with the readOnly flag this error gets thrown

Error: MDB_BAD_RSLOT: Invalid reuse of reader locktable slot

When opening both transactions without the readOnly flag the process simply freezes on the second call to txn.beginTxn()

The only two concurrent transaction can exist is when one is opened with readOnly flag and the other one without, however opening any other transaction on top of those two will lead to one of the errors mentioned above.

The behaviour can easily be reproduced in the example example3-multiple-transactions.js by either removing the readOnly flag from the txn.beginTxn() call on line 28 or adding it to the call on line 33 (which of course will lead to an error in line 35, but it nonetheless shows the erroneous behaviour)

I would expect a error from opening a second transaction without the readOnly flag, as to writing transactions are not allowed, but readOnly transaction should not be limited in number, as the lmdb documentation states:

. One simultaneous write transaction is allowed, however there is no limit on the number of read transactions even when a write transaction exists.

As there can only be one write transaction at time, attempting to begin two write transactions with the same environment will result in the process locking that is impossible for the code to exit without being terminated externally.

var txn1 = env.beginTxn();
var txn2 = env.beginTxn();

A few ideas:

• throw an error on the second attempt
• return null
• asynchronous callback option that could wait until a write transaction could be obtained

If you need to use the data later, you will have to copy it for yourself.

How do you recommend copying the strings?

I was hoping there would be a transactional method that would create a transaction, read a string and then close the transaction in the external string resource destructor.

mapSize settings larger than 2^32 - 1 don't work and are silently ignored.
However, file sizes of 80GB and more seem to be supported by LMDB.

Below is example code that triggers the error on Linux and OSX.

var lmdb = require('node-lmdb');

var env = new lmdb.Env();
env.open({ path: '.', mapSize: 16 * 1024 * 1024 * 1024 });
var dbi = env.openDbi({ name: 'test', create: true });

try {
  while (true) {
    var txn = env.beginTxn();
    txn.putString(dbi, randomString(128), randomString(512));
    txn.commit();
  }
}
catch (error) {
  console.log('database size', require('fs').statSync('data.mdb').size / 1024 / 1024, 'MB');
  console.log(error);
}

function randomString(length) {
  var result = '';
  while (length-- > 0)
    result += String.fromCharCode(97 + Math.floor(Math.random() * 26));
  return result;
}

It gives the following output:

database size 9.99609375 MB
[Error: MDB_MAP_FULL: Environment mapsize limit reached]

Hi,

Currently this module only supports string/uint as key, it's limited. For example, I would like to use float/double as key and keep them sorted.

bytewise module can serialize any structures into buffer including float/double.

As you have said in doc:

the data returned by txn.getString() and txn.getBinary() is only valid until the next put operation or the end of the transaction. If you need to use the data later, you will have to copy it for yourself.

What is the most reliable way to copy strings before closing environment and use that string later?

I have tested that ""+string and string.substr(0) would not work and (" "+string).slice(1) would work.

Create a wrapper for mdb_stat.

The module does not seem to be available through npm which is rather inconvenient.

Search for lmdb on npmjs gives something leveldb-related instead.

Update: just saw in an older closed issue you don't consider the module to be "ready". Would still be good to add it which would allow folks like me to do some testing more easily.

Currently the cursor get functions getCurrentString, getCurrentNumber, getCurrentBinary and getCurrentBoolean return the value asynchronously.

Would it be possible to add versions that return the value synchronously in a similar way to the general get functions as it would make iterator code much less complex?

There doesn't seem to be a license file or information for node-lmdb (though I see one for lmdb itself I think). Can you please provide one for node-lmdb?

Thanks

Thanks for this great lmdb binding. I was not able to find how to list current dbis / subdatabases in an environment. Sorry if this is obvious I am new to lmdb.

This is a followup to the discussion at #20 - I've finally figured out the real reason behind the problem.

The cursors example does something like this:

var dbi1 = env.openDbi({ name: "hello", create: true });
dbi1.drop();
var dbi2 = env.openDbi({ name: "hello", create: true });
// use dbi2

The real issue is NOT that the V8 GC ate dbi2 but that it (correctly) destroyed dbi1 which in turn called mdb_dbi_close on the MDB_dbi, while that MDB_dbi was still in use by dbi2.

The correct solution to this problem:

• Never call mdb_dbi_close as it is not required (confirmed by LMDB authors); thus eliminating the above issue

To take correct care of the GC:

• DbiWrap needs to Ref and Unref the EnvWrap it uses
• TxnWrap needs to Ref and Unref the EnvWrap it uses
• CursorWrap needs to Ref and Unref the DbiWrap and TxnWrap it uses

Do you have build instructions? I can't seem to install this with NPM.

Kudos on bringing LMDB to Node.

This module needs tests.

Imdb dbs created by other modules (e.g. rvagg) store their cursors as binary data.

For our importing script I was attempting to get around this by casting the string cursors from node-lmdb to binary and then to utf8, this works some of the time but always leads to corruption.

A simple function to getTheCurrentCursorAsBinary would seem to get around this.

We hacked together a fix as such and it's working for us so far.

$ node-gyp build
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | darwin | x64
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
CC(target) Release/obj.target/node-lmdb/libraries/liblmdb/mdb.o
../libraries/liblmdb/mdb.c:9671:46: warning: data argument not used by format string [-Wformat-extra-args]
(int)mr[i].mr_pid, (size_t)mr[i].mr_tid, txnid);
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~
/usr/include/secure/_stdio.h:47:56: note: expanded from macro 'sprintf'
__builtin___sprintf_chk (str, 0, __darwin_obsz(str), __VA_ARGS__)
^
1 warning generated.
CC(target) Release/obj.target/node-lmdb/libraries/liblmdb/midl.o
CXX(target) Release/obj.target/node-lmdb/src/node-lmdb.o
CXX(target) Release/obj.target/node-lmdb/src/env.o
../src/env.cpp:199:23: error: no matching constructor for initialization of 'Nan::Callback'
d->callback = new Nan::Callback(callback);
^ ~~~~~~~~
../node_modules/nan/nan.h:1361:12: note: candidate constructor not viable: no known conversion from 'Handle<v8::Function>' to 'const v8::Local<v8::Function>' for 1st argument
explicit Callback(const v8::Local<v8::Function> &fn) {
^
../node_modules/nan/nan.h:1439:33: note: candidate constructor not viable: no known conversion from 'Handle<v8::Function>' to 'const Nan::Callback' for 1st argument
NAN_DISALLOW_ASSIGN_COPY_MOVE(Callback)
^
../node_modules/nan/nan.h:129:23: note: expanded from macro 'NAN_DISALLOW_ASSIGN_COPY_MOVE'
NAN_DISALLOW_COPY(CLASS) \
^
../node_modules/nan/nan.h:105:35: note: expanded from macro 'NAN_DISALLOW_COPY'
# define NAN_DISALLOW_COPY(CLASS) CLASS(const CLASS&) = delete;
^
../node_modules/nan/nan.h:1439:33: note: candidate constructor not viable: no known conversion from 'Handle<v8::Function>' to 'Nan::Callback' for 1st argument
NAN_DISALLOW_ASSIGN_COPY_MOVE(Callback)
^
../node_modules/nan/nan.h:130:23: note: expanded from macro 'NAN_DISALLOW_ASSIGN_COPY_MOVE'
NAN_DISALLOW_MOVE(CLASS)
^
../node_modules/nan/nan.h:107:5: note: expanded from macro 'NAN_DISALLOW_MOVE'
CLASS(CLASS&&) = delete; /* NOLINT(build/c++11) */ \
^
../node_modules/nan/nan.h:1355:3: note: candidate constructor not viable: requires 0 arguments, but 1 was provided
Callback() {
^
../src/env.cpp:220:22: error: no matching member function for call to 'Call'
d->callback->Call(argc, argv);
~~~~~~~~~~~~~^~~~
../node_modules/nan/nan.h:1429:3: note: candidate function not viable: no known conversion from 'Handle<v8::Value> [1]' to 'v8::Local<v8::Value> *' for 2nd argument
Call(int argc, v8::Local<v8::Value> argv[]) const {
^
../node_modules/nan/nan.h:1417:3: note: candidate function not viable: requires 3 arguments, but 2 were provided
Call(v8::Local<v8::Object> target
^
2 errors generated.
make: *** [Release/obj.target/node-lmdb/src/env.o] Error 1

The docs at https://github.com/Venemo/node-lmdb#managing-the-lmdb-dependency seem to be outdated, has the process changed?

Create a wrapper for mdb_set_compare.

While doing testing, I've noticed that dupFixed in always being set to true, when dupSort is specified. For example:

var testDbi = env.openDbi({
  name: 'test',
  create: true,
  dupSort: true
});

And then putting two values with different sizes:

var value1 = new Buffer(new Array(8));
var value2 = new Buffer(new Array(4));
txn.putBinary(testDbi, key, value1);
txn.putBinary(testDbi, key, value2);

Will give the error:

Error: MDB_BAD_VALSIZE: Unsupported size of key/DB name/data, or wrong DUPFIXED size

From reading documentation the size of the value should only need to be fixed when MDB_DUPFIXED is specified.

v0.3.0 is the latest published version, v0.4.0 patches a memory leak

In order to access non named databases mdb_dbi_open needs to be called with NULL for the value of 'name'.

Currently this is not possible and so there is no way to access the data saved from other node lmdb modules such as https://github.com/rvagg/lmdb.

I recently ran into an issue where NodeJS reproducible crashed with an Segmentation fault error when trying to access data read from an node-lmdb environment.

The scenario was this: Small LMDB environment, 3 test entries keyIsUint32 was set to false, UTF8 string with about 12 characters where used as keys. Data was binary buffer.

The data was read with a cursor using getCurrentBinary and was stored in a Map using the key returned by the cursor as key and the binary buffer as data.

After the all data was read, the cursor, txn, dbi and environment were properly closed and the Map was returned to the calling code.

However upon accessing the keys of the map (mind not the data, only the key), the node process would quit with a segmentation fault error.
I debugged the node process with lldb and found that there was an illegal attempt to read invalid memory access causing a EXC_BAD_ACCESS error.

I then realised that this must be due to V8 the zero-copy feature used for string, which means that the string of the key actually resides in the memory used by LMDB and only the reference is passed to the JavaScript. Therefore the reference points to invalid memory once the environment is closed an LMDB frees the memory.

The solution therefore is quite simple: Copying the content of the key to a new memory address by calling new Buffer(key) and then make that new Buffer a string again.
After doing so, the string can safely be accessed.

I have a few questions about this:

1. Is the above conclusion correct and would you suggest another solution?
2. Why does this not affect the binary buffer read from LMDB? The README states that both, string and binary are provided via zero-copy, but accessing the binary buffer does not cause the segmentation fault.

I would greatly appreciate if this was mentioned clearly in the README and/or the examples as this might be very troublesome to debug for most.
If you want, I'll prepare and PR with updated README me and examples.

PS: This behaviour can easily be reproduced with example 4-cursors.js by storing the key and value in the printFunc on a new Map and log this Map after env.close() was called

Since getBinary, getString, putBinary, putString can perform io, should there be equivalent methods that are asynchronous?

I'm writing data to LMDB with another program. the keys are utf8 strings (in my case, really just ascii) and the values are snappy-encoded json.

I'm trying to get a node script to scan and process the same database. But strings are garbled coming out. The values in the database can be read and decoded correctly with cursor.getCurrentBinary(). But they keys are incorrectly assumed to be utf-16 strings.

Could some option be exposed to set the key encoding?

edit: there is a manual workaround using buffers.

cursor.getCurrentBinary(function(k, v) {
    var kutf8 = new Buffer(k, 'utf-16le').toString('utf-8');
    // ...
});

Create a wrapper for mdb_set_dupsort.

Test if node-lmdb can be built on Mac OS X and test it by running the example code in the repository.

Requiring node-lmdb currently is required with the full path:

var lmdb = require('node-lmdb/build/Release/node-lmdb');

Which when switching between bindings that are compiled with the debug flag, the require statements need to be updated to:

var lmdb = require('node-lmdb/build/Debug/node-lmdb');

This could be automatically handled with the bindings module or prebuild, and compatible with both with:

var lmdb = require('node-lmdb');

Test if node-lmdb can be built on Windows and test it by running the example code in the repository.

Hey, I'm playing around with your cursor example and noticed that cursor.getCurrentString is never called and therefore the while loop never stops when you run it against an empty database.

I tried to build node-lmdb on a Raspberry Pi, following your instructions. But unfortunately the build failed. After some google-ing around I found that I could get it working by opening the file “binding.gyp”; in the fragment

"cflags": [
"-fPIC",
"-fvisibility-inlines-hidden",
"-O3",
"-std=c++11"
]
I had to change “-std=c++11″ in “-std=c++0x”. After that it build without further errors.

Now I don't pretend to understand why this works (or why the build failed in the first place). But perhaps this information is useful to you in further developing node-lmdb. This build error was the only error I got; until now node-lmdb is working flawlesly for me (see: http://raspberryadventures.in/index.php/installing-and-using-lmdb-on-the-raspberry-pi/)

If I take e.g. dbi.close (or seemingly dbi.anything, really) and assign it to a local variable, calling that local variable crashes node-lmdb with a cryptic error.

Here's a test-case:

var lmdb = require("node-lmdb");
var env = new lmdb.Env();

console.log("Opening env");
env.open({
    "path"    : "./testdb",
    "max-dbs" : 1
});

console.log("Opening dbi");
dbi = env.openDbi({
    name : "test1",
    create : true
});

/*
// Works:
console.log("Closing dbi");
dbi.close();
*/

// Dies:
console.log("Assigning");
var close = dbi.close;

console.log("Closing dbi");
close(); // <-- Here

This is the complete output, running the above example from zsh with node on a 64-bit Linux system:

Opening env
Opening dbi
Assigning
Closing dbi
node: /home/aku/.node-gyp/0.10.22/src/node_object_wrap.h:61: static T* node::ObjectWrap::Unwrap(v8::Handle<v8::Object>) [with T = DbiWrap]: Assertion `handle->InternalFieldCount() > 0' failed.
zsh: abort (core dumped)  node lmdb-closure-bug-testcase.js

JS functions are first class, so this is a sensible operation in JS-land, but it seems to confuse the underlying C++.

I'm on 7a12bd3 (currently newest master) and node -v is v0.10.24.

Hi,

Trying to figure out how this works.. I was expecting to see callbacks, for instance when getting data txn.getString(dbi, 'abc', function(data) { ... })

... but instead there are synchronous calls everywhere?

same as title

Create a wrapper for mdb_env_stat.

(let me start by saying I'm confused. I've been using node-lmdb for several weeks apparently without a problem. As far as I can tell, the issue I'm having occurs only on my (development) Mac, and my (production) debian machine works fine)

    var fs = require("fs"),
        lmdb = require('node-lmdb');

    var env = new lmdb.Env();
    env.open({
        path: "./lmdb",
        mapSize: 2*1024*1024*1024, // maximum database size
        maxDbs: 2
    });
    var dbi = env.openDbi({
        name: "moindb",
        create: true
    })

    var nm='name',
        s1 = "This is an ordinary string";
    fs.writeFileSync('1st', s1, {encoding:'utf8'});
    var txnw = env.beginTxn();
    txnw.putString(dbi,nm,s1);
    txnw.commit();
    var txnr = env.beginTxn({ readOnly: true }),
        s2 = txnr.getString(dbi,nm);
    txnr.commit();
    fs.writeFileSync('2nd', s2, {encoding:'utf8'});

The output is in file 1st: This is an ordinary string, but in file 2nd: T�h�i�s� �i�s� �a�n� �o�r�. That file has the same length, with invisible characters between the visible ones. In another editor it looks liek: T^@h^@i...

Seems like an encoding issue, or does this have to do with the dire warning in the README: 'Because of the nature of LMDB, the data returned by txn.getString() and txn.getBinary() is only valid until the next put operation or the end of the transaction.'

Any suggestion on how to get around this?

Heyhey,
we stumbled across an error when trying to build the module for node v.0.11:

../src/node-lmdb.h:58:30: error: expected class name
class EnvWrap : public node::ObjectWrap {

This seems to be due to the api changes for native modules when changing from v.0.10.x to v.0.11.x. Do you plan to port node-lmdb to v.0.11 in the very near future? This would be great!
Maybe this could help: nan

Thanks a lot and cheers!

Hi,

It appears that none of the functions are async based, I'd have at least expected the open/close calls to be async. Can you confirm?

A callback-based interface adds a significant amount of overhead. The synchronous methods should ideally return values instead of using a callback.

Hi,

I'm trying to run the first example but it fails, see output:

node v0.8.25:

node example1-env.js

module.js:485
process.dlopen(filename, module.exports);
^
Error: /home/awel/dev/lmdb/node_modules/node-lmdb/build/Release/node-lmdb.node: undefined symbol: init
at Object.Module._extensions..node (module.js:485:11)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:362:17)
at require (module.js:378:17)
at Object. (/home/awel/dev/lmdb/node_modules/node-lmdb/example1-env.js:3:12)
at Module._compile (module.js:449:26)
at Object.Module._extensions..js (module.js:467:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)

node v0.10.12:

node example1-env.js
Current lmdb version is { versionString: 'MDB 0.9.7: (January 10, 2013)',
major: 0,
minor: 9,
patch: 7 }

/home/awel/dev/lmdb/node_modules/node-lmdb/example1-env.js:12
env.open({
^
Error: No such file or directory
at Object. (/home/awel/dev/lmdb/node_modules/node-lmdb/example1-env.js:12:5)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Function.Module.runMain (module.js:497:10)
at startup (node.js:119:16)
at node.js:901:3

both under Linux Mint 15

Any thoughts?