ozra / mmap-io Goto Github PK
View Code? Open in Web Editor NEWClean straight forward mmap-bindings for node.js
License: MIT License
Clean straight forward mmap-bindings for node.js
License: MIT License
It looks like the current version of package.json no longer has a "main" field (it got lost in 29070c9), and that means that the package will fail to import:
> require('mmap-io')
{ Error: Cannot find module 'mmap-io'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:581:15)
at Function.Module._load (internal/modules/cjs/loader.js:507:25)
at Module.require (internal/modules/cjs/loader.js:637:17)
at require (internal/modules/cjs/helpers.js:22:18) code: 'MODULE_NOT_FOUND' }
Unmapped buffer should behave like transferred buffer.
Update 12/2/18 -
This is context-aware issue with C++ addons. See this issue thread for more details and potential fixes: nodejs/node#21783
--
I'm trying to use this library with my multithreading (Node.js worker-threads). I am encountering this error message when I use mmap-io
in both my main thread and worker thread. See the following code and error out in order to reproduce.
script.js
const fs = require('fs')
const mmap = require('mmap-io')
const { Worker, isMainThread, parentPort, workerData } = require('worker_threads')
const { getRecord } = require('./helpers')
// Open File
const fdArgs = ['./data_test', 'r+']
const fd = fs.openSync(...fdArgs)
// Get Size
const size = fs.fstatSync(fd).size
// Define record size in bytes
const recordSize = 64
// Calculate number of records in file
const nRecords = parseInt((size + 1) / recordSize)
const worker = new Worker('./worker.js', {
workerData: {
size,
fdArgs,
swapItemPos: [3, 11],
recordSize
}
});
worker.on('message', res => {
console.log(`Response from thread: ${res.threadId}`)
const buffer = mmap.map(size, mmap.PROT_WRITE, mmap.MAP_SHARED, fd)
console.log(`Record 3: ${getRecord(buffer, recordSize, 3).toString()}`)
console.log(`Record 11: ${getRecord(buffer, recordSize, 11).toString()}`)
});
worker.on('error', err => {
console.error(err)
});
worker.on('exit', (code) => {
if (code !== 0)
throw new Error(`Worker stopped with exit code ${code}`)
})
worker.js
const fs = require('fs')
const mmap = require('mmap-io')
const { parentPort, workerData, threadId } = require('worker_threads')
const { swapRecords } = require('./helpers')
const { size, fdArgs, swapItemPos, recordSize } = workerData
const fd = fs.openSync(...fdArgs)
const buffer = mmap.map(size, mmap.PROT_WRITE, mmap.MAP_SHARED, fd)
swapRecords(buffer, recordSize, swapItemPos[0], swapItemPos[1])
parentPort.postMessage({ threadId })
std.err (Ignore line numbers as I have removed some comments and extra console.log statements to simplify the code in this issue)
size: 2047
nRecords: 32
Error: Module did not self-register.
at Object.Module._extensions..node (internal/modules/cjs/loader.js:718:18)
at Module.load (internal/modules/cjs/loader.js:599:32)
at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
at Function.Module._load (internal/modules/cjs/loader.js:530:3)
at Module.require (internal/modules/cjs/loader.js:637:17)
at require (internal/modules/cjs/helpers.js:20:18)
at Object.<anonymous> (/Users/ethanarrowood/Documents/Wentworth/Semesters/Fall2018/OperatingSystems/Project/node_modules/mmap-io/mmap-io.js:3:10)
at Module._compile (internal/modules/cjs/loader.js:689:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
at Module.load (internal/modules/cjs/loader.js:599:32)
/Users/ethanarrowood/Documents/Wentworth/Semesters/Fall2018/OperatingSystems/Project/script.js:41
throw new Error(`Worker stopped with exit code ${code}`)
^
Error: Worker stopped with exit code 1
at Worker.worker.on (/Users/ethanarrowood/Documents/Wentworth/Semesters/Fall2018/OperatingSystems/Project/script.js:41:13)
at Worker.emit (events.js:182:13)
at Worker.[kOnExit] (internal/worker.js:285:10)
at Worker.(anonymous function).onexit (internal/worker.js:241:51)
Once I remove mmap-io
from my script.js
the program runs as expect without errors (and the records are swapped in the file as intended by the logic in worker.js
). Could this have something to do with the read/write access on the "./build/Release/mmap-io.node"
file?
@ozra First off, really cool library. Any chance of adding support for mapping large files on windows? From the notes it seemed like that was limited to linux as of now? And I attempted a 3gb and mmap failed. Use case is visualizing large data sets like pointclouds.
Hi, thank you for maintaining this! I've just used it successfully for building a data labeling webapp, where the datasets are 100s of MB each.
I've cobbled this together from a stackoverflow answer (https://stackoverflow.com/questions/23747892/how-would-i-design-and-implement-a-non-blocking-memory-mapping-module-for-node-j), and perhaps others would enjoy something like as a starter example in the README too
var mmap = require('mmap-io')
var fs = require('fs')
fs.open('/file/path', 'r', (err, fd) => {
fs.fstat(fd, (err, stats) => {
var buf = mmap.map(stats.size, mmap.PROT_READ, mmap.MAP_SHARED, fd, 0);
const start = 0;
const len = 100;
var line = buf.slice(start, len).toString();
...
})
})
So it can be postMessage to worker with zero copy.
And is it cross platform?
Thanks
https://github.com/ozra/mmap-io/blob/master/src/mman.h#L52-L53
Should this be as below?
const DWORD dwEndLow = (sizeof(size_t) > sizeof(DWORD)) ? DWORD(end & 0x0000FFFFL) : DWORD(end);
const DWORD dwEndHigh = (sizeof(size_t) > sizeof(DWORD)) ? DWORD(end>>16) : DWORD(0);
For offset:
https://github.com/ozra/mmap-io/blob/master/src/mman.h#L54-L55
const DWORD dwOffsetLow = (sizeof(size_t) > sizeof(DWORD)) ? DWORD(offset & 0x0000FFFFL) : DWORD(offset);
const DWORD dwOffsetHigh = (sizeof(size_t) > sizeof(DWORD)) ? DWORD(offset>>16) : DWORD(0);
When trying to run a node.js file that requires mmap-io, I get the following error message:
node:internal/modules/cjs/loader:922
throw err;
^
Error: Cannot find module '../build/Release/mmap-io'
Require stack:
- D:\Desktop\projects\f1_app\node_modules\mmap-io\dist\mmap-io.js
- D:\Desktop\projects\f1_app\node_modules\mmap-io\mmap-io.js
- D:\Desktop\projects\f1_app\temp.js
at Function.Module._resolveFilename (node:internal/modules/cjs/loader:919:15)
at Function.Module._load (node:internal/modules/cjs/loader:763:27)
at Module.require (node:internal/modules/cjs/loader:991:19)
at require (node:internal/modules/cjs/helpers:92:18)
at Object.<anonymous> (D:\Desktop\projects\f1_app\node_modules\mmap-io\dist\mmap-io.js:3:23)
at Module._compile (node:internal/modules/cjs/loader:1102:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1131:10)
at Module.load (node:internal/modules/cjs/loader:967:32)
at Function.Module._load (node:internal/modules/cjs/loader:807:14)
at Module.require (node:internal/modules/cjs/loader:991:19) {
code: 'MODULE_NOT_FOUND',
requireStack: [
'D:\\Desktop\\projects\\f1_app\\node_modules\\mmap-io\\dist\\mmap-io.js',
'D:\\Desktop\\projects\\f1_app\\node_modules\\mmap-io\\mmap-io.js',
'D:\\Desktop\\projects\\f1_app\\temp.js'
]
}
When I went into the node-modules folder and located mmap-io, I saw that the build folder was empty. I did install mmap-io on a different machine but with the same OS (although this OS had WSL2) and it worked, so I'm not sure why the build folder would be empty.
In regards to compatibility, I was hoping I could use mmap-io in a phone app that must parse data from a very large text file; does mmap-io work with React/Expo-cli?
The readme suggests using make
to build mmap-io from git, but the Makefile was removed in 29070c9
The readme should be updated with the current build instructions.
Hi,
currently mmap-io have wildcard dependency version for nan library:
"dependencies": {
"bindings": "1.2.1",
"nan": "*"
},
and doesn't work with latest nan version (2.0.5), can you pin 'nan' version to 1.7.0 for example (this one is works correctly)?
Log of compilation with new version
make: Entering directory `/usr/lib/node_modules/mmap-io/build'
CXX(target) Release/obj.target/mmap-io/src/mmap-io.o
../src/mmap-io.cc: In function ‘Nan::NAN_METHOD_RETURN_TYPE mmap_map(Nan::NAN_METHOD_ARGS_TYPE)’:
../src/mmap-io.cc:45:22: error: ‘NanScope’ was not declared in this scope
NanScope();
^
../src/mmap-io.cc:47:17: error: ‘args’ was not declared in this scope
if (args.Length() < 4 && args.Length() > 6) {
^
../src/mmap-io.cc:51:17: error: ‘NanThrowError’ was not declared in this scope
);
^
../src/mmap-io.cc:51:17: error: return-statement with a value, in function returning 'void' [-fpermissive]
../src/mmap-io.cc:55:18: error: ‘args’ was not declared in this scope
if (!args[0]->IsNumber()) return NanThrowError("mmap: size (arg[0]) must be an integer", EINVAL);
^
../src/mmap-io.cc:55:111: error: ‘NanThrowError’ was not declared in this scope
if (!args[0]->IsNumber()) return NanThrowError("mmap: size (arg[0]) must be an integer", EINVAL);
^
../src/mmap-io.cc:55:111: error: return-statement with a value, in function returning 'void' [-fpermissive]
../src/mmap-io.cc:56:18: error: ‘args’ was not declared in this scope
if (!args[1]->IsNumber()) return NanThrowError("mmap: protection_flags (arg[1]) must be an integer", EINVAL);
^
../src/mmap-io.cc:56:123: error: ‘NanThrowError’ was not declared in this scope
if (!args[1]->IsNumber()) return NanThrowError("mmap: protection_flags (arg[1]) must be an integer", EINVAL);
^
../src/mmap-io.cc:56:123: error: return-statement with a value, in function returning 'void' [-fpermissive]
../src/mmap-io.cc:57:18: error: ‘args’ was not declared in this scope
if (!args[2]->IsNumber()) return NanThrowError("mmap: flags (arg[2]) must be an integer", EINVAL);
^
../src/mmap-io.cc:57:112: error: ‘NanThrowError’ was not declared in this scope
if (!args[2]->IsNumber()) return NanThrowError("mmap: flags (arg[2]) must be an integer", EINVAL);
^
../src/mmap-io.cc:57:112: error: return-statement with a value, in function returning 'void' [-fpermissive]
../src/mmap-io.cc:58:18: error: ‘args’ was not declared in this scope
if (!args[3]->IsNumber()) return NanThrowError("mmap: fd (arg[3]) must be an integer (a file descriptor)", EINVAL);
^
../src/mmap-io.cc:58:129: error: ‘NanThrowError’ was not declared in this scope
if (!args[3]->IsNumber()) return NanThrowError("mmap: fd (arg[3]) must be an integer (a file descriptor)", EINVAL);
^
../src/mmap-io.cc:58:129: error: return-statement with a value, in function returning 'void' [-fpermissive]
../src/mmap-io.cc:62:47: error: ‘args’ was not declared in this scope
const size_t size = args[0]->ToInteger()->Value(); // ToUint64()->Value();
^
../src/mmap-io.cc:72:58: error: ‘NanThrowError’ was not declared in this scope
return NanThrowError("mmap failed", errno);
^
../src/mmap-io.cc:72:58: error: return-statement with a value, in function returning 'void' [-fpermissive]
../src/mmap-io.cc:78:71: error: ‘NanThrowError’ was not declared in this scope
return NanThrowError("madvise() failed", errno);
^
../src/mmap-io.cc:78:71: error: return-statement with a value, in function returning 'void' [-fpermissive]
../src/mmap-io.cc:83:113: error: ‘NanNewBufferHandle’ was not declared in this scope
Local<Object> buf = NanNewBufferHandle(data, size, do_mmap_cleanup, static_cast<void*>(map_info));
^
../src/mmap-io.cc:84:35: error: ‘NanReturnValue’ was not declared in this scope
NanReturnValue(buf);
^
../src/mmap-io.cc: In function ‘Nan::NAN_METHOD_RETURN_TYPE mmap_advise(Nan::NAN_METHOD_ARGS_TYPE)’:
../src/mmap-io.cc:89:22: error: ‘NanScope’ was not declared in this scope
NanScope();
^
../src/mmap-io.cc:91:17: error: ‘args’ was not declared in this scope
if (args.Length() != 2 && args.Length() != 4) {
^
../src/mmap-io.cc:95:17: error: ‘NanThrowError’ was not declared in this scope
);
^
../src/mmap-io.cc:95:17: error: return-statement with a value, in function returning 'void' [-fpermissive]
../src/mmap-io.cc:97:18: error: ‘args’ was not declared in this scope
if (!args[0]->IsObject()) return NanThrowError("advice(): buffer (arg[0]) must be a Buffer", EINVAL);
^
../src/mmap-io.cc:97:115: error: ‘NanThrowError’ was not declared in this scope
if (!args[0]->IsObject()) return NanThrowError("advice(): buffer (arg[0]) must be a Buffer", EINVAL);
^
../src/mmap-io.cc:97:115: error: return-statement with a value, in function returning 'void' [-fpermissive]
../src/mmap-io.cc:98:18: error: ‘args’ was not declared in this scope
if (!args[1]->IsNumber()) return NanThrowError("advice(): (arg[1]) must be an integer", EINVAL);
^
../src/mmap-io.cc:98:110: error: ‘NanThrowError’ was not declared in this scope
if (!args[1]->IsNumber()) return NanThrowError("advice(): (arg[1]) must be an integer", EINVAL);
^
../src/mmap-io.cc:98:110: error: return-statement with a value, in function returning 'void' [-fpermissive]
../src/mmap-io.cc:100:39: error: ‘args’ was not declared in this scope
Local<Object> buf = args[0]->ToObject();
^
../src/mmap-io.cc:116:63: error: ‘NanThrowError’ was not declared in this scope
return NanThrowError("madvise() failed", errno);
^
../src/mmap-io.cc:116:63: error: return-statement with a value, in function returning 'void' [-fpermissive]
../src/mmap-io.cc:118:32: error: ‘NanReturnUndefined’ was not declared in this scope
NanReturnUndefined();
^
../src/mmap-io.cc: In function ‘Nan::NAN_METHOD_RETURN_TYPE mmap_sync_lib_private_(Nan::NAN_METHOD_ARGS_TYPE)’:
../src/mmap-io.cc:122:22: error: ‘NanScope’ was not declared in this scope
NanScope();
^
../src/mmap-io.cc:126:17: error: ‘args’ was not declared in this scope
if (args.Length() != 5) {
^
../src/mmap-io.cc:130:17: error: ‘NanThrowError’ was not declared in this scope
);
^
../src/mmap-io.cc:130:17: error: return-statement with a value, in function returning 'void' [-fpermissive]
../src/mmap-io.cc:132:18: error: ‘args’ was not declared in this scope
if (!args[0]->IsObject()) return NanThrowError("sync(): buffer (arg[0]) must be a Buffer", EINVAL);
^
../src/mmap-io.cc:132:113: error: ‘NanThrowError’ was not declared in this scope
if (!args[0]->IsObject()) return NanThrowError("sync(): buffer (arg[0]) must be a Buffer", EINVAL);
^
../src/mmap-io.cc:132:113: error: return-statement with a value, in function returning 'void' [-fpermissive]
../src/mmap-io.cc:135:39: error: ‘args’ was not declared in this scope
Local<Object> buf = args[0]->ToObject();
^
../src/mmap-io.cc:148:61: error: ‘NanThrowError’ was not declared in this scope
return NanThrowError("msync() failed", errno);
^
../src/mmap-io.cc:148:61: error: return-statement with a value, in function returning 'void' [-fpermissive]
../src/mmap-io.cc:150:32: error: ‘NanReturnUndefined’ was not declared in this scope
NanReturnUndefined();
^
../src/mmap-io.cc: In lambda function:
../src/mmap-io.cc:158:48: error: ‘NanNew’ was not declared in this scope
exports->ForceSet(NanNew(key), NanNew(val), property_attrs);
^
../src/mmap-io.cc: In function ‘void Init(v8::Handle<v8::Object>, v8::Handle<v8::Object>)’:
../src/mmap-io.cc:181:43: error: ‘NanNew’ was not declared in this scope
exports->ForceSet(NanNew("map"), NanNew<FunctionTemplate>(mmap_map)->GetFunction(), property_attrs);
^
../src/mmap-io.cc:181:69: error: expected primary-expression before ‘>’ token
exports->ForceSet(NanNew("map"), NanNew<FunctionTemplate>(mmap_map)->GetFunction(), property_attrs);
^
../src/mmap-io.cc:181:82: error: request for member ‘GetFunction’ in ‘mmap_map’, which is of non-class type ‘Nan::NAN_METHOD_RETURN_TYPE(Nan::NAN_METHOD_ARGS_TYPE) {aka void(const Nan::FunctionCallbackInfo<v8::Value>&)}’
exports->ForceSet(NanNew("map"), NanNew<FunctionTemplate>(mmap_map)->GetFunction(), property_attrs);
^
../src/mmap-io.cc:182:72: error: expected primary-expression before ‘>’ token
exports->ForceSet(NanNew("advise"), NanNew<FunctionTemplate>(mmap_advise)->GetFunction(), property_attrs);
^
../src/mmap-io.cc:182:88: error: request for member ‘GetFunction’ in ‘mmap_advise’, which is of non-class type ‘Nan::NAN_METHOD_RETURN_TYPE(Nan::NAN_METHOD_ARGS_TYPE) {aka void(const Nan::FunctionCallbackInfo<v8::Value>&)}’
exports->ForceSet(NanNew("advise"), NanNew<FunctionTemplate>(mmap_advise)->GetFunction(), property_attrs);
^
../src/mmap-io.cc:186:84: error: expected primary-expression before ‘>’ token
exports->ForceSet(NanNew("sync_lib_private__"), NanNew<FunctionTemplate>(mmap_sync_lib_private_)->GetFunction(), PropertyAttribute::None);
^
../src/mmap-io.cc:186:111: error: request for member ‘GetFunction’ in ‘mmap_sync_lib_private_’, which is of non-class type ‘Nan::NAN_METHOD_RETURN_TYPE(Nan::NAN_METHOD_ARGS_TYPE) {aka void(const Nan::FunctionCallbackInfo<v8::Value>&)}’
exports->ForceSet(NanNew("sync_lib_private__"), NanNew<FunctionTemplate>(mmap_sync_lib_private_)->GetFunction(), PropertyAttribute::None);
^
make: *** [Release/obj.target/mmap-io/src/mmap-io.o] Error 1
js code to reproduce:
...
const mmap = require("mmap-io")
const fd = ... // openSync, ftruncateSync, etc
const buf = mmap.map(mmap.PAGESIZE, mmap.PROT_WRITE | mmap.PROT_READ, mmap.MAP_SHARED, fd, 0x80000000);
console.log(buf); // will cause bus error;
...
The possible reason is the use of int
:
Line 124 in c751580
Ran the example use in readme through a livescript to js compile on here http://livescript.net/
node v10.14.1 Windows 10 has an error running sample code.
internal/fs/utils.js:453
throw err;
^
TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be one of type string, Buffer, or URL. Received type number
at Object.statSync (fs.js:844:3)
at Object.<anonymous> (E:\mmap\index.js:8:11)
Code:
var mmap, fs, someFile, fd, fdW, size, rxProt, priv, buffer, buffer2, wBuffer, coreStats;
mmap = require("mmap-io");
fs = require("fs");
someFile = "./foo.bar";
fd = fs.openSync(someFile, "r");
fdW = fs.openSync(someFile, "r+");
size = fs.statSync(fd).size; // This line has the error.
rxProt = mmap.PROT_READ | mmap.PROT_EXECUTE;
priv = mmap.MAP_SHARED;
buffer = mmap.map(size, rxProt, priv, fd);
buffer2 = mmap.map(size, mmap.PROT_READ, priv, fd, 0, mmap.MADV_SEQUENTIAL);
wBuffer = mmap.map(size, mmap.PROT_WRITE, priv, fdW);
mmap.advise(wBuffer, mmap.MADV_RANDOM);
mmap.sync(wBuffer);
mmap.sync(wBuffer, true);
mmap.sync(wBuffer, 0, size);
mmap.sync(wBuffer, 0, size, true);
mmap.sync(wBuffer, 0, size, true, false);
coreStats = mmap.incore(buffer);
fs.openSync
Returns an integer representing the file descriptor.
Need to use fs.fstatSync
https://github.com/ozra/mmap-io/blob/master/src/mman.h#L57
Should it be as follows?
HANDLE h = (fd != -1) ? HANDLE(_get_osfhandle(fd)) : INVALID_HANDLE_VALUE;
But anyways, this will still be broken due to this problem: nodejs/node#6369
Hi,
I'm trying to do some programming in a type script project, so can this be used in TS? any API examples?
the goal is to share a piece of memory between a C program & TS.
Many thanks,
A
The latest version (1.1.7) doesn't include the latest commits on master. Specifically, I need daeff71 to use this package in Electron 9+. Do you mind releasing a new version from master?
P.S. Thanks for this great package!
NPM install fails on windows with message:
> [email protected] postinstall C:\Users\howard\data\Workspace\node-named-pipe-test\node_modules\mmap-io
> make ls
'make' is not recognized as an internal or external command,
operable program or batch file.
npm WARN [email protected] No description
npm WARN [email protected] No repository field.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] postinstall: `make ls`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] postinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\howard\AppData\Roaming\npm-cache\_logs\2018-02-24T07_42_20_854Z-debug.log
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.