Git Product home page Git Product logo

node-webworker-threads's Issues

Segmentation fault when postMessage-ing a Buffer

e.x. new Worker(function() {}).postMessage(new Buffer(""))

It would be nice if it handled Buffers more gracefully, at a minimum not crashing, and possibly automagically converting to ArrayBuffer (maybe dependent on #18)?

PID 587 received SIGSEGV for address: 0x0
0   segfault-handler.node               0x0000000100fe990f _ZL16segfault_handleriP9__siginfoPv + 287
1   libsystem_platform.dylib            0x00007fff8af3ff1a _sigtramp + 26
2   ???                                 0x00000001054806e0 0x0 + 4383573728
3   node                                0x00000001005cb828 _ZN4node6Buffer3NewEPN2v87IsolateEPKcm + 58
4   WebWorkerThreads.node               0x0000000103bc9e9b _ZN16BSONDeserializer16DeserializeValueE8BsonTypeb + 375
5   WebWorkerThreads.node               0x0000000103bc9bba _ZN16BSONDeserializer27DeserializeDocumentInternalEb + 154
6   WebWorkerThreads.node               0x0000000103bc9b00 _ZN16BSONDeserializer19DeserializeDocumentEb + 66
7   WebWorkerThreads.node               0x0000000103bc9e4c _ZN16BSONDeserializer16DeserializeValueE8BsonTypeb + 296
8   WebWorkerThreads.node               0x0000000103bc9bba _ZN16BSONDeserializer27DeserializeDocumentInternalEb + 154
9   WebWorkerThreads.node               0x0000000103bc9b00 _ZN16BSONDeserializer19DeserializeDocumentEb + 66
10  WebWorkerThreads.node               0x0000000103bd114a _ZL9eventLoopP10typeThread + 1963
11  WebWorkerThreads.node               0x0000000103bd035b _ZL7aThreadPv + 113
12  libsystem_pthread.dylib             0x00007fff91c8a268 _pthread_body + 131
13  libsystem_pthread.dylib             0x00007fff91c8a1e5 _pthread_body + 0
14  libsystem_pthread.dylib             0x00007fff91c8841d thread_start + 13

Can't pass a function to worker when created with via new Function();

new Worker(new Function("console.log('workerTest')")) doesn't work.

I'm getting "Error opening the file [object Object]."

I imagine for some reason its treating the argument as a file instead of as a function(){}?

I want to use the function constructor as I'm creating the worker function dynamically. For now as a work around I'm using
eval('var evalWorker=function(){'+workerText+'}');
worker = new Worker(evalWorker);

Thanks.

Segfault when passing Uint8Array on Node 0.12

Hi. The following code works on Node 0.10 but segfaults on Node 0.12:

$ node -v
v0.12.7
$ cat 1.js
var Worker = require("webworker-threads").Worker;
new Worker(function() {}).postMessage(new Uint8Array(65));

Here is the gdb backtrace with debug symbols:

$ gdb node
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from node...Reading symbols from /usr/lib/debug//usr/bin/nodejs...done.
done.
(gdb) run 1.js
Starting program: /usr/bin/node 1.js
Traceback (most recent call last):
  File "/usr/share/gdb/auto-load/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19-gdb.py", line 63, in <module>
    from libstdcxx.v6.printers import register_libstdcxx_printers
ImportError: No module named 'libstdcxx'
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff6bca700 (LWP 1779)]
[New Thread 0x7ffff7fe7700 (LWP 1780)]
[New Thread 0x7ffff7fd6700 (LWP 1781)]
[New Thread 0x7ffff7fc5700 (LWP 1782)]
[New Thread 0x7ffff6199700 (LWP 1783)]
[New Thread 0x7ffff5998700 (LWP 1784)]
[New Thread 0x7ffff7fb4700 (LWP 1785)]
[New Thread 0x7ffff7fa3700 (LWP 1786)]
[New Thread 0x7ffff7f92700 (LWP 1787)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff6199700 (LWP 1783)]
0x00000000007981e3 in v8::EscapableHandleScope::EscapableHandleScope(v8::Isolate*) ()
(gdb) bt full
#0  0x00000000007981e3 in v8::EscapableHandleScope::EscapableHandleScope(v8::Isolate*) ()
No symbol table info available.
#1  0x0000000000bb0f25 in node::Buffer::New(v8::Isolate*, char const*, unsigned long) ()
No symbol table info available.
#2  0x00007ffff61b94ea in NanNewBufferHandle (data=0x12d5a46 "", size=65)
    at ../node_modules/nan/nan.h:672
No locals.
#3  0x00007ffff61aef0e in BSONDeserializer::DeserializeValue (this=0x7ffff6198a00, 
    type=BSON_TYPE_BINARY, promoteLongs=true) at ../src/bson.cc:539
        length = 65
        subType = 0
        buffer = {<v8::Handle<v8::Object>> = {val_ = 0x7ffff61999c0}, <No data fields>}
        argv = {{val_ = 0x7ffff6198900}, {val_ = 0x7ffff61b843c <v8::Value::IsNull() const+24>}}
#4  0x00007ffff61ae545 in BSONDeserializer::DeserializeDocumentInternal (this=0x7ffff6198a00, 
    promoteLongs=true) at ../src/bson.cc:427
        type = BSON_TYPE_BINARY
        name = @0x7ffff6198940: {val_ = 0x7ffff0044a38}
        value = @0x0: <error reading variable>
        returnObject = {<v8::Handle<v8::Object>> = {val_ = 0x7ffff0044a30}, <No data fields>}
#5  0x00007ffff61ae491 in BSONDeserializer::DeserializeDocument (this=0x7ffff6198c40, 
    promoteLongs=true) at ../src/bson.cc:414
        length = 81
        documentDeserializer = {bson = 0x7ffff004bdb0, pStart = 0x12d5a3b "\005data", 
          p = 0x12d5a46 "", pEnd = 0x12d5a87 ""}
#6  0x00007ffff61af17f in BSONDeserializer::DeserializeValue (this=0x7ffff6198c40, 
    type=BSON_TYPE_OBJECT, promoteLongs=true) at ../src/bson.cc:576
No locals.
#7  0x00007ffff61ae545 in BSONDeserializer::DeserializeDocumentInternal (this=0x7ffff6198c40, 
    promoteLongs=true) at ../src/bson.cc:427
        type = BSON_TYPE_OBJECT
        name = @0x7ffff6198b80: {val_ = 0x7ffff0044a18}
        value = @0x7ffff6198e10: {val_ = 0x7ffff004bdb0}
        returnObject = {<v8::Handle<v8::Object>> = {val_ = 0x7ffff0044a10}, <No data fields>}
#8  0x00007ffff61ae491 in BSONDeserializer::DeserializeDocument (this=0x7ffff6198e10, 
    promoteLongs=true) at ../src/bson.cc:414
        length = 89
        documentDeserializer = {bson = 0x7ffff004bdb0, pStart = 0x12d5a34 "\003\060", 
          p = 0x12d5a88 "", pEnd = 0x12d5a88 ""}
#9  0x00007ffff61b4ca6 in eventLoop (thread=0x11cc6e0) at ../src/WebWorkerThreads.cc:418
        bson = 0x7ffff004bdb0
        data = 0x12d5a30 "Y"
        i = -166097600
        size = 89
        deserializer = {bson = 0x7ffff004bdb0, pStart = 0x12d5a30 "Y", p = 0x12d5a89 "", 
          pEnd = 0x12d5a88 ""}
        result = {<v8::Handle<v8::Object>> = {val_ = 0x7ffff0044800}, <No data fields>}
        args = {{<v8::Handle<v8::Value>> = {val_ = 0x7ffff00448f8}, <No data fields>}, 
          {<v8::Handle<v8::Value>> = {val_ = 0x7ffff0044908}, <No data fields>}}
        len = 1
        array = {<v8::Handle<v8::Array>> = {val_ = 0x7ffff0044908}, <No data fields>}
        onError = {isolate_ = 0x7ffff00008c0, next_ = 0x0, exception_ = 0x2a4a76804181, 
          message_obj_ = 0x2a4a76804181, message_script_ = 0x2a4a76804181, 
          js_stack_comparable_address_ = 0x7ffff6198c48, message_start_pos_ = 0, 
          message_end_pos_ = 0, is_verbose_ = false, can_continue_ = true, capture_message_ = true, 
          rethrow_ = false, has_terminated_ = false}
        str = 0x12d59f0
        resultado = {<v8::Handle<v8::Value>> = {val_ = 0x7ffff00448f0}, <No data fields>}
        source = {<v8::Handle<v8::String>> = {val_ = 0x7ffff00448b8}, <No data fields>}
        script = {<v8::Handle<v8::Script>> = {val_ = 0x7ffff00448e0}, <No data fields>}
        job = 0x12d59b0
        qitem = 0x11cfc30
        scope = {isolate_ = 0x7ffff00008c0, prev_next_ = 0x0, prev_limit_ = 0x0}
        ctx = {<v8::Handle<v8::Context>> = {val_ = 0x7ffff00443c8}, <No data fields>}
        ctr = 2
        ftmpl = {<v8::Handle<v8::FunctionTemplate>> = {val_ = 0x7ffff00443b0}, <No data fields>}
        otmpl = {<v8::Handle<v8::ObjectTemplate>> = {val_ = 0x7ffff00443c0}, <No data fields>}
        fs_obj = {val_ = 0x7ffff00443f0}
---Type <return> to continue, or q <return> to quit---
        console_obj = {val_ = 0x7ffff0044480}
        dispatchEvents = {<v8::Handle<v8::Object>> = {val_ = 0x7ffff0044818}, <No data fields>}
        dispatchNextTicks = {<v8::Handle<v8::Object>> = {val_ = 0x7ffff0044858}, <No data fields>}
        _ntq = 0x7ffff0044870
        nextTickQueueLength = 0
        extensions = {name_count_ = 0, names_ = 0x0}
        global = {<v8::Handle<v8::Object>> = {val_ = 0x7ffff00443d8}, <No data fields>}
        threadObject = {<v8::Handle<v8::Object>> = {val_ = 0x7ffff0044748}, <No data fields>}
        isolate_scope = {isolate_ = 0x7ffff00008c0}
#10 0x00007ffff61b3acf in aThread (arg=0x11cc6e0) at ../src/WebWorkerThreads.cc:243
        myLocker = {has_lock_ = true, top_level_ = true, isolate_ = 0x7ffff00008c0, 
          static active_ = true}
        dummy = 0
        thread = 0x11cc6e0
#11 0x00007ffff6f98182 in start_thread (arg=0x7ffff6199700) at pthread_create.c:312
        __res = <optimized out>
        pd = 0x7ffff6199700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737322260224, 6434267845187848142, 1, 4, 
                140737322260928, 140737322260224, -6434253325479983154, -6434252500698019890}, 
              mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, 
              cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#12 0x00007ffff6cc547d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Segmentation Fault when using WebWorker-Threads during Load Testing

I'm working on load testing CPU intensive functions with both synchronous calls as well as web workers on a multi-core VM, and am running into SegFaults when using WebWorker-Threads. It runs about 20-30 requests then dumps the core.

The project is up here: https://github.com/WakeskaterX/NodeThreading

And I have a question on Stack Overflow:
http://stackoverflow.com/questions/26812646/segmentation-fault-during-high-load-concurrency-test-with-webworker-threads

I've tried running it with out the heavy process and it still faults after 20-30 requests.

I'm using Oracle VM Virtual Box 4.3.18 with a 32 bit Ubuntu 14.04 VM with 4 Cores and 4 Gigs of RAM.

The load testing command that breaks it (when the web worker section is uncommented) is:
loadtest -c 4 -t 20 http://localhost:3030/fib?num=30

Issue installing from npm on Windows 8.1

When installing with npm on Windows 8.1 I get the following error:

C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V110\Microsoft.Cpp.Platform.targets(44,5): error MSB8020: The builds tools for v120 (Platform Toolset = 'v120') cannot be found. To build using the v120 build tools, either click the Project menu or right-click the solution, and then select "Update VC++ Projects...". Install v120 to build using the v120 build tools. [S:\dev\nodegyptest\node_modules\webworker-threads\build\WebWorkerThreads.vcxproj]

I am not sure how to change the node-gyp build settings so a work around for anyone experiencing this issue follows:

  1. Pull the source
  2. Navigate a command prompt to the source directory
  3. type npm install nan
  4. type node-gyp configure
  5. type node-gyp build /tv:12.0

Builds without issue for me using this method.
Setup:
Windows 8.1
Visual Studio 2013
Python 2.7.9

Segfault when using worker

Hello,

I'm using this module and I'm having an extreme level of difficulty trying to figure out why I'm getting segmentation faults. I'm trying to use this module to be able to execute a regular expression, but with a timeout. The regular expressions are provided by users and thus we need to protect against users who put in a malicious regex that takes a long time to execute.

No matter what I do and how I modify the code, it always seems to crash with exactly the same segmentation fault error:

PID 27647 received SIGSEGV for address: 0x7e68
/home/bradley/sensibill-api/node_modules/segfault-handler/build/Release/segfault-handler.node(+0x10e5)[0x7f63535fd0e5]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7f635a50c340]
/usr/local/bin/node[0x6c16ef]
/usr/local/bin/node(ZN2v85Utils16ReportApiFailureEPKcS2+0x12)[0x6c2302]
/usr/local/bin/node(_ZN2v87Isolate7DisposeEv+0x25)[0x6d8ac5]
/home/bradley/sensibill-api/node_modules/webworker-threads/build/Debug/WebWorkerThreads.node(+0x17377)[0x7f63531db377]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7f635a504182]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f635a23147d]

The address is always EXACTLY the same - 0x7e68. It never changes no matter what I do to the code, its always segfaulting on that address.

Code looks like this:

var Threads = require('webworker-threads'),
Worker = Threads.Worker;

var globalWorker = null;

/**

  • This function executes the given Regex asynchronously using a background thread.
  • @param regex

  • @param str

  • @param callback
    */
    module.exports.match = function (regex, str, callback)
    {
    var error = null;
    var match = null;
    if (!regex)
    {
    callback(error, match);
    return;
    }

    var timeoutID = null;

    globalWorker = new Worker(matchWorker);
    globalWorker.onmessage = function (evt)
    {
    // When the worker says its loaded start a timer.
    if (evt.data == "onload")
    {
    timeoutID = setTimeout(function ()
    {
    globalWorker.terminate();
    callback("timeout", match);
    }, 250);
    }
    else
    {
    match = evt.data.match;
    error = evt.data.error;
    clearTimeout(timeoutID);
    globalWorker.terminate();
    callback(error, match);
    }
    };
    globalWorker.postMessage({regex: regex, str: str});
    };

function matchWorker ()
{
var self = this;
this.onmessage = function (evt)
{
"use strict";

    var regex = new RegExp(evt.data.regex);
    var str = evt.data.str;
    var error = null;
    var match;

    // Let the callee know we're loaded
    self.postMessage("onload");

    match = regex.exec(str);

    self.postMessage({error: error, match: match});
    self.close();
}

}

threadpool.any.emit causes fatal application error

When i use
threadpool.any.emit('string', object);

I get the following error, but if i use threadpool.all.emit, it works.

Could anybody provide any info? Thanks

undefined:5
tion(e,d){var f;nextJob(t);f=job.cbOrData;if(f){return job.cbOrData.call(t,e,d
^
TypeError: Object # has no method 'call'
at null. (:5:170)

Use WebSocket from within worker?

Hi

Firstly, great work. I have a huge OCD for not using 3rd party libraries, but your 'node-webworker-threads' feels like it should be part of the core node.

As context, I've written a browser based wep-app that relies on WebWorkers, and now I'm able to use your library to simulate a headless client at the server side with node, with almost no coding changes to the App. A real breakthrough for me!

I've come up against a challenge though, as I also require WebSockets - initiated from within the WebWorker.

Inside my worker code I have :
connection = new WebSocket( url, protocol);

I've run some successfull tests from a node main thread using:
var WebSocket = require('websocket').client; [from https://github.com/Worlize/WebSocket-Node]

However I can't use require to pull in modules into a node-webworker.

Can you let me know your recommended approach to get around this?

Many thanks.

Does not compile on 0.11.14

node-gyp rebuild

child_process: customFds option is deprecated, use stdio instead.
make: Entering directory `/home/marco.minetti/Desktop/test/node_modules/webworker-threads/build'
CXX(target) Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o
In file included from ../src/WebWorkerThreads.cc:6:0:
/home/marco.minetti/.node-gyp/0.11.14/src/node.h:377:43: error: overloaded function with no contextual type information
(node::addon_register_func) (regfunc),
^
/home/marco.minetti/.node-gyp/0.11.14/src/node.h:408:3: note: in expansion of macro ‘NODE_MODULE_X’
NODE_MODULE_X(modname, regfunc, NULL, 0)
^
../src/WebWorkerThreads.cc:971:1: note: in expansion of macro ‘NODE_MODULE’
NODE_MODULE(WebWorkerThreads, Init)
^
make: *** [Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o] Error 1

worker.onmessage is not executed

Hi,
first, thankyou for your module, it has the best interface ever.. but I have a problem, in my code I have this (inside a foreach):
.........
if(l !== null){
var worker = new Worker(function(){
this.onmessage = function(event){
var level = event.data.level;
console.log(level);
var p = dowork(level); //dowork defined first
console.log(p);
postMessage(s);
console.log("Processing level: "+level);
self.close();
}});
worker.onmessage = function(event) {
console.log("I'm here");
pile.push(event.data);
};
worker.postMessage({'level':l.level});
}
......

worker.onmessage is not receiving the message from worker on the Master thread, but all the logging tell me that worker performed it's task correctly (receives data from main thread and so on).. and closes.

Is there a bug, or am I missing something?

CHECK(allow_empty_handle || that != __null) failed

I am trying to pass an object to a worker

function update(data) {
  var worker = new Worker(function() {
    postMessage('start processing');
    this.onmessage = function(event) {
      // ...
      postMessage('finished processing');
      self.close();
    };
  });
  worker.onmessage = function(event) {
    console.log("Worker said : " + event.data);
  };

  worker.postMessage(data);
}

when running on small strings / objects it works fine, when trying to load my big object in
(couple of kb) I get

#
# Fatal error in ../deps/v8/src/api.h, line 297
# CHECK(allow_empty_handle || that != __null) failed
#

any idea ?

Node webkit : mach-o, but wrong architecture

Hi,

I try to use webworker-threads on my node webkit project.

After a npm-install, I got the following error when trying to run nw . :
Uncaught Error: dlopen(.../WebWorkerThreads.node, 1): no suitable image found. Did find:
/.../build/Release/WebWorkerThreads.node: mach-o, but wrong architecture", source: module.js (346)

I tried to rebuild with nw-gyp rebuild.

I also tried to modify the binding.gyp and add the following flags '-arch i386', '-arch x86_64', but still the same error.

But still the same error.

Do you have any idea ?

ReferenceError: fibo is not defined

I have a small file threads.js based on this example the content of the file is:

var threads = require('webworker-threads'),
    sleep = require('sleep').sleep;

var random = function(low, high) {
    return Math.floor(Math.random() * (high - low + 1) + low);
};
function fibo(n, low, high) {
    var rnd = random(low, high);
    sleep(rnd);
    return n > 1 ? fibo(n - 1) + fibo(n - 2) : 1;
}

var threadOne = threads.create(),
    threadTwo = threads.create(),
    threadThree = threads.create();
threadOne.eval('fibo(10, 60, 120)', function(err, result) {
    if (err) { console.error(err); }

    console.log('[ thread-1 ] result: %s', result);
});
threadTwo.eval('fibo(10, 60, 120)', function(err, result) {
    if (err) { console.error(err); }

    console.log('[ thread-2 ] result: %s', result);
});
threadThree.eval('fibo(10, 60, 120)', function(err, result) {
    if (err) { console.error(err); }

    console.log('[ thread-3 ] result: %s', result);
});

When I execute node threads.js:

vagrant@vagrant-ubuntu-trusty-64:~/nodejs$ node threads.js
[Error: ReferenceError: fibo is not defined]
[ thread-2 ] result: null
[Error: ReferenceError: fibo is not defined]
[ thread-3 ] result: null
[Error: ReferenceError: fibo is not defined]
[ thread-1 ] result: null

For every thread I receive the error [Error: ReferenceError: fibo is not defined] but as we can see I have the function declared as the example and for an experiment, I changed to var fibo = function(n, low, high) { ... } still did not worked.

Install fails with latest node/npm on Mac (v0.5.2 of lib)

I get a failure during node-gyp rebuild:

CXX(target) Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o
../src/WebWorkerThreads.cc:943:35: error: function definition is not allowed
      here
void Init (Handle<Object> target) {
                                  ^
../src/WebWorkerThreads.cc:975:1: error: expected unqualified-id
NODE_MODULE(WebWorkerThreads, Init)
^
/Users/Matthew/.node-gyp/0.10.13/src/node.h:224:10: note: expanded from macro
      'NODE_MODULE'
  extern "C" {                                                        \
         ^
../src/WebWorkerThreads.cc:979:3: error: expected '}'
*/
  ^
../src/WebWorkerThreads.cc:940:57: note: to match this '{'
void Init (Handle<Object> target, Handle<Value> module) {
                                                        ^
3 errors generated.
make: *** [Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o] Error 1
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:267:23)
gyp ERR! stack     at ChildProcess.EventEmitter.emit (events.js:98:17)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (child_process.js:789:12)
gyp ERR! System Darwin 14.0.0
gyp ERR! command "node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"

...etc

Message posted from worker to main thread is lost.

Hi,
I have an application which spawns several Workers at a time and posting "init" messages from the main thread right away. Then the worker responds "initdone".
I see that main thread is not getting some responses ("initdone") from Workers (I see thru console.log that Workers are receiving the main's thread message "init" and using the postMessage to respond back) . Is it possible that the queue is not working properly?
Thanks
Jose
PS: trying to isolate an example with no luck yet. The app where I see this behavior is complex.

Build failing on Mac OSX Yosemite / io.js 2.0.0

Installing webworker-threads fails when using io.js 2.0.0:

> [email protected] install /home/travis/build/andywer/threadpool-js/node_modules/webworker-threads
> node-gyp rebuild
make: Entering directory `/home/travis/build/andywer/threadpool-js/node_modules/webworker-threads/build'
  CXX(target) Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o
In file included from ../node_modules/nan/nan_new.h:190:0,
                 from ../node_modules/nan/nan.h:74,
                 from ../src/WebWorkerThreads.cc:13:
../node_modules/nan/nan_implementation_12_inl.h: In static member function ‘static NanIntern::FactoryBase<v8::Signature>::return_t NanIntern::Factory<v8::Signature>::New(NanIntern::Factory<v8::Signature>::FTH, int, NanIntern::Factory<v8::Signature>::FTH*)’:
../node_modules/nan/nan_implementation_12_inl.h:181:76: error: no matching function for call to ‘v8::Signature::New(v8::Isolate*, NanIntern::Factory<v8::Signature>::FTH&, int&, NanIntern::Factory<v8::Signature>::FTH*&)’
../node_modules/nan/nan_implementation_12_inl.h:181:76: note: candidate is:
/home/travis/.node-gyp/2.0.0/deps/v8/include/v8.h:4188:27: note: static v8::Local<v8::Signature> v8::Signature::New(v8::Isolate*, v8::Handle<v8::FunctionTemplate>)
/home/travis/.node-gyp/2.0.0/deps/v8/include/v8.h:4188:27: note:   candidate expects 2 arguments, 4 provided
In file included from ../src/WebWorkerThreads.cc:38:0:
../src/bson.cc: In static member function ‘static void BSON::BSONDeserialize(const v8::FunctionCallbackInfo<v8::Value>&)’:
../src/bson.cc:796:45: warning: ‘ssize_t node::DecodeBytes(v8::Handle<v8::Value>, node::encoding)’ is deprecated (declared at /home/travis/.node-gyp/2.0.0/src/node.h:297): Use DecodeBytes(isolate, ...) [-Wdeprecated-declarations]
../src/bson.cc:803:41: warning: ‘ssize_t node::DecodeWrite(char*, size_t, v8::Handle<v8::Value>, node::encoding)’ is deprecated (declared at /home/travis/.node-gyp/2.0.0/src/node.h:310): Use DecodeWrite(isolate, ...) [-Wdeprecated-declarations]
In file included from ../src/WebWorkerThreads.cc:38:0:
../src/bson.cc: In static member function ‘static void BSON::BSONSerialize(const v8::FunctionCallbackInfo<v8::Value>&)’:
../src/bson.cc:897:73: warning: ‘v8::Local<v8::Value> node::Encode(const void*, size_t, node::encoding)’ is deprecated (declared at /home/travis/.node-gyp/2.0.0/src/node.h:278): Use Encode(isolate, ...) [-Wdeprecated-declarations]
../src/WebWorkerThreads.cc: In function ‘void Callback(uv_async_t*, int)’:
../src/WebWorkerThreads.cc:540:37: warning: ‘void node::FatalException(const v8::TryCatch&)’ is deprecated (declared at /home/travis/.node-gyp/2.0.0/src/node.h:262): Use FatalException(isolate, ...) [-Wdeprecated-declarations]
make: *** [Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o] Error 1

See travis log.

Hint: The failure seems NOT to be related to #55. At least not on first glance ;)

Use sqlite3 in web worker doesn't work

hi,
i'm trying to access to a sqlite3 db inside the web worker, i use this sqlite3 library:

https://github.com/mapbox/node-sqlite3

this is what i do:

var Worker = require('webworker-threads').Worker;
var worker = new Worker(function(){
  postMessage("I'm working before postMessage('ali').");
  this.onmessage = function(event) {
    console.log("before sqlite3");
    var sqlite3 = require('sqlite3').verbose();
    var db = new sqlite3.Database('MyDB.db');
    console.log("after sqlite3");
  };
});

worker.postMessage('test');

this is the result:

before sqlite3

and then block, how i can solve the problem? i do something wrong?

error on npm install during "node-gyp rebuild"

npm http GET https://registry.npmjs.org/webworker-threads
npm http 304 https://registry.npmjs.org/webworker-threads

[email protected] install /Users/vivekanandan/Source/spawn/node_modules/webworker-threads
node-gyp rebuild

CXX(target) Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o
In file included from ../src/WebWorkerThreads.cc:36:
../src/bson.cc:199:18: error: use of undeclared identifier 'NanPersistentToLocal'
if(object->Has(NanPersistentToLocal(bson->_bsontypeString)))
^
../src/bson.cc:202:7: error: use of undeclared identifier 'NanPersistentToLocal'
if(NanPersistentToLocal(bson->longString)->StrictEquals(constructorString))
^
../src/bson.cc:205:30: error: use of undeclared identifier 'NanPersistentToLocal'
this->WriteInt32(object, NanPersistentToLocal(bson->_longLowString));
^
../src/bson.cc:206:30: error: use of undeclared identifier 'NanPersistentToLocal'
this->WriteInt32(object, NanPersistentToLocal(bson->_longHighString));
^
../src/bson.cc:208:12: error: use of undeclared identifier 'NanPersistentToLocal'
else if(NanPersistentToLocal(bson->timestampString)->StrictEquals(constructorString))
^
../src/bson.cc:211:30: error: use of undeclared identifier 'NanPersistentToLocal'
this->WriteInt32(object, NanPersistentToLocal(bson->_longLowString));
^
../src/bson.cc:212:30: error: use of undeclared identifier 'NanPersistentToLocal'
this->WriteInt32(object, NanPersistentToLocal(bson->_longHighString));
^
../src/bson.cc:214:12: error: use of undeclared identifier 'NanPersistentToLocal'
else if(NanPersistentToLocal(bson->objectIDString)->StrictEquals(constructorString))
^
../src/bson.cc:217:33: error: use of undeclared identifier 'NanPersistentToLocal'
this->WriteObjectId(object, NanPersistentToLocal(bson->_objectIDidString));
^
../src/bson.cc:219:12: error: use of undeclared identifier 'NanPersistentToLocal'
else if(NanPersistentToLocal(bson->binaryString)->StrictEquals(constructorString))
^
../src/bson.cc:223:35: error: use of undeclared identifier 'NanPersistentToLocal'
uint32_t length = object->Get(NanPersistentToLocal(bson->_binaryPositionString))->Uint32Value();
^
../src/bson.cc:224:43: error: use of undeclared identifier 'NanPersistentToLocal'
Local bufferObj = object->Get(NanPersistentToLocal(bson->_binaryBufferString))->ToObject();
^
../src/bson.cc:227:29: error: use of undeclared identifier 'NanPersistentToLocal'
this->WriteByte(object, NanPersistentToLocal(bson->_binarySubTypeString)); // write subtype
^
../src/bson.cc:229:20: error: use of undeclared identifier 'NanPersistentToLocal'
if(object->Get(NanPersistentToLocal(bson->_binarySubTypeString))->Int32Value() == 0x02) {
^
../src/bson.cc:235:12: error: use of undeclared identifier 'NanPersistentToLocal'
else if(NanPersistentToLocal(bson->doubleString)->StrictEquals(constructorString))
^
../src/bson.cc:238:31: error: use of undeclared identifier 'NanPersistentToLocal'
this->WriteDouble(object, NanPersistentToLocal(bson->_doubleValueString));
^
../src/bson.cc:240:12: error: use of undeclared identifier 'NanPersistentToLocal'
else if(NanPersistentToLocal(bson->symbolString)->StrictEquals(constructorString))
^
../src/bson.cc:243:49: error: use of undeclared identifier 'NanPersistentToLocal'
this->WriteLengthPrefixedString(object->Get(NanPersistentToLocal(bson->_symbolValueString))->...
^
../src/bson.cc:245:12: error: use of undeclared identifier 'NanPersistentToLocal'
else if(NanPersistentToLocal(bson->codeString)->StrictEquals(constructorString))
^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make: *** [Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: make failed with exit code: 2
gyp ERR! stack at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:267:23)
gyp ERR! stack at ChildProcess.EventEmitter.emit (events.js:98:17)
gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:797:12)
gyp ERR! System Darwin 13.1.0
gyp ERR! command "node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/vivekanandan/Source/spawn/node_modules/webworker-threads
gyp ERR! node -v v0.10.26
gyp ERR! node-gyp -v v0.12.2
gyp ERR! not ok
npm ERR! [email protected] install: node-gyp rebuild
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the webworker-threads package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node-gyp rebuild
npm ERR! You can get their info via:
npm ERR! npm owner ls webworker-threads
npm ERR! There is likely additional logging output above.

npm ERR! System Darwin 13.1.0
npm ERR! command "node" "/usr/local/bin/npm" "install" "webworker-threads"
npm ERR! cwd /Users/vivekanandan/Source/spawn
npm ERR! node -v v0.10.26
npm ERR! npm -v 1.4.3
npm ERR! code ELIFECYCLE
npm ERR!
npm ERR! Additional logging details can be found in:

ArrayBuffer is not accessible to a worker

I cannot use ArrayBuffer from a worker:

worker.js:

console.log(typeof ArrayBuffer);

main.js:

console.log(typeof ArrayBuffer);
Worker = require('webworker-threads').Worker;
new Worker('./worker.js');

Then:

> nodejs main.js

prints

function
undefined

NodeJS v0.10.17, webworker-threads v0.4.7

Avoid serializing Transferable objects

I'm attempting to use the library to run a few background threads in a node app, each processing an 8-32MB Uint8Array. Running with 4 workers and 4MB of data each, it currently takes more than 15 minutes to post the initial messages (before it can even start processing). The performance is rather bad, since typed arrays are supposed to be Transferable.

So far as I can tell, each array is being serialized repeatedly when being transferred between threads. This appears to be happening here (thanks to apsillers for finding that). Producing those strings peaks around 7GB of RES memory.

My test app consists of two files:

  • index.js:
var Worker = require('webworker-threads').Worker;

var workerCount = 4, finishedCount = 0, bufferSize = (1024 * 1024 * 4);
var workers = {};
var buffers = {};

function messageReceiver(event) {
  var data = event.data;
  var id = data.id;
  var buffer = data.buffer;
  console.log('Received', buffer[0], 'from', id);
  workers[id].terminate();
  if (++finishedCount === workerCount) {
    console.log('All workers finished.');
  }
}

for (var i = 0; i < workerCount; ++i) {
  console.log('Spawning worker', i);
  var worker = new Worker('worker.js');
  worker.onmessage = messageReceiver;
  workers[i] = worker;
}
console.log('All workers spawned.');

for (var i = 0; i < workerCount; ++i) {
  console.log('Allocating buffer', i);
  buffers[i] = new Uint32Array(new Buffer(bufferSize));
}
console.log('All buffers allocated.');

for (var i = 0; i < workerCount; ++i) {
  console.log('Launching worker', i);
  var w = workers[i];
  var a = buffers[i];
  var data = {
    id: i,
    buffer: a.buffer
  };
  w.postMessage(data, [data.buffer]);
}
console.log('All workers launched.');
  • worker.js:
function finishWorker(id, buffer) {
  console.log('Thread ' + id + ' finished filling buffer.');
  var data = {
    id: id,
    buffer: buffer
  };
  postMessage(data, [data.buffer]);
}

function fillBuffer(buffer) {
  for (var i = 0; i < buffer.byteLength; ++i) {
    if (i % (1024 * 1024) === 0) {
      console.log('Thread ' + id + ' filled ' + i + ' bytes');
    }
    buffer[i] = ((Math.random() * 255) | 0) % 255;
  }
}

this.onmessage = function (event) {
  var id = event.data.id, buffer = event.data.buffer;
  console.log('Thread ' + id + ' received buffer of ' + buffer.byteLength);
  fillBuffer(buffer);
  finishWorker(id, buffer);
};

Can't compile on Ubuntu, but can in OS X?

I'm investigating this library as a replacement for child processes for most situations, but I've run into a problem while testing the library -- an Ubuntu server with build-essential and node-gyp can't compile the code.

> [email protected] install /home/damocles/webworker-childproc-showdown/node_modules/webworker-threads
> node-gyp rebuild

make: Entering directory `/home/damocles/webworker-childproc-showdown/node_modules/webworker-threads/build'
  CXX(target) Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o
../src/WebWorkerThreads.cc: In function ‘char* readFile(v8::Handle<v8::String>)’:
../src/WebWorkerThreads.cc:649:25: warning: ignoring return value of ‘size_t fread(void*, size_t, size_t, FILE*)’, declared with attribute warn_unused_result [-Wunused-result]
  SOLINK_MODULE(target) Release/obj.target/WebWorkerThreads.node
/usr/bin/ld: errno: TLS definition in /lib/x86_64-linux-gnu/libc.so.6 section .tbss mismatches non-TLS reference in Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o
/lib/x86_64-linux-gnu/libc.so.6: could not read symbols: Bad value
collect2: ld returned 1 exit status
make: *** [Release/obj.target/WebWorkerThreads.node] Error 1
make: Leaving directory `/home/damocles/webworker-childproc-showdown/node_modules/webworker-threads/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/local/lib/node_modules/node-gyp/lib/build.js:232:23)
gyp ERR! stack     at ChildProcess.EventEmitter.emit (events.js:91:17)
gyp ERR! stack     at Process._handle.onexit (child_process.js:674:10)
gyp ERR! System Linux 3.2.0-23-virtual
gyp ERR! command "node" "/usr/local/bin/node-gyp" "rebuild"
gyp ERR! cwd /home/damocles/webworker-childproc-showdown/node_modules/webworker-threads
gyp ERR! node -v v0.8.8
gyp ERR! node-gyp -v v0.8.0
gyp ERR! not ok 
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! `sh "-c" "node-gyp rebuild"` failed with 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the webworker-threads package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-gyp rebuild
npm ERR! You can get their info via:
npm ERR!     npm owner ls webworker-threads
npm ERR! There is likely additional logging output above.

npm ERR! System Linux 3.2.0-23-virtual
npm ERR! command "node" "/usr/bin/npm" "install"
npm ERR! cwd /home/damocles/webworker-childproc-showdown
npm ERR! node -v v0.8.8
npm ERR! npm -v 1.1.59
npm ERR! code ELIFECYCLE
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /home/damocles/webworker-childproc-showdown/npm-debug.log
npm ERR! not ok code 0

According to this forum, this is the offending line of code, and the solution is to replace that line of code with #include <errno.h>

I will be forking and testing this. If this fixes it (and doesn't screw up OS X) I will make a PR.

How can I require modules in Worker instances?

When I tried to require some native modules in threads like this:

// Generated by CoffeeScript 1.6.2
var assert, equal;

assert = require('assert');

equal = assert.equal;

self.onmessage = function(e) {
  return self.postMessage({
    test: 'QAQ'
  });
};

and worker.postMessage doesn't work

// Generated by CoffeeScript 1.6.2
var Worker;

Worker = require('webworker-threads').Worker;

setInterval(function() {
  var worker;

  worker = new Worker('client.js');
  worker.onmessage = function() {//this function has never called since the thread requires modules
    return console.dir(arguments);
  };
  return worker.postMessage({
    foo: 'bar'
  });
}, 1000);

Build failing on Mac OSX Yosemite / io.js 1.8.1

I can't tell if this is a problem with node-gyp, or node-webworker-threads.

Here is the relevant log:

> [email protected] install /Users/will/Code/happy/node_modules/webworker-threads
> node-gyp rebuild

WARN install got an error, rolling back install
gyp ERR! configure error 
gyp ERR! stack Error: 404 status code downloading tarball
gyp ERR! stack     at Request.<anonymous> (/Users/will/.nvm/versions/io.js/v1.8.1/lib/node_modules/npm/node_modules/node-gyp/lib/install.js:246:14)
gyp ERR! stack     at emitOne (events.js:82:20)
gyp ERR! stack     at Request.emit (events.js:166:7)
gyp ERR! stack     at Request.onRequestResponse (/Users/will/.nvm/versions/io.js/v1.8.1/lib/node_modules/npm/node_modules/request/request.js:1156:10)
gyp ERR! stack     at emitOne (events.js:77:13)
gyp ERR! stack     at ClientRequest.emit (events.js:166:7)
gyp ERR! stack     at HTTPParser.parserOnIncomingClient (_http_client.js:412:21)
gyp ERR! stack     at HTTPParser.parserOnHeadersComplete (_http_common.js:88:23)
gyp ERR! stack     at Socket.socketOnData (_http_client.js:302:20)
gyp ERR! stack     at emitOne (events.js:77:13)
gyp ERR! System Darwin 14.3.0
gyp ERR! command "/Users/will/.nvm/versions/io.js/v1.8.1/bin/iojs" "/Users/will/.nvm/versions/io.js/v1.8.1/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/will/Code/happy/node_modules/webworker-threads
gyp ERR! node -v v1.8.1
gyp ERR! node-gyp -v v1.0.3
gyp ERR! not ok 
npm ERR! Darwin 14.3.0
npm ERR! argv "/Users/will/.nvm/versions/io.js/v1.8.1/bin/iojs" "/Users/will/.nvm/versions/io.js/v1.8.1/bin/npm" "install" "webworker-threads"
npm ERR! node v1.8.1
npm ERR! npm  v2.8.4
npm ERR! code ELIFECYCLE

npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script 'node-gyp rebuild'.
npm ERR! This is most likely a problem with the webworker-threads package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-gyp rebuild
npm ERR! You can get their info via:
npm ERR!     npm owner ls webworker-threads
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/will/Code/happy/npm-debug.log

My googling has shown that this might be an issue with the latest versions of node and io and node-gyp, although I have seen other packages that have submitted fixes for similar issues.

Support for Node 0.11.x

The gyp build fails on node 0.11.x. 0.10.9 builds fine.

> [email protected] install /tmp/wwtest/node_modules/webworker-threads
> node-gyp rebuild

make: Entering directory `/tmp/wwtest/node_modules/webworker-threads/build'
  CXX(target) Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o
[...]
../src/WebWorkerThreads.cc: In function ‘typeThread* isAThread(v8::Handle<v8::Object>)’:
../src/WebWorkerThreads.cc:152:39: error: ‘class v8::Object’ has no member named ‘GetPointerFromInternalField’
[...]

For the full log see this gist.

New binding.gyp file breaks install + fix

There's a small error in your new binding.gyp file that breaks my npm install process (node 8.20, ubuntu):
'-fno-exceptions -DV8_USE_UNSAFE_HANDLES' should be '-fno-exceptions', '-DV8_USE_UNSAFE_HANDLES'
(sorry, no pull request, seemed a bit too much effort for this small thing)

Thread exhausting CPU doing nothing?

When I run a loop inside thread, thread still keeps running at 50% cpu even after loop finished.
Node v0.10.26, Windows 8.1

Is there explicit need to close the Worker/Thread? Thread should be idle, waiting for another onmessage and not exhausting processor.

Use case:

var Threads = require('webworker-threads');

var worker = new Threads.Worker(function() {
    function loop() {
        var a = 0;
        for(var i = 0; i < 900000000; i++) {
            a = Math.pow(2, 2) + i;
            a /= 4;
        }
        return a;
    }
    function fibo (n) {
        return n > 1 ? fibo(n - 1) + fibo(n - 2) : 1;
    }
    this.onmessage = function (event) {
        // works ok
        //fibo(44);
        //console.log('thread finished');

        // does not work: if not closed, keeps exhausting CPU
        loop();
        console.log('thread finished');
        //self.close(); // I dont want to close, just keep in the background to reuse
    };
});

console.log('starting thread');
worker.postMessage();

It seems like it locks on cpu usage end keep "using" :) Once loop exhaust CPU to > 50%, it stays there even when loop has finished.
Executing fibo function works as expected, both functions exhaust CPU and last several seconds, but loop behaves wrong.

Thread examples work, Worker doesn't

Greetings

I'm running the following example:

var Worker = require('webworker-threads').Worker;
  var fibo = new Worker(function() {
        function fibo (n) {
          return n > 1 ? fibo(n - 1) + fibo(n - 2) : 1;
        }
        onmessage = function (event) {
          postMessage(fibo(event.data));
        }
  });
  fibo.onmessage = function (event) {
    res.end('fib(40) = ' + event.data);
  };
  fibo.postMessage(40);

The Worker is created, but postMessage mostly does nothing, i.e. the onmessage is not being executed, for whatever reasons.
I've tried many of the Threads examples though, and they work fine.

What am I doing wrong?

Thanks in advance for some help!

Thread running a function that returns a promise is considered unactive

When using a thread to execute a function that returns a promise,
on the function's immediate return the thread is considered to be terminated,
therefore the threads' pool is initializing more parallel threads then it is supposed to :(

Besides, I fail to get the promise returned, it appears as " 'undefined' "

Solution? workaround?

THANKS

Error with npm install: No Xcode is selected

No dependencies are listed on the main repo page however I've encountered the following error when trying to install. I do not currently have Xcode installed on my Mac:

bash-3.2$ npm install webworker-threads
npm WARN package.json [email protected] No repository field.
npm http GET https://registry.npmjs.org/webworker-threads
npm http 200 https://registry.npmjs.org/webworker-threads
npm http GET https://registry.npmjs.org/webworker-threads/-/webworker-threads-0.4.7.tgz
npm http 200 https://registry.npmjs.org/webworker-threads/-/webworker-threads-0.4.7.tgz

[email protected] install /Users//Documents/git/node-geoservices-adaptor/node_modules/webworker-threads
node-gyp rebuild

gyp http GET http://nodejs.org/dist/v0.10.15/node-v0.10.15.tar.gz
gyp http 200 http://nodejs.org/dist/v0.10.15/node-v0.10.15.tar.gz
xcode-select: Error: No Xcode is selected. Use xcode-select -switch , or see the xcode-select manpage (man xcode-select) for further information.

gyp: Error 2 running xcodebuild
gyp ERR! configure error
gyp ERR! stack Error: gyp failed with exit code: 1
gyp ERR! stack at ChildProcess.onCpExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:424:16)
gyp ERR! stack at ChildProcess.EventEmitter.emit (events.js:98:17)
gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:789:12)
gyp ERR! System Darwin 12.4.0
gyp ERR! command "node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users//Documents/git/node-geoservices-adaptor/node_modules/webworker-threads
gyp ERR! node -v v0.10.15
gyp ERR! node-gyp -v v0.10.6
gyp ERR! not ok
npm ERR! weird error 1
npm ERR! not ok code 0
bash-3.2$

idleThreads

Hi

Ive setup a worker thread to run that roughly takes around 3 seconds to process.
In a timer ive got another process that every 500milliseconds runs and determines how many threads are available and if there are any available repeat the first step again.

After the first iteration of the 500minisecond timer, while the first worker is still running my pool shows 1 idle thread even though it is still executing its code (using thread.nextTick so the application doesnt lock)

Am i looking at this wrong or is there an issue with that?

Thanks

Regards

Craig

Node 0.12 on OS X: Unexpected thread counts

Hi, I am running the example code listed but with fibo(30) instead of fibo(40) and using a valid port number:

var Worker = require('webworker-threads').Worker;
require('http').createServer(function (req,res) {
  var fibo = new Worker(function() {
    function fibo (n) {
      return n > 1 ? fibo(n - 1) + fibo(n - 2) : 1;
    }
    this.onmessage = function (event) {
      postMessage(fibo(event.data));
    }
  });
  fibo.onmessage = function (event) {
    res.end('fib(30) = ' + event.data);
  };
  fibo.postMessage(30);
}).listen(8080);

Steps to reproduce:

  1. Using OSX 10.10.2 and node 0.12.0, start server using code listed above
  2. Observe node process is using 5 threads
  3. Using Chrome, navigate to localhost:8080
  4. Observe node process is using 17 threads, an increase of twelve
  5. Click refresh (at localhost:8080)
  6. Observe node process using 27 threads
  7. Continue to repeat steps 5 & 6, observing an increase of ten threads for each refresh.

Although this Worker api is not explicitly configured using a thread pool, I do expect the thread count to start off with some kind of default pool size and remain there. Both the size of the thread count increase is unexpected and as is the behavior of a never-ending increase in count.

This is just a basic example that serves to show the api; however this behavior really precludes the use of this lib in a long-lived application. It looks like there is fine-grained control over threading using some of the other apis. I am interested to see a better usage of the Worker api, that prevents this behavior using the steps listed, together with an explanation for what I am observing.

Thanks!

Regression in 0.4.3

On OS X using 0.4.3 every file I try to load gives me Error reading the file some_file.js

This appears to be caused by the most recent commit because running git checkout HEAD~1 && npm install and then ln -s that into the repo, it loads perfectly fine.

I suspect an off-by-one error, here, and will make another PR if I can fix it, but I wouldn't mind a simple revert since the OS failing to provide the entire file means the server is probably going to fail, anyways, and this sort of check is overkill for a dynamic language where the developer has no control over memory usage.

Also, in the future could you increment the version number and keep the old one in NPM, though, even if you think its broken? Starting today I want to use webworker-threads in production, but having what I'm installing change after I've specified a hardwired version number is not fun. (Caught this by dumping my node_modules directory and re-installing with --production from scratch before merging my internal PR.)

I'd rather stick with a version with known issues that I know isn't affecting my code and then upgrade to the better version after testing than be automatically updated and not know about it. :)

Thanks again,

David

Module fails to compile on Windows 7 with both Visual Studio Express 2012 and 2010

$ npm install --save webworker-threads --msvs_version=2012
npm WARN package.json [email protected] No repository field.

> [email protected] install c:\Users\amcwatters\Documents\Repositories\api
-v1\node_modules\webworker-threads
> node-gyp rebuild


c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\webworker-threads
>node "c:\nodejs\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\
bin\node-gyp.js" rebuild
Building the projects in this solution one at a time. To enable parallel build,
please add the "/m" switch.
  WebWorkerThreads.cc
C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\include\xlocale(336): wa
rning C4530: C++ exception handler used, but unwind semantics are not enabled.
Specify /EHsc [c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\w
ebworker-threads\build\WebWorkerThreads.vcxproj]
c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\webworker-thread
s\node_modules\nan\nan.h(1185): warning C4267: '=' : conversion from 'size_t' t
o 'int', possible loss of data [c:\Users\amcwatters\Documents\Repositories\api-
v1\node_modules\webworker-threads\build\WebWorkerThreads.vcxproj]
c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\webworker-thread
s\node_modules\nan\nan.h(1202): warning C4267: '=' : conversion from 'size_t' t
o 'int', possible loss of data [c:\Users\amcwatters\Documents\Repositories\api-
v1\node_modules\webworker-threads\build\WebWorkerThreads.vcxproj]
c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\webworker-thread
s\node_modules\nan\nan.h(1215): warning C4267: 'initializing' : conversion from
 'size_t' to 'int', possible loss of data [c:\Users\amcwatters\Documents\Reposi
tories\api-v1\node_modules\webworker-threads\build\WebWorkerThreads.vcxproj]
c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\webworker-thread
s\node_modules\nan\nan.h(1229): warning C4267: 'initializing' : conversion from
 'size_t' to 'int', possible loss of data [c:\Users\amcwatters\Documents\Reposi
tories\api-v1\node_modules\webworker-threads\build\WebWorkerThreads.vcxproj]
c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\webworker-thread
s\node_modules\nan\nan.h(1318): warning C4244: 'return' : conversion from 'intp
tr_t' to 'int', possible loss of data [c:\Users\amcwatters\Documents\Repositori
es\api-v1\node_modules\webworker-threads\build\WebWorkerThreads.vcxproj]
c:\users\amcwatters\documents\repositories\api-v1\node_modules\webworker-thread
s\src\bson.cc(395): warning C4344: behavior change: use of explicit template ar
guments results in call to 'v8::Local<T> NanNew<v8::String,char*>(P,int)' [c:\U
sers\amcwatters\Documents\Repositories\api-v1\node_modules\webworker-threads\bu
ild\WebWorkerThreads.vcxproj]
          with
          [
              T=v8::String,
              P=char *
          ]
          but the regular function 'v8::Local<T> NanNew(const char *,int)' is a
   better match
          with
          [
              T=v8::String
          ]
          if you expect 'v8::Local<T> NanNew(const char *,int)' to be called th
  en you need to make it an explicit specialization
          with
          [
              T=v8::String
          ]
c:\users\amcwatters\documents\repositories\api-v1\node_modules\webworker-thread
s\src\bson.cc(405): warning C4344: behavior change: use of explicit template ar
guments results in call to 'v8::Local<T> NanNew<v8::String,uint16_t*>(P,int)' [
c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\webworker-thread
s\build\WebWorkerThreads.vcxproj]
          with
          [
              T=v8::String,
              P=uint16_t *
          ]
          but the regular function 'v8::Local<T> NanNew(const uint16_t *,int)'
  is a better match
          with
          [
              T=v8::String
          ]
          if you expect 'v8::Local<T> NanNew(const uint16_t *,int)' to be calle
  d then you need to make it an explicit specialization
          with
          [
              T=v8::String
          ]
c:\users\amcwatters\documents\repositories\api-v1\node_modules\webworker-thread
s\src\bson.cc(891): warning C4267: 'argument' : conversion from 'size_t' to 'ui
nt32_t', possible loss of data [c:\Users\amcwatters\Documents\Repositories\api-
v1\node_modules\webworker-threads\build\WebWorkerThreads.vcxproj]
c:\users\amcwatters\documents\repositories\api-v1\node_modules\webworker-thread
s\src\jslib.cc(79): warning C4344: behavior change: use of explicit template ar
guments results in call to 'v8::Local<T> NanNew<v8::String,char*>(P)' [c:\Users
\amcwatters\Documents\Repositories\api-v1\node_modules\webworker-threads\build\
WebWorkerThreads.vcxproj]
          with
          [
              T=v8::String,
              P=char *
          ]
          but the regular function 'v8::Local<T> NanNew(const char *,int)' is a
   better match
          with
          [
              T=v8::String
          ]
          if you expect 'v8::Local<T> NanNew(const char *,int)' to be called th
  en you need to make it an explicit specialization
          with
          [
              T=v8::String
          ]
..\src\WebWorkerThreads.cc(226): warning C4101: 'dummy' : unreferenced local va
riable [c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\webworke
r-threads\build\WebWorkerThreads.vcxproj]
..\src\WebWorkerThreads.cc(943): error C2601: 'Init' : local function definitio
ns are illegal [c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\
webworker-threads\build\WebWorkerThreads.vcxproj]
          ..\src\WebWorkerThreads.cc(940): this line contains a '{' which has n
  ot yet been matched
..\src\WebWorkerThreads.cc(975): error C2598: linkage specification must be at
global scope [c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\we
bworker-threads\build\WebWorkerThreads.vcxproj]
..\src\WebWorkerThreads.cc(975): error C2071: 'WebWorkerThreads_module' : illeg
al storage class [c:\Users\amcwatters\Documents\Repositories\api-v1\node_module
s\webworker-threads\build\WebWorkerThreads.vcxproj]
..\src\WebWorkerThreads.cc(980): fatal error C1075: end of file found before th
e left brace '{' at '..\src\WebWorkerThreads.cc(940)' was matched [c:\Users\amc
watters\Documents\Repositories\api-v1\node_modules\webworker-threads\build\WebW
orkerThreads.vcxproj]
gyp ERR! build error
gyp ERR! stack Error: `C:\windows\Microsoft.NET\Framework\v4.0.30319\msbuild.exe
` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onExit (c:\nodejs\node_modules\npm\node_modul
es\node-gyp\lib\build.js:267:23)
gyp ERR! stack     at ChildProcess.emit (events.js:98:17)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (child_process.js:810:
12)
gyp ERR! System Windows_NT 6.1.7601
gyp ERR! command "node" "c:\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\
bin\\node-gyp.js" "rebuild"
gyp ERR! cwd c:\Users\amcwatters\Documents\Repositories\api-v1\node_modules\webw
orker-threads
gyp ERR! node -v v0.10.32
gyp ERR! node-gyp -v v1.0.1
gyp ERR! not ok

npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the webworker-threads package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-gyp rebuild
npm ERR! You can get their info via:
npm ERR!     npm owner ls webworker-threads
npm ERR! There is likely additional logging output above.
npm ERR! System Windows_NT 6.1.7601
npm ERR! command "c:\\nodejs\\node.exe" "c:\\nodejs\\node_modules\\npm\\bin\\npm
-cli.js" "install" "--save" "webworker-threads" "--msvs_version=2012"
npm ERR! cwd c:\Users\amcwatters\Documents\Repositories\api-v1
npm ERR! node -v v0.10.32
npm ERR! npm -v 1.4.28
npm ERR! code ELIFECYCLE
npm ERR! not ok code 0

onmessage events not triggered

Using Node.js 0.10.29 built from source, NPM 1.4.14, and [email protected], on Ubuntu 12.0.4.4 LTS (x86_64), I don't get the expected output from examples/quickIntro_oneThreadEvented.js, or an even simpler example:

var fs = require('fs');
var Worker = require('webworker-threads').Worker;

var w = new Worker(function() {
  var self = this;
  self.onmessage(function(event) {
    self.postMessage(event.data+1);
  });
});

w.onmessage = function callback(event) {
   process.stdout.write(event.data);
   fs.writeFileSync("myoutput.txt", event.data, {'encoding': 'utf-8'});
}
w.postMessage(5);
process.stdout.write("Did anything happen?\n");

/* Expect 6 to be written to stdout eventually */

I just see "Did anything happen?", and the program hangs forever without output (no file is written either).

JS eventloop on worker thread

I've been doing some tests with workers and I realize that several async functions (e.g. setInterval, setTimer, xhr requests, etc) are not working inside threads.
I understand that this happens because of the lack of a "true" JS eventLoop in th thread.
Is this something that can be implemented? How hard it is?

[Question] How to use nodejs package in child-thread?

I want to parse HTML in child-thread (need jsdom) because it is much slower than the speed fetching web pages, if I parse in main thread, it will fill up the event loop, cause socket timeout and finally ruin the program.

But jsdom is nodejs package, I can't use global variable jsdom, or declare new variable var jsdom = require('jsdom'); as I do in main-thread,
I also try to pass it to child-thread via parameter, just like worker.postMessage([jsdom, html]); and of course, failed. Actually I get the jsdom in child-thread. console.log(event.data[0]); have some output, but It's not the same with main-thread.

I have a test on serveral nodejs-defined variable, such as console.log(typeof(require)); console.log(typeof(global)); and I get undefined or very different.

I think the child-thread using a completely different context with main-thread, so is it possible to use nodejs package in child-thread?

Or, what's the right way to do this? (Fork, spawn, cluster? or anything better? These three things will make the program much more complex, and what I want to do is just a simple spider.)

setInterval problem

Greetings!
I'm running the my code:
main.js:

var Worker = require('webworker-threads').Worker;
var worker = new Worker('worker.js');
worker.onmessage = function (event) {
     console.log(event.data);
};
worker.postMessage('John');

worker.js:

onmessage = function(event) {
    setInterval(function(){
        postMessage('Hi, ' + event.data);
    }, 3000);
};

but the code does not work.
What am I doing wrong?
Thanks in advance for help!

build failed when using io.js 1.0.4 and node v0.11.16

the build is failed when using latest io.js and node.js

> node-gyp rebuild

child_process: customFds option is deprecated, use stdio instead.
  CXX(target) Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o
../src/WebWorkerThreads.cc:291:5: error: no matching member function for call to 'Set'
    JSObjFn(fs_obj, "readFileSync", readFileSync_);
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/jslib.cc:25:7: note: expanded from macro 'JSObjFn'
        obj->Set(NanNew<String>(name), NanNew<FunctionTemplate>(fnname)->GetFunction(), attribute_ro_dd);
        ~~~~~^~~
/Users/gitawego/.node-gyp/0.11.16/deps/v8/include/v8.h:2112:8: note: candidate function not viable: requires 2 arguments, but 3 were provided
  bool Set(Handle<Value> key, Handle<Value> value);
       ^
/Users/gitawego/.node-gyp/0.11.16/deps/v8/include/v8.h:2114:8: note: candidate function not viable: requires 2 arguments, but 3 were provided
  bool Set(uint32_t index, Handle<Value> value);
       ^
../src/WebWorkerThreads.cc:292:13: error: no matching member function for call to 'Set'
    global->Set(NanNew<String>("native_fs_"), fs_obj, attribute_ro_dd);
    ~~~~~~~~^~~
/Users/gitawego/.node-gyp/0.11.16/deps/v8/include/v8.h:2112:8: note: candidate function not viable: requires 2 arguments, but 3 were provided
  bool Set(Handle<Value> key, Handle<Value> value);
       ^
/Users/gitawego/.node-gyp/0.11.16/deps/v8/include/v8.h:2114:8: note: candidate function not viable: requires 2 arguments, but 3 were provided
  bool Set(uint32_t index, Handle<Value> value);
       ^
../src/WebWorkerThreads.cc:295:5: error: no matching member function for call to 'Set'
    JSObjFn(console_obj, "log", console_log);
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/jslib.cc:25:7: note: expanded from macro 'JSObjFn'
        obj->Set(NanNew<String>(name), NanNew<FunctionTemplate>(fnname)->GetFunction(), attribute_ro_dd);
        ~~~~~^~~
/Users/gitawego/.node-gyp/0.11.16/deps/v8/include/v8.h:2112:8: note: candidate function not viable: requires 2 arguments, but 3 were provided
  bool Set(Handle<Value> key, Handle<Value> value);
       ^
/Users/gitawego/.node-gyp/0.11.16/deps/v8/include/v8.h:2114:8: note: candidate function not viable: requires 2 arguments, but 3 were provided
  bool Set(uint32_t index, Handle<Value> value);
       ^
../src/WebWorkerThreads.cc:296:5: error: no matching member function for call to 'Set'
    JSObjFn(console_obj, "error", console_error);
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/jslib.cc:25:7: note: expanded from macro 'JSObjFn'
        obj->Set(NanNew<String>(name), NanNew<FunctionTemplate>(fnname)->GetFunction(), attribute_ro_dd);
        ~~~~~^~~
/Users/gitawego/.node-gyp/0.11.16/deps/v8/include/v8.h:2112:8: note: candidate function not viable: requires 2 arguments, but 3 were provided
  bool Set(Handle<Value> key, Handle<Value> value);
       ^
/Users/gitawego/.node-gyp/0.11.16/deps/v8/include/v8.h:2114:8: note: candidate function not viable: requires 2 arguments, but 3 were provided
  bool Set(uint32_t index, Handle<Value> value);
       ^
../src/WebWorkerThreads.cc:297:13: error: no matching member function for call to 'Set'
    global->Set(NanNew<String>("console"), console_obj, attribute_ro_dd);
    ~~~~~~~~^~~
/Users/gitawego/.node-gyp/0.11.16/deps/v8/include/v8.h:2112:8: note: candidate function not viable: requires 2 arguments, but 3 were provided
  bool Set(Handle<Value> key, Handle<Value> value);
       ^
/Users/gitawego/.node-gyp/0.11.16/deps/v8/include/v8.h:2114:8: note: candidate function not viable: requires 2 arguments, but 3 were provided
  bool Set(uint32_t index, Handle<Value> value);
       ^
../src/WebWorkerThreads.cc:345:17: error: no member named 'IdleNotification' in 'v8::V8'
            V8::IdleNotification();
            ~~~~^
../src/WebWorkerThreads.cc:430:17: error: no member named 'IdleNotification' in 'v8::V8'
            V8::IdleNotification();
            ~~~~^
7 errors generated.
make: *** [Release/obj.target/WebWorkerThreads/src/WebWorkerThreads.o] Error 1
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/Users/gitawego/.nvm/v0.11.16/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:267:23)
gyp ERR! stack     at ChildProcess.emit (events.js:110:17)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (child_process.js:1067:12)
gyp ERR! System Darwin 14.1.0
gyp ERR! command "node" "/Users/gitawego/.nvm/v0.11.16/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/gitawego/workspace/audit-server/node_modules/webworker-threads
gyp ERR! node -v v0.11.16
gyp ERR! node-gyp -v v1.0.2
gyp ERR! not ok 

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.