coopernurse / node-pool Goto Github PK

Since destroy() decrements count, the pool will no longer be full. Consequently we should probably call dispense() to unblock a waiting acquire() call.

hello,

can i have sample running code for connection pooling in node js.

thanks
pranav

Not sure that it is a bug, but in 'tests drain' there is a mistype:

        max : 2,
        idletimeoutMillis : 300000
    });

I think that it should be idleTimeoutMillis.

I'm getting these warnings:

WARNING: ev_ref is deprecated, use uv_ref
WARNING: ev_unref is deprecated, use uv_unref

Otherwise it's working great! Thanks.

Sorry - the warnings are not related to generic-pool. They are related to the db-mysql module.

Please close.

Currently if release is called twice for the same resource the only way to know it is to enable logging and watch for it.

That error "handling" logic may leads to issues like that: https://github.com/brianc/node-postgres/issues/11111

I suggest to somehow raise that error by using exceptions or another node error emitting technique.

If I call pool.destroy() it will decrease the count even if one didn't pass in any arguments, or the object is not found in the pool. It would be great to have some kind of a check for both cases.
Maybe not passing any argument could mean, that the user would want to destroy the whole pool (all items), or throw a warning if you don't like that idea.
The latter is harder to implement as I see atm, there is no keeping track of the used items, just the waiting. If you dont like the idea of keeping track of the objects, maybe remove the the public method remove. Although not keeping track of the in use objects also questions the value of the returnToPool method, as there is also no check if the object was generated by the factory, and returned by the pool with the use of borrow, or it is just made up.

Hi,

Since there are some old unresolved issues, I was just wondering if this package is still maintained?

well ,after we create the pool ,
for example , a connection is lost and therefor useless ,
how to remove it from available queues?

Hi there,

We're using the pool to handle MongoDB connections. On our MongoDB server, we're seeing up to 80 connections used even though we specify max:10 in our Pool config block. Is there any output or info we can provide to help debug?

Thanks!

Rob

Sometimes you need to remove something from pool. For example you have a redis connection in pool and sometimes it's get corrpuped and pool keeps to borrow it to clients and there's no obvious way to stop it from borrowing it. Maybe you can keep it forever so it won't be borrowed, but it's not a good idea I guess. Maybe I miss something here?

Does using an object as a key actually do what the code thinks it should?

availableObjects.push(obj);
objectTimeout[obj] = (new Date().getTime() + idleTimeoutMillis);
var size = 0;
objectTimeout.test = 12345;
for (i in objectTimeout) { size++; }
log("timeout: " + objectTimeout[obj] + ", length: " + size);

In the logs:

pool test_pool - timeout: 1295298653460, length: 1
pool test_pool - timeout: 1295298653461, length: 1
pool test_pool - timeout: 1295298653461, length: 1

For instance, in JavaScript...

objectTimeout[{foo: 0}] === objectTimeout[{bar: 1}]

...which could be the cause of this.

Thanks!

Would it be an idea to be able to grow and shrink the pool ? I know it would in my use case :)

Imagine I have a net.createServer setup, and want to keep a maximum of connections, and delete idle connections, could this module be used for that? I'm failing to understand the logic on creating a "client" socket pool on a server, like how can I limit the connections one server can receive? And how the old idle connections are dropped? Would I need another façade class for the new connections?

Arity checks for acquire() callback (added with commit a3b65a371e) are incompatible with IcedCoffeeScript callbacks.

IcedCoffeeScript callbacks internally use arguments[] and does not reveal the exact number of arguments via .length property.

I suggest to not check arity, at least by default.

Just curious, how do you check if a resource is idle?

Hi..
Is it possible to manually invoke createResource?

It's useful, for an instance if we're going to maintain the minimum threshold to be the 'minimum available resources' and not 'minimum resources regardless available or used'. Then we could call createResource() when needed.

Currently resources are only tested when being pulled from the available pool.

We should add flags to allow (names debatable).

• testOnBorrow
• testOnReturn
• testOnCreates
• testWhileIdle - idle objects in the pool will be validated by the idle object evictor reapIntervalMillis

(unrelated, we should remove factory.refreshIdle, renameidleTimeoutMillis to something like getMinEvictableIdleTimeMillis and only refresh idle things if we have reapIntervalMillis

By default testOnBorrow will be true if validation function is supplied, false if not and testOnReturn will be false (which roughly follows current conventions)
I borrowed those names from some apache generic object pool

Hi guys,

I'm new in node js and I'm trying to integrate node-postgres package, which requires node-pool package but I'm having some errors. Could somebody please help me?

W20140324-12:49:25.508(-6)? (STDERR) /Users/crowdint/.meteor/tools/8e197f29c5/lib/node_modules/fibers/future.js:173
W20140324-12:49:25.509(-6)? (STDERR) throw(ex);
W20140324-12:49:25.509(-6)? (STDERR)      ^
W20140324-12:49:25.513(-6)? (STDERR) ReferenceError: exports is not defined
W20140324-12:49:25.514(-6)? (STDERR)     at app/node_modules/pg/node_modules/generic-pool/lib/generic-pool.js:99:1
W20140324-12:49:25.514(-6)? (STDERR)     at app/node_modules/pg/node_modules/generic-pool/lib/generic-pool.js:469:3
W20140324-12:49:25.514(-6)? (STDERR)     at /Users/crowdint/projects/kardex/.meteor/local/build/programs/server/boot.js:155:10
W20140324-12:49:25.515(-6)? (STDERR)     at Array.forEach (native)
W20140324-12:49:25.515(-6)? (STDERR)     at Function._.each._.forEach (/Users/crowdint/.meteor/tools/8e197f29c5/lib/node_modules/underscore/underscore.js:79:11)
W20140324-12:49:25.517(-6)? (STDERR)     at /Users/crowdint/projects/kardex/.meteor/local/build/programs/server/boot.js:82:5

This is the file:

exports.Pool = function (factory) {
  var me = {},
. . .

I did this change but I'm not sure it's what I have to do.

exports = typeof exports === "undefined" ? {} : exports
exports.Pool = function (factory) {
  var me = {},

I wonder where exports it's defined or if there is a problem in the order the files are getting loaded with meteor or something

I am using node-cubrid, a CUBRID Database driver for Node.js with node-pool.

I have intentionally turned off the database server while having started the Node.js app which uses node-pool. I wanted to check what kind of an error I would receive from node-pool when acquiring a connection. However, node-pool was just hanging in there awaiting for a connection to be established for infinite time. Over 10 minutes has passed now, but node-pool is still not responding.

So, implementing a timeout functionality for acquire would be very useful.

Hi,

Great module - I didn't see it from briefly scanning your code: it's pretty common that with various drivers (especially stateful/long-lived ones), I'd want to have the pool auto-purge a client that emits an error event, so it can recreate another one to replace it. Is there a way to do that?

Thanks!
~Mark

I notice that recent tags have been uploaded to npm, which is great, but they appear not to have been pushed to github, which would also be awesome to see (while less critical).

Hi all,
I am developing a website using a node. js, tedious. js (database : MSSQL 2008). I am also using 'generic-pool' for pooling.
Here is my code

TediousPooler.js

var poolModule = require('generic-pool'), Connection = require('tedious').Connection;
var pool;
exports = module.exports = TediousPooler;
function TediousPooler(config, connection) {
pool = poolModule.Pool({
name: 'tedious',
create: function (callback) {
var connection = new Connection(config);
connection.on('connect', function (err) {
if (err) { console.log(err); }
else { callback(null, connection); }
});
},
destroy: function (client) { connection.close(); },
max: 10,
min: 2,
idleTimeoutMillis: 30000,
log: true
});
}

TediousPooler.prototype.callProcedure = function (request) {
pool.acquire(function (err, connection) {
if (err) { console.log(err); }
else {
connection.callProcedure(request)
}
});
}

TediousPooler.prototype.poolRelease = function (connection) {
pool.release(connection);
}

SqlConnect.js (here i am prepare statement for SQL eg. assigning parameters and get result from database )

var Request = require('tedious').Request;
var TYPES = require('tedious').TYPES;

var tediousPooler = require('./TediousPooler');

var config = {
server: '192.XXX.X.XXX',
userName: 'UserName',
password: 'Password',
options: {
database: 'DatabaseName'
}
};

var pooler;
module.exports.execute = function (sp, aParam, callback) {
var paramList = [];
var rowCollection = [];
paramList = aParam;
exec(sp);

function exec(sql) {
    request = new Request(sql, statementComplete, function (err) {
        if (err) { console.log('Error while executing request : ' + err); }
    });

    paramList.forEach(function (param) {
        if (param['paramType'] === 'Numeric') {
            param['paramType'] = TYPES.Numeric;
        }
        else if (param['paramType'] === 'BigInt') {
            param['paramType'] = TYPES.Int;
        }
        else if (param['paramType'] === 'Decimal') {
            param['paramType'] = TYPES.Decimal;
        }
        else {
            param['paramType'] = TYPES.VarChar;
        }
        request.addParameter(param['paramName'], param['paramType'], param['paramValue']);
    });

    paramList = [];

    request.on('columnMetadata', columnMetadata);
    request.on('row', row);


    if (pooler === undefined) {

        pooler = new tediousPooler(config);
     }
    pooler.callProcedure(request);
}

function statementComplete(err) {
    if (err) { callback('Statement failed: ' + err); }
    else {
        callback(rowCollection);
    }
}

function row(columns) {
    var row = {};
    columns.forEach(function (column) {
        if (column.isNull) {
        } else {
            row[column.metadata.colName] = column.value;
        }
    });
    rowCollection.push(row);
}

function ErrorMessage(error) {
    return;
}

function end() {
    pooler.drain(function () { pooler.destroyAllNow(); });
}

};

App.js - calling storeprocedure

SQLConnect = require('./SQLConnect')

var sp = "usp_ProcedureName"
var aSQLParam = [];
aSQLParam.push({ paramName: 'Name', paramType: 'VarChar', paramValue: 'Value' });

SQLConnect.execute(sp, aSQLParam, function (result) {
//here i am getting result/output from database
})

I want to know that the above process is valid or any other good can be available. and when i have to release pool object.

Currently we only keep a list of available objects/resource, and then rely on manual bookeeping to keep track of the number of borrowed items. This leaves lots of room for accounting errors and getting things into an inconsistent state.

We should have a list/array of borrowed objects.

Hi,

I'd like to log my own custom errors. Also, I don't want to log every second the message "availableObjects.length=x". Can you inherite the pool from EventEmitter and just emit events so we can catch them and do whatever we want? It's a very easy modification. Just replace this line when you export the module:

var me = {},

By this:

var Pool = function (){
  events.EventEmitter.call (this);
};
Pool.prototype = Object.create (events.EventEmitter.prototype);
Pool.prototype.constructor = Pool;
var me = new Pool (),

You must add the events module:

var events = require ("events");

Then, instead of calling the log function you can throw an event with the data. For example, the createResource function:

me.emit ("create-resource", count, factory.min, factory.max);

And to catch it from outside your library:

pool.on ("create-resource", function (count, min, max){
  ...
});

First, thanks for merging #14.

I just ran into an interesting edge case where the callback .length based arg signature detection fails. If you use .bind() on a function, node will generate a new function which appears to be wrapped in such a way that .length does not accurately represent the "inner" function's arguments, e.g.

pool.acquire(function(err, resource) {
    // this works.
});
pool.acquire(function(err, resource) {
    // this fails -- looks like a 1 arg callback to node-pool.
}.bind(this));

I'm not sure what the best solution will be to this problem, or if it will affect most API users at all. It's easy enough to workaround so it might be worth documenting until we have a fix or the 1 arg callback support can be deprecated.

When all the resource are used, is there some way to set a timeout for acquire? I red the source code and found that is there is no resource available and the max limit is reached, the callback of acquire is never called.

Pool.destroy() (that is, NOT the factory.destroy function) is not documented at all in the README.md, but is in fact useful.

It is especially if the resource being managed can fail and somehow signal that failure. For example, if the resource is a child_process. that child might die, and we then want to remove it from the pool.

This works, but only if pool.release(obj) is called BEFORE calling pool.destroy(). That should also be documented. The comments for pool.destroy say:

"This should be called within an acquire() block as an alternative to release()."

And this statement is just wrong. If release() is not called, the object will not be in availableObjects to be removed by the filter() call in pool.destroy.

Keep up the great work. This is a super useful library.

"reset" method that can be overridden is a very important addition that will let a "resting" a resource before putting it back to the pool. The resource shouldn't be available until it was "rested" and should be released on err.

var poolModule = require('generic-pool');
var pool = poolModule.Pool({

    reset: function(resource, done) { 
          resource.clean(function(err){}{ 
             done(err);
          });
    }

}

Ok I have a min of 2 max of 3. I run a query on a page hit. I hit the page once fine. I refresh gets another connection, fine. Hit the page again it grows the pool and gets a connection. Hit the page again and nothing happens. I would think that if I'm releasing the pool after each db query then the pool would shrink? I thought that was the whole point of a pool?

var pool = poolModule.Pool({
    name: 'mysql',
    create: function (callback) {

        console.log('create');

        var connection = mysql.createConnection({
            host: 'localhost',
            user: 'admin',
            password: 'test',
            database: 'test'
        });

        connection.connect(function () {
            return callback(null, connection);
        });

    },
    destroy: function (client) { // this is never called.  ever

        console.log('destroy');
        client.destroy();
    },
    max: 3,
    min: 2,

    idleTimeoutMillis: 3000, //changed this  to 3000
    log: true
});


pool.query = function (query, data, callback) {

    try {

        pool.acquire(function (err, client) {

          //called 3 times.  never called again.

            client.query(query, data, function (err, results, fields) {

                try {

                    pool.release(client); // I've also tried pool.destroy(client);          
                    return callback(err, results);

                } catch (err) {
                    console.log(err)
                }

            });
        });
    } catch (err) {
        console.log(err);
    }   
};

I just had a question about when aquire returns false. If it does this, that means all connections are currently in use. Does it automatically hold the request and wait for a connection to open up or does it just skip all remaining requests?

borrow and returnToPool have both been depreciated for a while now so lets bin them.

• remove functions
• update docs

Please, for some reasons, it's useful to put a callback function that does not have any params defined,
and in this case adjustCallback function calls callback with one parameter only, without an error information...
I suggest to change there a condition
to send ONE parameter only in ONE case - cb.length == 1, not cb.length<=1...
original function:

  function adjustCallback(cb, err, obj) {
    if (!cb) return;
    if (cb.length <= 1) {
      cb(obj);
    } else {
      cb(err, obj);
    }
  }

Can you add this parameter? I use node-pool to create pool of FireFox browsers for parsing sites, and every 2h I have to restart my script because of memory leak. Can you help me with my project?=)

Quick ones:

1. Any chance of getting a git tag for the current release (and future releases)?
2. Under what license is node-pool released?

Thanks!

Hi,

I'm using node-pool just as a concurrency limiter, without the need to create or destroy resources each time. Hence I don't need the create() and destroy() arguments but they are mandatory. Could there be default values for them ? like

             'create':function(callback) {
                callback();
            },
            'destroy':function() {}

Thanks!

I'm having trouble getting my pool keep a minimum number of resources alive. Essentially, I want to have at least 4 available at all times and spawn up to 4 more when load becomes heavier.

This is my code:

this.generatorPool = poolModule.Pool({
    name: 'generator-pool',
    create: function (resolveResource) {
      var generator = new Generator();
      generator.start().then(
        // Generator load successful
        function() {
          resolveResource(null, generator);
        },
        // Generator failed to load
        function () {
          resolveResource(new Error('Failed to load generator'));
        }
      );
    },
    destroy: function (generator) {
      console.log('DESTROYING THE GENERATOR)
      generator.stop();
    },
    max: 8,
    min: 4,
    refreshIdle: false,
    idleTimeoutMillis: 30000
  });

For some reason the first 4 resources are still being destroyed after 30 seconds. What could be up?

Sometimes you need to remove something from pool. For example you have a redis connection in pool and sometimes it's get corrpuped and pool keeps to borrow it to clients and there's no obvious way to stop it from borrowing it. Maybe you can keep it forever so it won't be borrowed, but it's not a good idea I guess. Maybe I miss something here?

Please put 'var err, obj;' in the callback given to factory.create.

https://github.com/coopernurse/node-pool/blob/master/lib/generic-pool.js#L246

We are using generic-pool with Oracle back end for connection pooling in our project. We are seeing a weird issue of inactive connections/sessions in oracle.
Our factory.max is set to 10 and min is set 2 so our understanding is that, number of connections won’t cross 10, but somehow in oracle we see a connection per request and its goes up very high till it reaches oracle limit (about 500). We can see old connections in inactive state in oracle and when we stop the node server all these connections go away from oracle. Our connection pool’s size never reaches 10, it keeps jumping between 4 - 6, and I am not sure what is going wrong.
We have kept idleTimeoutMillis to 5 minutes, but connections stay in inactive state even beyond 5 minutes. I experimented with reducing the time to 30 secs, no help.
In our code we are releasing connections, and that can be verified because connection pool size never reaches 10. any suggestion help is much appreciated.

The documentation says
"if this is set > max, the pool will silently set the min to factory.max - 1"

In fact, if you try to set min == max, which should be allowed by the statement above, it is set to max - 1. I worked around this by resetting it in the first call to my create() function. It works fine with min == max, and keeps exactly max objects in the pool as expected.

Thanks for this great library...it really helped me out.

I don't know if I'm missing something, but in the code, me.destroy() is as follows:

https://github.com/coopernurse/node-pool/blob/master/lib/generic-pool.js#L114

me.destroy = function(obj) {
    count -= 1;
    factory.destroy(obj);
};

and, looking at the possible scenarios:

• If the connection that is going to be destroyed it's available, then it's on availableObjects array, which is an array of objects of the form {obj:obj, timeout:timeout_in_ml}, just disconnecting the object (which is what the custom defined 'destroy' function at the constructor is meant for) won't remove the element {obj:obj, timeout:timeout_in_ms} from the availableObjects array, thus that item will still be available, but the object it returns won't be working anymore!.
• If the connection that is going to be destroyed it's "in use", it should also be released (or that could be left in the hands of the user, I don't know).

I guess when destroying the conn, the 'availableObjects' corresponding item should be shifted, too.

Please, let me know if I'm missing something. I've read the code a zillion times and can't find how it can be otherwise than what I just said.

txs

Hi!

I'm also using node-pool in the browser to act as a AJAX request pool. It works great !

So I was just wondering if that was one of the goals of this project, at least maybe it could be mentionned somewhere in the doc because it may be useful for other people!

cheers

I'm setting a pool minimum and still seeing a wasteful destroy/create cycle for each resource in the queue. Given this simple nodejs file:

var i = 0;
var Pooler = require('generic-pool');
var pool = Pooler.Pool({
    idleTimeoutMillis : 3000,
    reapInterval : 1000,
    create: function(callback){
        console.log('expensive create', i);
        i++;
        callback(null, {i:i});
    },
    destroy: function(client){
        console.log('destroy', client.i);
    },
    min: 5,
    max: 10
});

I'm seeing the following log:

expensive create 0              
expensive create 1              
expensive create 2              
expensive create 3              
expensive create 4              
destroy 1                       
expensive create 5              
destroy 2                       
expensive create 6              
destroy 3                       
expensive create 7              
destroy 4                       
expensive create 8              
destroy 5                       
expensive create 9              
destroy 6                       
expensive create 10             
destroy 7                       
expensive create 11             
destroy 8                       
expensive create 12             
destroy 9                       
expensive create 13             
...

The behavior of destroying then immediately recreating these resources doesn't seem like desired behavior. Yes these timeouts are short, but I'd expect the pooler to only cleanup resources above my specified minimum that were unused. Am I missing something here?

Currently the create() / acquire() method that is used to get a resource only returns one object in a callback - the resource in question. There are a few cases (invalid input, no disk space) in which object cannot be acquired and an error parameter would be useful. A simple-ish implementation of this could also ensure backwards compatibility.

Calling .drain() doesn't clear the timeouts set by node-pool. This means node won't exit until all objects expired, even when clearing the pool manually or calling .destroyAllNow()

The current test library has been depreciated for around 3 years but somehow still works. We should probably replace it with something maintained like tap or lab etc before it gets broken by a new release of node.

Open to suggestions or PRs for this. It's going to be a fair chunk of work to rewrite everything I suspect.

We're noticing connections getting reaped aggressively with this message:

pool test_pool - removeIdle() destroying obj - now:1295293476740 timeout:undefined
pool test_pool - removeIdle() destroying obj - now:1295293476740 timeout:undefined
pool test_pool - removeIdle() destroying obj - now:1295293476740 timeout:undefined

Does the order of these lines need to be transposed?

availableObjects.push(obj);
objectTimeout[obj] = (new Date().getTime() + idleTimeoutMillis);

Rob

Within createResource(), clientCb may be null. The if (err) does not check for null, but the else does.
You can indeed hit this condition.

I see that you have created a generic pooling mechanism. That is an awesome extraction which goes beyond node-pool.

It's time to move this repo to https://github.com/coopernurse/generic-pool and have a link to generic-pool in https://github.com/coopernurse/node-pool