Git Product home page Git Product logo

node-pool's Issues

.drain() doesn't clear timeouts

Calling .drain() doesn't clear the timeouts set by node-pool. This means node won't exit until all objects expired, even when clearing the pool manually or calling .destroyAllNow()

how to deal with reconnect issue?

well ,after we create the pool ,
for example , a connection is lost and therefor useless ,
how to remove it from available queues?

What is the right way to use pooling

Hi all,
I am developing a website using a node. js, tedious. js (database : MSSQL 2008). I am also using 'generic-pool' for pooling.
Here is my code

TediousPooler.js

var poolModule = require('generic-pool'), Connection = require('tedious').Connection;
var pool;
exports = module.exports = TediousPooler;
function TediousPooler(config, connection) {
pool = poolModule.Pool({
name: 'tedious',
create: function (callback) {
var connection = new Connection(config);
connection.on('connect', function (err) {
if (err) { console.log(err); }
else { callback(null, connection); }
});
},
destroy: function (client) { connection.close(); },
max: 10,
min: 2,
idleTimeoutMillis: 30000,
log: true
});
}

TediousPooler.prototype.callProcedure = function (request) {
pool.acquire(function (err, connection) {
if (err) { console.log(err); }
else {
connection.callProcedure(request)
}
});
}

TediousPooler.prototype.poolRelease = function (connection) {
pool.release(connection);
}


SqlConnect.js (here i am prepare statement for SQL eg. assigning parameters and get result from database )

var Request = require('tedious').Request;
var TYPES = require('tedious').TYPES;

var tediousPooler = require('./TediousPooler');

var config = {
server: '192.XXX.X.XXX',
userName: 'UserName',
password: 'Password',
options: {
database: 'DatabaseName'
}
};

var pooler;
module.exports.execute = function (sp, aParam, callback) {
var paramList = [];
var rowCollection = [];
paramList = aParam;
exec(sp);

function exec(sql) {
    request = new Request(sql, statementComplete, function (err) {
        if (err) { console.log('Error while executing request : ' + err); }
    });

    paramList.forEach(function (param) {
        if (param['paramType'] === 'Numeric') {
            param['paramType'] = TYPES.Numeric;
        }
        else if (param['paramType'] === 'BigInt') {
            param['paramType'] = TYPES.Int;
        }
        else if (param['paramType'] === 'Decimal') {
            param['paramType'] = TYPES.Decimal;
        }
        else {
            param['paramType'] = TYPES.VarChar;
        }
        request.addParameter(param['paramName'], param['paramType'], param['paramValue']);
    });

    paramList = [];

    request.on('columnMetadata', columnMetadata);
    request.on('row', row);


    if (pooler === undefined) {

        pooler = new tediousPooler(config);
     }
    pooler.callProcedure(request);
}

function statementComplete(err) {
    if (err) { callback('Statement failed: ' + err); }
    else {
        callback(rowCollection);
    }
}

function row(columns) {
    var row = {};
    columns.forEach(function (column) {
        if (column.isNull) {
        } else {
            row[column.metadata.colName] = column.value;
        }
    });
    rowCollection.push(row);
}

function ErrorMessage(error) {
    return;
}

function end() {
    pooler.drain(function () { pooler.destroyAllNow(); });
}

};


App.js - calling storeprocedure

SQLConnect = require('./SQLConnect')

var sp = "usp_ProcedureName"
var aSQLParam = [];
aSQLParam.push({ paramName: 'Name', paramType: 'VarChar', paramValue: 'Value' });

SQLConnect.execute(sp, aSQLParam, function (result) {
//here i am getting result/output from database
})

I want to know that the above process is valid or any other good can be available. and when i have to release pool object.

Implement an "acquire" timeout functionality

I am using node-cubrid, a CUBRID Database driver for Node.js with node-pool.

I have intentionally turned off the database server while having started the Node.js app which uses node-pool. I wanted to check what kind of an error I would receive from node-pool when acquiring a connection. However, node-pool was just hanging in there awaiting for a connection to be established for infinite time. Over 10 minutes has passed now, but node-pool is still not responding.

So, implementing a timeout functionality for acquire would be very useful.

Couple questions

Quick ones:

  1. Any chance of getting a git tag for the current release (and future releases)?
  2. Under what license is node-pool released?

Thanks!

Removing something from pool

Sometimes you need to remove something from pool. For example you have a redis connection in pool and sometimes it's get corrpuped and pool keeps to borrow it to clients and there's no obvious way to stop it from borrowing it. Maybe you can keep it forever so it won't be borrowed, but it's not a good idea I guess. Maybe I miss something here?

Replace test library

The current test library has been depreciated for around 3 years but somehow still works. We should probably replace it with something maintained like tap or lab etc before it gets broken by a new release of node.

Open to suggestions or PRs for this. It's going to be a fair chunk of work to rewrite everything I suspect.

remove deprecated functions

borrow and returnToPool have both been depreciated for a while now so lets bin them.

  • remove functions
  • update docs

pool.destroy(obj) is not documented, and requires release() call first

Pool.destroy() (that is, NOT the factory.destroy function) is not documented at all in the README.md, but is in fact useful.

It is especially if the resource being managed can fail and somehow signal that failure. For example, if the resource is a child_process. that child might die, and we then want to remove it from the pool.

This works, but only if pool.release(obj) is called BEFORE calling pool.destroy(). That should also be documented. The comments for pool.destroy say:

"This should be called within an acquire() block as an alternative to release()."

And this statement is just wrong. If release() is not called, the object will not be in availableObjects to be removed by the filter() call in pool.destroy.

Keep up the great work. This is a super useful library.

Client-side

Hi!

I'm also using node-pool in the browser to act as a AJAX request pool. It works great !

So I was just wondering if that was one of the goals of this project, at least maybe it could be mentionned somewhere in the doc because it may be useful for other people!

cheers

Pool size exceeds maximum

Hi there,

We're using the pool to handle MongoDB connections. On our MongoDB server, we're seeing up to 80 connections used even though we specify max:10 in our Pool config block. Is there any output or info we can provide to help debug?

Thanks!

Rob

Is there any hooks for .on('error') to remove a driver from the pool?

Hi,

Great module - I didn't see it from briefly scanning your code: it's pretty common that with various drivers (especially stateful/long-lived ones), I'd want to have the pool auto-purge a client that emits an error event, so it can recreate another one to replace it. Is there a way to do that?

Thanks!
~Mark

destroy

If I call pool.destroy() it will decrease the count even if one didn't pass in any arguments, or the object is not found in the pool. It would be great to have some kind of a check for both cases.
Maybe not passing any argument could mean, that the user would want to destroy the whole pool (all items), or throw a warning if you don't like that idea.
The latter is harder to implement as I see atm, there is no keeping track of the used items, just the waiting. If you dont like the idea of keeping track of the objects, maybe remove the the public method remove. Although not keeping track of the in use objects also questions the value of the returnToPool method, as there is also no check if the object was generated by the factory, and returned by the pool with the use of borrow, or it is just made up.

meteor js integration

Hi guys,

I'm new in node js and I'm trying to integrate node-postgres package, which requires node-pool package but I'm having some errors. Could somebody please help me?

W20140324-12:49:25.508(-6)? (STDERR) /Users/crowdint/.meteor/tools/8e197f29c5/lib/node_modules/fibers/future.js:173
W20140324-12:49:25.509(-6)? (STDERR) throw(ex);
W20140324-12:49:25.509(-6)? (STDERR)      ^
W20140324-12:49:25.513(-6)? (STDERR) ReferenceError: exports is not defined
W20140324-12:49:25.514(-6)? (STDERR)     at app/node_modules/pg/node_modules/generic-pool/lib/generic-pool.js:99:1
W20140324-12:49:25.514(-6)? (STDERR)     at app/node_modules/pg/node_modules/generic-pool/lib/generic-pool.js:469:3
W20140324-12:49:25.514(-6)? (STDERR)     at /Users/crowdint/projects/kardex/.meteor/local/build/programs/server/boot.js:155:10
W20140324-12:49:25.515(-6)? (STDERR)     at Array.forEach (native)
W20140324-12:49:25.515(-6)? (STDERR)     at Function._.each._.forEach (/Users/crowdint/.meteor/tools/8e197f29c5/lib/node_modules/underscore/underscore.js:79:11)
W20140324-12:49:25.517(-6)? (STDERR)     at /Users/crowdint/projects/kardex/.meteor/local/build/programs/server/boot.js:82:5

This is the file:

exports.Pool = function (factory) {
  var me = {},
. . .

I did this change but I'm not sure it's what I have to do.

exports = typeof exports === "undefined" ? {} : exports
exports.Pool = function (factory) {
  var me = {},

I wonder where exports it's defined or if there is a problem in the order the files are getting loaded with meteor or something

git tags

I notice that recent tags have been uploaded to npm, which is great, but they appear not to have been pushed to github, which would also be awesome to see (while less critical).

Objects being aggressively reaped with 'undefined' timeouts

We're noticing connections getting reaped aggressively with this message:

pool test_pool - removeIdle() destroying obj - now:1295293476740 timeout:undefined
pool test_pool - removeIdle() destroying obj - now:1295293476740 timeout:undefined
pool test_pool - removeIdle() destroying obj - now:1295293476740 timeout:undefined

Does the order of these lines need to be transposed?

availableObjects.push(obj);
objectTimeout[obj] = (new Date().getTime() + idleTimeoutMillis);

Rob

Support errors on acquire

Currently the create() / acquire() method that is used to get a resource only returns one object in a callback - the resource in question. There are a few cases (invalid input, no disk space) in which object cannot be acquired and an error parameter would be useful. A simple-ish implementation of this could also ensure backwards compatibility.

ev_ref and ev_unref are deprecated

I'm getting these warnings:

WARNING: ev_ref is deprecated, use uv_ref
WARNING: ev_unref is deprecated, use uv_unref

Otherwise it's working great! Thanks.

Sorry - the warnings are not related to generic-pool. They are related to the db-mysql module.

Please close.

add "reset" method

"reset" method that can be overridden is a very important addition that will let a "resting" a resource before putting it back to the pool. The resource shouldn't be available until it was "rested" and should be released on err.

var poolModule = require('generic-pool');
var pool = poolModule.Pool({

    reset: function(resource, done) { 
          resource.clean(function(err){}{ 
             done(err);
          });
    }

}

Potential bug with use of object as a key (objectTimeout bug?)

Does using an object as a key actually do what the code thinks it should?

availableObjects.push(obj);
objectTimeout[obj] = (new Date().getTime() + idleTimeoutMillis);
var size = 0;
objectTimeout.test = 12345;
for (i in objectTimeout) { size++; }
log("timeout: " + objectTimeout[obj] + ", length: " + size);

In the logs:

pool test_pool - timeout: 1295298653460, length: 1
pool test_pool - timeout: 1295298653461, length: 1
pool test_pool - timeout: 1295298653461, length: 1

For instance, in JavaScript...

objectTimeout[{foo: 0}] === objectTimeout[{bar: 1}]

...which could be the cause of this.

Thanks!

aquire returning false

I just had a question about when aquire returns false. If it does this, that means all connections are currently in use. Does it automatically hold the request and wait for a connection to open up or does it just skip all remaining requests?

Removing something from pool

Sometimes you need to remove something from pool. For example you have a redis connection in pool and sometimes it's get corrpuped and pool keeps to borrow it to clients and there's no obvious way to stop it from borrowing it. Maybe you can keep it forever so it won't be borrowed, but it's not a good idea I guess. Maybe I miss something here?

What am I missing with releasing pool?

Ok I have a min of 2 max of 3. I run a query on a page hit. I hit the page once fine. I refresh gets another connection, fine. Hit the page again it grows the pool and gets a connection. Hit the page again and nothing happens. I would think that if I'm releasing the pool after each db query then the pool would shrink? I thought that was the whole point of a pool?

var pool = poolModule.Pool({
    name: 'mysql',
    create: function (callback) {

        console.log('create');

        var connection = mysql.createConnection({
            host: 'localhost',
            user: 'admin',
            password: 'test',
            database: 'test'
        });

        connection.connect(function () {
            return callback(null, connection);
        });

    },
    destroy: function (client) { // this is never called.  ever

        console.log('destroy');
        client.destroy();
    },
    max: 3,
    min: 2,

    idleTimeoutMillis: 3000, //changed this  to 3000
    log: true
});


pool.query = function (query, data, callback) {

    try {

        pool.acquire(function (err, client) {

          //called 3 times.  never called again.

            client.query(query, data, function (err, results, fields) {

                try {

                    pool.release(client); // I've also tried pool.destroy(client);          
                    return callback(err, results);

                } catch (err) {
                    console.log(err)
                }

            });
        });
    } catch (err) {
        console.log(err);
    }   
};

Can this be used for a TCP server?

Imagine I have a net.createServer setup, and want to keep a maximum of connections, and delete idle connections, could this module be used for that? I'm failing to understand the logic on creating a "client" socket pool on a server, like how can I limit the connections one server can receive? And how the old idle connections are dropped? Would I need another façade class for the new connections?

Invoke createResource() manually?

Hi..
Is it possible to manually invoke createResource?

It's useful, for an instance if we're going to maintain the minimum threshold to be the 'minimum available resources' and not 'minimum resources regardless available or used'. Then we could call createResource() when needed.

should keep a list of "borrowed objects"

Currently we only keep a list of available objects/resource, and then rely on manual bookeeping to keep track of the number of borrowed items. This leaves lots of room for accounting errors and getting things into an inconsistent state.

We should have a list/array of borrowed objects.

Allow resource validation time to be configurable

Currently resources are only tested when being pulled from the available pool.

We should add flags to allow (names debatable).

  • testOnBorrow
  • testOnReturn
  • testOnCreates
  • testWhileIdle - idle objects in the pool will be validated by the idle object evictor reapIntervalMillis

(unrelated, we should remove factory.refreshIdle, renameidleTimeoutMillis to something like getMinEvictableIdleTimeMillis and only refresh idle things if we have reapIntervalMillis

By default testOnBorrow will be true if validation function is supplied, false if not and testOnReturn will be false (which roughly follows current conventions)
I borrowed those names from some apache generic object pool

sample code required

hello,

can i have sample running code for connection pooling in node js.

thanks
pranav

Max lifetime parameter

Can you add this parameter? I use node-pool to create pool of FireFox browsers for parsing sites, and every 2h I have to restart my script because of memory leak. Can you help me with my project?=)

Update tests to new version of expresso

diff --git a/test/generic-pool.test.js b/test/generic-pool.test.js
index 8ac55f6..9352136 100644
--- a/test/generic-pool.test.js
+++ b/test/generic-pool.test.js
@@ -1,8 +1,9 @@
 var poolModule = require('generic-pool');
+var assert = require('assert');

 module.exports = {

-    'expands to max limit' : function (assert, beforeExit) {
+    'expands to max limit' : function (beforeExit) {
         var createCount  = 0;
         var destroyCount = 0;
         var borrowCount  = 0;
@@ -36,7 +37,7 @@ module.exports = {
         });
     },

-    'supports priority on borrow' : function(assert, beforeExit) {
+    'supports priority on borrow' : function(beforeExit) {
         var borrowTimeLow  = 0;
         var borrowTimeHigh = 0;
         var borrowCount = 0;
@@ -83,7 +84,7 @@ module.exports = {
         });
     },

-    'removes correct object on reap' : function (assert, beforeExit) {
+    'removes correct object on reap' : function (beforeExit) {
         var destroyed = [];
         var clientCount = 0;

Maintaining a minimum active number of resources.

I'm having trouble getting my pool keep a minimum number of resources alive. Essentially, I want to have at least 4 available at all times and spawn up to 4 more when load becomes heavier.

This is my code:

this.generatorPool = poolModule.Pool({
    name: 'generator-pool',
    create: function (resolveResource) {
      var generator = new Generator();
      generator.start().then(
        // Generator load successful
        function() {
          resolveResource(null, generator);
        },
        // Generator failed to load
        function () {
          resolveResource(new Error('Failed to load generator'));
        }
      );
    },
    destroy: function (generator) {
      console.log('DESTROYING THE GENERATOR)
      generator.stop();
    },
    max: 8,
    min: 4,
    refreshIdle: false,
    idleTimeoutMillis: 30000
  });

For some reason the first 4 resources are still being destroyed after 30 seconds. What could be up?

Heads up about .bind() and #14

First, thanks for merging #14.

I just ran into an interesting edge case where the callback .length based arg signature detection fails. If you use .bind() on a function, node will generate a new function which appears to be wrapped in such a way that .length does not accurately represent the "inner" function's arguments, e.g.

pool.acquire(function(err, resource) {
    // this works.
});
pool.acquire(function(err, resource) {
    // this fails -- looks like a 1 arg callback to node-pool.
}.bind(this));

I'm not sure what the best solution will be to this problem, or if it will affect most API users at all. It's easy enough to workaround so it might be worth documenting until we have a fix or the 1 arg callback support can be deprecated.

Events instead of log function

Hi,

I'd like to log my own custom errors. Also, I don't want to log every second the message "availableObjects.length=x". Can you inherite the pool from EventEmitter and just emit events so we can catch them and do whatever we want? It's a very easy modification. Just replace this line when you export the module:

var me = {},

By this:

var Pool = function (){
  events.EventEmitter.call (this);
};
Pool.prototype = Object.create (events.EventEmitter.prototype);
Pool.prototype.constructor = Pool;
var me = new Pool (),

You must add the events module:

var events = require ("events");

Then, instead of calling the log function you can throw an event with the data. For example, the createResource function:

me.emit ("create-resource", count, factory.min, factory.max);

And to catch it from outside your library:

pool.on ("create-resource", function (count, min, max){
  ...
});

Still maintained?

Hi,

Since there are some old unresolved issues, I was just wondering if this package is still maintained?

wasteful destroy/create of idle resources

I'm setting a pool minimum and still seeing a wasteful destroy/create cycle for each resource in the queue. Given this simple nodejs file:

var i = 0;
var Pooler = require('generic-pool');
var pool = Pooler.Pool({
    idleTimeoutMillis : 3000,
    reapInterval : 1000,
    create: function(callback){
        console.log('expensive create', i);
        i++;
        callback(null, {i:i});
    },
    destroy: function(client){
        console.log('destroy', client.i);
    },
    min: 5,
    max: 10
});

I'm seeing the following log:

expensive create 0              
expensive create 1              
expensive create 2              
expensive create 3              
expensive create 4              
destroy 1                       
expensive create 5              
destroy 2                       
expensive create 6              
destroy 3                       
expensive create 7              
destroy 4                       
expensive create 8              
destroy 5                       
expensive create 9              
destroy 6                       
expensive create 10             
destroy 7                       
expensive create 11             
destroy 8                       
expensive create 12             
destroy 9                       
expensive create 13             
...

The behavior of destroying then immediately recreating these resources doesn't seem like desired behavior. Yes these timeouts are short, but I'd expect the pooler to only cleanup resources above my specified minimum that were unused. Am I missing something here?

factory.min should be able to be == factory.max

The documentation says
"if this is set > max, the pool will silently set the min to factory.max - 1"

In fact, if you try to set min == max, which should be allowed by the statement above, it is set to max - 1. I worked around this by resetting it in the first call to my create() function. It works fine with min == max, and keeps exactly max objects in the pool as expected.

Thanks for this great library...it really helped me out.

adjustCallback to use callbacks without params

Please, for some reasons, it's useful to put a callback function that does not have any params defined,
and in this case adjustCallback function calls callback with one parameter only, without an error information...
I suggest to change there a condition
to send ONE parameter only in ONE case - cb.length == 1, not cb.length<=1...
original function:

  function adjustCallback(cb, err, obj) {
    if (!cb) return;
    if (cb.length <= 1) {
      cb(obj);
    } else {
      cb(err, obj);
    }
  }

Default values for create() and destroy() ?

Hi,

I'm using node-pool just as a concurrency limiter, without the need to create or destroy resources each time. Hence I don't need the create() and destroy() arguments but they are mandatory. Could there be default values for them ? like

             'create':function(callback) {
                callback();
            },
            'destroy':function() {}

Thanks!

me.destroy() not destroying

I don't know if I'm missing something, but in the code, me.destroy() is as follows:

https://github.com/coopernurse/node-pool/blob/master/lib/generic-pool.js#L114

me.destroy = function(obj) {
    count -= 1;
    factory.destroy(obj);
};

and, looking at the possible scenarios:

  • If the connection that is going to be destroyed it's available, then it's on availableObjects array, which is an array of objects of the form {obj:obj, timeout:timeout_in_ml}, just disconnecting the object (which is what the custom defined 'destroy' function at the constructor is meant for) won't remove the element {obj:obj, timeout:timeout_in_ms} from the availableObjects array, thus that item will still be available, but the object it returns won't be working anymore!.
  • If the connection that is going to be destroyed it's "in use", it should also be released (or that could be left in the hands of the user, I don't know).

I guess when destroying the conn, the 'availableObjects' corresponding item should be shifted, too.

Please, let me know if I'm missing something. I've read the code a zillion times and can't find how it can be otherwise than what I just said.

txs

Grow pool

Would it be an idea to be able to grow and shrink the pool ? I know it would in my use case :)

Inactive sessions in oracle when connection-pooling with node-pool's generic-pool

We are using generic-pool with Oracle back end for connection pooling in our project. We are seeing a weird issue of inactive connections/sessions in oracle.
Our factory.max is set to 10 and min is set 2 so our understanding is that, number of connections won’t cross 10, but somehow in oracle we see a connection per request and its goes up very high till it reaches oracle limit (about 500). We can see old connections in inactive state in oracle and when we stop the node server all these connections go away from oracle. Our connection pool’s size never reaches 10, it keeps jumping between 4 - 6, and I am not sure what is going wrong.
We have kept idleTimeoutMillis to 5 minutes, but connections stay in inactive state even beyond 5 minutes. I experimented with reducing the time to 30 secs, no help.
In our code we are releasing connections, and that can be verified because connection pool size never reaches 10. any suggestion help is much appreciated.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.