Git Product home page Git Product logo

metroplex's Introduction

Metroplex

Version npmCI

Metroplex a Redis based spark/connection registry for Primus.

Installation

Metroplex is released in the npm registry and can therefor be installed using:

npm install --save metroplex

Once you've installed the module you need to tell Primus to use the plugin which is done using the primus.plugin method:

'use strict';

var http = require('http').createServer()
  , Primus = require('primus')
  , primus = new Primus(http, { transformer: 'sockjs' });

primus.plugin('metroplex', require('metroplex'));

Usage

In the example above you've seen how to add the plugin to your Primus server but not how to configure it. We have various of options that can be configured in this plugin:

  • redis: Metroplex is currently using Redis as its default back-end for storing the state of the connections. If you do not supply us with a pre-defined Redis client (or authorized) we will create a Redis client which only connects to localhost and Redis's default port number. When provided this must be an ioredis client.
  • namespace: As the databases are usually shared with other programs it's good to prefix all the data that you store, in Metroplex we prefix every key with the set namespace. The default namespace is metroplex.
  • interval: We are using "alive" suffixed keys in the database to see which node process is still alive. The interval determines the interval of these updates. When the interval is reached we update the key in the database with the current EPOCH as well as start a scan for possible dead servers and removing them. The default interval 300000 ms
  • latency: The maximum time it would take to update the alive key in Redis. This time is subtracted from the set interval so we update the key BEFORE it expires. Defaults to 2000 ms.
  • address The address or public URL on which this SPECIFIC server is reachable. Should be without path name. When nothing is supplied we try to be somewhat smart and read the address and port and server type from the server that Primus is attached to and compose an URL like: http://0.0.0.0:8080 from it.

These options should be provided in the options object of the Primus server:

primus = new Primus(http, {
  transformer: 'sockjs',
  namespace: 'metroplex',
  redis: require('redis').createClient()
});

primus.plugin('metroplex', require('metroplex'));

Metroplex

The orchestration is all done using the metroplex library which is bundled in this plugin. The Metroplex instance is exposed on the Primus instance when you use this plugin:

primus.metroplex.servers(function (err, servers) {
  console.log('registered servers:', servers);
});

The following public methods are available.

metroplex.servers

metroplex.servers(fn)

List all the servers in our current registry.

metroplex.servers(function (err, servers) {
  console.log(servers);
});

metroplex.spark

metroplex.spark(id, fn)

Get the server for the given spark id. It does not check if the spark is hosted on the current server. That's up to the developer to implement.

metroplex.spark(id, function (err, server) {
  console.log(server);
});

metroplex.sparks

metroplex.sparks(sparks, fn)

Get the servers for each id in the given sparks array. It will return an object and just like metroplex.spark it does not check if the spark is hosted on the current server.

Omega Supreme integration

If you load the omega-supreme plugin before metroplex, you can use some additional convenience methods. These methods are added to the primus.forward object:

forward.broadcast

forward.broadcast(msg, fn)

Broadcast a message to all sparks in the cluster.

forward.broadcast('data', function (err, result) {
  // result is an object with details about the result of the operation.
  console.log(result);
});

forward.sparks

forward.sparks(ids, msg, fn)

Send a message to a set of sparks in the cluster.

forward.sparks(['ad8a-280z-18', 'y97x-42480-13'], 'data', function (err, result) {
  console.log(result);
});

forward.spark

forward.spark(id, msg, fn)

Send a message to a single spark in the cluster.

forward.spark('ad8a-280z-18', 'data', function (err, result) {
  console.log(result);
});

License

MIT

Metroplex

metroplex's People

Contributors

3rd-eden avatar greenkeeper[bot] avatar lpinca avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

metroplex's Issues

error using omeg-supreme method

I have added metroplex with omega-supreme.

  primus.use( 'omega-supreme',    require( 'omega-supreme'    )); // before metroplex
  primus.use( 'metroplex',        require( 'metroplex'        )); // registry/map 

Then, I have the following code snippet

primus.forward.sparks( sparks2send, 
                      msg,
                     function( err, results ){
        if( err ) {
          Logger.errorc( 'Error: ' + err  );
          return done( err, null );
                }
//  ........
});

The code above prints following error:

Error: Error: Invalid URI "/primus/omega/supreme"

Example list of sparks:

sparks2send = ["64xRm74gJ2dsMXVJAAAA","5iQsCn0grQ5EImWMAAAB","LBD9Vtykb5vo5qWrAAAB","UoA4Ir54PP9sCO-qAAAB","mlNYdagAleP1AzuJAAAA","xoc7pJzICFj1IlINAAAB","ffSiCA3n6HGVaUO5AAAA","Wi744PWyRCbmp5OfAAAI","nbIvK_GwjSLKGihtAAAA","eE5FnKSTBxNryBRXAAAA","BfadUHua5RZ3HVqEAAAA","ySd-xtbqgHo4gLNmAAAA","W5ioExC5VRKwcxB0AAAA","vnKVluWrieQP0jCSAAAA"]

I may be missing something in configuration I think:

  primus_options =   {  
        pathname:     '/primus'   // The URL namespace that Primus can own
      , parser:       'JSON'
      , transformer:  'engine.io'     
      , namespace:    'mp'        // for 'mp'=metroplex prefixed keys
      , address:      add
      , username:     'user'  
      , password:     'password'  
      , redis: Redis.getClient()
      , concurrently: 30 //How many servers can we broadcast to at once, defaults to 10
      , url: '/primus/omega/supreme'  //-->doesn't work with/without this option
   };

questions

Hi,

  1. Are spark IDs always unique?

  2. What would be the most efficient way to associate additional information with each spark ID? So, even if sparks are disconnected and reconnected, we could transfer the associated info to new spark IDs.

  3. Can we easily make it check current server for spark IDs etc (ref: .sparks())?

  4. Does forward.sparks() also include current/local server for sending messages?

Thanks.

send_command is not a function

I am not sure if its an actual issue (could be a documentation bug at least).

I am currently testing metroplex and found out that its somehow dependent on the particular Redis library you're using with it (ie. https://github.com/mranney/node_redis). The reason is that it is using a function that is not supported by one of the popular Redis client libraries ioredis.

For details, see the issue I opened with ioredis --> redis/ioredis#90

security update reqd for lodash dependency

┌───────────────┬──────────────────────────────────────────────────────────────┐
│ high          │ Prototype Pollution                                          │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package       │ lodash                                                       │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Patched in    │ >=4.17.12                                                    │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ metroplex                                                    │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path          │ metroplex > async > lodash                                   │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info     │ https://nodesecurity.io/advisories/1065                      │
└───────────────┴──────────────────────────────────────────────────────────────┘

node process crashes with error in metroplex+leverage

I am using the latest metroplex and primus branches and when I kill node process using ^C on MacOSX, it crashes with following trace.

2015-10-15T13:53:44.968Z - error: uncaughtException: Connection is closed. 
date=Thu Oct 15 2015 09:53:44 GMT-0400 (EDT), pid=6407, uid=501, gid=20, cwd=/test3, execPath=/usr/local/bin/iojs, version=v4.0.0, argv=[/usr/local/bin/iojs, /test3/app.js], rss=103784448, heapTotal=81414656, heapUsed=60043736, loadavg=[2.8427734375, 2.58837890625, 4.53515625], uptime=10027, 

trace=[column=21, file=/test3/node_modules/metroplex/node_modules/leverage/index.js, function=Leverage.seval, line=549, method=seval, native=false, 
column=21, file=/test3/node_modules/metroplex/node_modules/leverage/index.js, function=Leverage.evals [as annihilate], line=659, method=evals [as annihilate], native=false, 
column=22, file=/test3/node_modules/metroplex/metroplex.js, function=Metroplex.unregister, line=141, method=unregister, native=false, 
column=15, file=/test3/node_modules/metroplex/index.js, function=Primus.close, line=23, method=close, native=false, 
column=17, file=/test3/node_modules/primus/node_modules/asyncemit/index.js, function=each, line=63, method=null, native=false, 
column=5, file=/test3/node_modules/primus/node_modules/asyncemit/index.js, function=Primus.asyncemit, line=71, method=asyncemit, native=false, 
column=10, file=/test3/node_modules/primus/index.js, function=Primus.close, line=946, method=close, native=false, 
column=12, file=/test3/node_modules/primus/index.js, function=[object Object].close, line=925, method=close, native=false, 
column=15, file=timers.js, function=Timer.listOnTimeout, line=89, method=listOnTimeout, native=false], stack=[Error: Connection is closed.,     at Leverage.seval (/test3/node_modules/metroplex/node_modules/leverage/index.js:549:21),     at Leverage.evals [as annihilate] (/test3/node_modules/metroplex/node_modules/leverage/index.js:659:21),     at Metroplex.unregister (/test3/node_modules/metroplex/metroplex.js:141:22),     at Primus.close (/test3/node_modules/metroplex/index.js:23:15),     at each (/test3/node_modules/primus/node_modules/asyncemit/index.js:63:17),     at Primus.asyncemit (/test3/node_modules/primus/node_modules/asyncemit/index.js:71:5),     at Primus.close (/test3/node_modules/primus/index.js:946:10),     at [object Object].close (/test3/node_modules/primus/index.js:925:12),     at Timer.listOnTimeout (timers.js:89:15)]

An in-range update of ws is breaking the build 🚨

The devDependency ws was updated from 7.0.0 to 7.0.1.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

ws is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).

Commits

The new version differs by 9 commits.

  • 38d3bf2 [dist] 7.0.1
  • a0af764 [test] Use the correct value for the Content-Length header
  • b086179 [fix] Allow to disable sending the SNI extension (#1587)
  • e9e8ba5 [pkg] Update eslint-config-prettier to version 5.0.0 (#1588)
  • 36ef757 [doc] Add missing dependency in code snippet (#1581)
  • 911bb6f [minor] Fix typo in JSDoc comment (#1565)
  • 8050d5f [lint] Enable quotes rule
  • 995c527 [test] Enable --throw-deprecation
  • fbc077b [test] Do not use the deprecated outgoingMessage._headers property

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

Knowing when a server dies

I'm trying to create a real time application where I need to keep track of all my users (in real time), and I want to have high availability. Up until now I've been using socket.io, but the clustering support seems to be lackluster at best so I am seriously considering migrating to primus. I do have a few questions:

  1. I want to know if a server dies in real time, so that one of the servers left in the cluster can clean up after it. Does metroplex support something like this? If not I'll have to come up with my own solution to the problem.

  2. Using metroplex together with omega-supreme and primus-emit or emitter, can you get an acknowledgement when forwarding messages to a certain spark in another server? This would be a killer feature for me since in my case, users can interact individually with each other in real time, and I would like to acknowledge their interactions.

If these two things are available out of the box it'll be an easy choice... otherwise I have to decide in which environment it'll be easier to implement these two things

broadcasting through omega-supreme only works on the same process

this could possibly be a documentation bug or perhaps am doing something wrong.

through my tests, it was revealed that the broadcast messages are only sent to clients connected to the same node process and not to the "cluster" as is mentioned in the documentation.

Here is whats written: "Broadcast a message to all sparks in the cluster."

I tested by using two node processes behind HAProxy. I fired up two browser sessions and made sure clients were connected to separate node processes. I sent messages and only received them back on the sender but the other client that was connected to the other node process did not receive anything.

I killed one of the node processes and forced the client connected to it to use the other node process (behind HAProxy). This time I could see that messages were received on both clients (because both were connected to the same process now).

The code I used to broadcast message is:

                primus.forward.broadcast( data.message, function (err, result) {
                  logger.debug("In send message: " + JSON.stringify( result ) + 
                                         " error: " + JSON.stringify( err ));
                });

The debug line returned (after a few minutes of sending message).

debug: In send message: [null] error: {"code":"ECONNRESET","url":"http://192.168.1.9:3000/primus/omega/supreme","status":500,"body":{},"type":"broadcast","packet":{"sparks":"","msg":"hello"}}

Its the same error object that is returned no matter if the messages were delivered or not.

change description

The description is cute and all but I could barely make sense of what the project does. Same with omega supreme. It wasn't until I read this issue that it made sense.

Please change the description or make it more clear what this project does.

Cannot read property 'reduce' of undefined

I got this error when I tested sending msgs through primus.forward.sparks connected to separate instances

error: uncaughtException: Cannot read property 'reduce' of undefined date=...., cwd=/test3, ...
stack=[TypeError: Cannot read property 'reduce' of undefined,     at calculate (/test3/node_modules/metroplex/omega.js:73:24),     at done (/test3/node_modules/metroplex/node_modules/async/lib/async.js:167:19),     at /test3/node_modules/metroplex/node_modules/async/lib/async.js:40:16,     at calculate (/test3/node_modules/omega-supreme/supreme.js:188:7),     at /test3/node_modules/omega-supreme/node_modules/async/lib/async.js:251:17,     at /test3/node_modules/omega-supreme/node_modules/async/lib/async.js:200:33,     at /test3/node_modules/omega-supreme/node_modules/async/lib/async.js:248:21,     at Request.requested (/test3/node_modules/omega-supreme/supreme.js:177:9),     at Request.self.callback (/test3/node_modules/omega-supreme/node_modules/request/request.js:368:22),     at emitTwo (events.js:100:13),     at Request.emit (events.js:185:7),     at Request.<anonymous> (/test3/node_modules/omega-supreme/node_modules/request/request.js:1219:14),     at emitOne (events.js:95:20),     at Request.emit (events.js:182:7),     at IncomingMessage.<anonymous> (/test3/node_modules/omega-supreme/node_modules/request/request.js:1167:12),     at emitNone (events.js:85:20),     at IncomingMessage.emit (events.js:179:7),     at endReadableNT (_stream_readable.js:913:12),     at _combinedTickCallback (node.js:377:13),     at process._tickDomainCallback (node.js:425:11)]

getting spark objects

I can see that messages can be sent to Sparks using spark IDs but how do we get the actual spark object to be used, for example, in primus-emitter.

Something like:

primus.getSpark( [<spark ID1>,..], function( err, sparks ){
  sparks[0].send(....
});

The reason why I want to use primus-emitter is because I want to get a return value from the client right away on the same object.

An in-range update of ws is breaking the build 🚨

The devDependency ws was updated from 6.1.0 to 6.1.1.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

ws is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details
  • continuous-integration/travis-ci/push: The Travis CI build failed (Details).

Release Notes for 6.1.1

Bug fixes

  • Queued messages, when permessage-deflate is enabled, are now discarded if the
    socket closes prematurely (#1464, #1471).
Commits

The new version differs by 15 commits.

  • 029de0c [dist] 6.1.1
  • b213bee [pkg] Update list of published files
  • 95bf991 [pkg] Update dev dependencies
  • f26fac8 [minor] Ignore callbacks when clearing the send queue (#1471)
  • 5914206 [doc] Fix nits
  • 3fa0e03 [doc] Suggest implementation of heartbeat on the client (#1469)
  • 7d51fb9 [fix] Do not waste time compressing when socket is closed (#1464)
  • 1ebff19 [ci] Test on node 11
  • 7e061bc [pkg] Update nyc to version 13.1.0
  • bc0f8ab [pkg] Update eslint to version 5.8.0
  • 45f817b chore(package): update eslint-plugin-node to version 8.0.0 (#1466)
  • cddbcf6 [test] Remove unused variables
  • 0da3fdb [minor] Do not use the legacy URL API
  • d2317b1 [benchmark] Add Unix domain sockets to bench (#1456)
  • 9022a0d [doc] Remove clientMaxWindowBits option from README example (#1454)

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.