apparatus / mu Goto Github PK

Im new to the scene and trying to understand. Can someone compare and contrast mv and senecajs transport?

we need to clearly define timeout logic, and provide options to configure

I'm seeing this a lot:

    mu.log.debug('node: ' + muid + ' <- ' + JSON.stringify(msg))

That's going to JSON.stringify every time, even if the logging level is above debug

Instead let pino deal with it

    mu.log.debug('node: %s <- %j', muid, msg)

This way the object will only be (safely) stringified if logging is on

Question: Why I have two error objects in the remoteStack which point to the same error in the service c ?

I create following scenario:

service-a

const mu = require('mu')({
  dev: process.NODE_ENV !== 'production'
})
const tcp = require('mu-tcp')

// define routing:

mu.inbound({
  role: 'some'
}, tcp.server({
  port: 3000,
  host: '127.0.0.1'
}))

mu.outbound({
  role: 'other'
}, tcp.client({
  port: 3001,
  host: '127.0.0.1'
}))

// define patterns:

mu.define({
  role: 'some',
  cmd: 'sub'
}, function (args, cb) {

  mu.dispatch({
    role: 'other',
    cmd: 'sub',
    a: 1,
    b: 2
  }, function (err, result) {

    if (err) {
      return cb(err);
    }

    cb(null, result);
  })
})

service b

const mu = require('mu')({dev: process.NODE_ENV !== 'production'})
const tcp = require('mu-tcp')

// define routing:

mu.outbound({role: 'some'}, tcp.client({port: 3000, host: '127.0.0.1'}))

// define patterns:

mu.dispatch({ role: 'some', cmd: 'sub', a: 1, b: 2 }, function (err, result) {
  
  console.log(JSON.stringify(err))
})

service c

const mu = require('mu')({
  dev: process.NODE_ENV !== 'production'
})
const tcp = require('mu-tcp')

// define routing:

mu.inbound({
  role: 'other'
}, tcp.server({
  port: 3001,
  host: '127.0.0.1'
}))

// define patterns:

mu.define({
  role: 'other',
  cmd: 'sub'
}, function (args, cb) {
  
  cb(mu.error.wrapRemote(mu.error('invalid'), 1, 500, { msg: 'jesus' }))
  
})

Steps to reproduce

1. Start service c
2. Start service a
3. Start service b

I got following error payload

{
  "data": null,
  "isBoom": true,
  "isServer": true,
  "output": {
    "statusCode": 500,
    "payload": {
      "statusCode": 500,
      "error": "Internal Server Error",
      "message": "An internal server error occurred",
      "mu": {
        "code": 1,
        "error": "service error",
        "message": "invalid"
      }
    },
    "headers": {},
    "mu": {
      "code": 1,
      "error": "service error",
      "message": "invalid"
    }
  },
  "isMu": true,
  "remoteStacks": [{
        "timestamp": 1478711395108,
        "stack": "Error: invalid\n    at makeMuError (E:\\Repositorys\\mu-test\\node_modules\\mu-error\\index.js:56:21)\n    at monoMorphMuError (E:\\Repositorys\\mu-test\\node_modules\\mu-error\\index.js:188:14)\n    at Object.mue [as error] (E:\\Repositorys\\mu-test\\node_modules\\mu-error\\index.js:49:14)\n    at Object.tf (E:\\Repositorys\\mu-test\\service-c.js:22:29)\n    at Object.route (E:\\Repositorys\\mu-test\\node_modules\\mu-router\\index.js:71:14)\n    at Object.dispatch (E:\\Repositorys\\mu-test\\node_modules\\mu\\index.js:87:12)\n    at receive (E:\\Repositorys\\mu-test\\node_modules\\mu-transport\\index.js:56:8)\n    at DestroyableTransform._transform (E:\\Repositorys\\mu-test\\node_modules\\mu-tcp\\driver.js:87:11)\n    at DestroyableTransform.Transform._read (E:\\Repositorys\\mu-test\\node_modules\\readable-stream\\lib\\_stream_transform.js:159:10)\n
        at DestroyableTransform.Transform._write(E: \\Repositorys\\ mu - test\\ node_modules\\ readable - stream\\ lib\\ _stream_transform.js: 147: 83)
        "},{"
        timestamp ":1478711395110,"
        stack ":"
        Error: invalid\ n at Error(native)\ n at Function.wrapRemote(E: \\Repositorys\\ mu - test\\ node_modules\\ mu - error\\ index.js: 132: 19)\ n
        at Object.tf(E: \\Repositorys\\ mu - test\\ service - c.js: 22: 15)\ n at Object.route(E: \\Repositorys\\ mu - test\\ node_modules\\ mu - router\\ index.js: 71: 14)\ n at Object.dispatch(E: \\Repositorys\\ mu - test\\ node_modules\\ mu\\ index.js: 87: 12)\ n at receive(E: \\Repositorys\\ mu - test\\ node_modules\\ mu - transport\\ index.js: 56: 8)\ n at DestroyableTransform._transform(E: \\Repositorys\\ mu - test\\ node_modules\\ mu - tcp\\ driver.js: 87: 11)\ n at DestroyableTransform.Transform._read(E: \\Repositorys\\ mu - test\\ node_modules\\ readable - stream\\ lib\\ _stream_transform.js: 159: 10)\ n at DestroyableTransform.Transform._write(E: \\Repositorys\\ mu - test\\ node_modules\\ readable - stream\\ lib\\ _stream_transform.js: 147: 83)\ n at doWrite(E: \\Repositorys\\ mu - test\\ node_modules\\ readable - stream\\ lib\\ _stream_writable.js: 313: 64)
        "}],"
        message ":"
        invalid ","
        stack ":"
        Error: invalid\ n at Error(native)\ n at Function.wrapRemote(E: \\Repositorys\\ mu - test\\ node_modules\\ mu - error\\ index.js: 132: 19)\ n at Object.route(E: \\Repositorys\\ mu - test\\ node_modules\\ mu - router\\ index.js: 111: 30)\ n at Object.dispatch(E: \\Repositorys\\ mu - test\\ node_modules\\ mu\\ index.js: 87: 12)\ n at receive(E: \\Repositorys\\ mu - test\\ node_modules\\ mu - transport\\ index.js: 56: 8)\ n
        at null. < anonymous > (E: \\Repositorys\\ mu - test\\ node_modules\\ mu - tcp\\ driver.js: 59: 11)\ n at emitOne(events.js: 77: 13)\ n at emit(events.js: 169: 7)\ n
        at readableAddChunk(E: \\Repositorys\\ mu - test\\ node_modules\\ readable - stream\\ lib\\ _stream_readable.js: 198: 18)\ n at Readable.push(E: \\Repositorys\\ mu - test\\ node_modules\\ readable - stream\\ lib\\ _stream_readable.js: 157: 10)
        "}

server.close() has a callback, this is important for tests when you want to do mu.tearDown(() => t.end())

if you do t.end() outside of cb, test ends before server closes

Hello anyone have a project using mu-redis that I can check out?

1. it should be a hard requirement that patterns are registered at initialization (as far as I can tell there's no reason for lazy initialization of patterns)
2. since this is the case, we can run the process with an instrumented mu (proxyquire), which could allow us to grab all patterns and functions
3. we can use these patterns (and functions) to generate api doc template

and generally calling functions on objects that hoist values to scoped variables

I think to be more functional, we should be taking a more vertical state approach
by which I mean passing all state into a function which calls another functions, and so on

cc @pelger @mcollina @lucamaraschi @mcdonnelldean

can we just get rid of the suffix?

it's already in the test folder, we know it's a test (put fixtures in their own sub folder)

Should be fairly straightforward to implement, one catch though - have alt browser client only incarnation that doesnt use streams

Hey @davidmarkclements, @mcollina,
Just reviewing the mu code so wanted to flag this one. It seems to me that the API is a little inconsistent. namely mu.define receives the entire message (pattern and protocol) whilst the final callback handler receives err and result. IMO the following might be better:

mu.define({...}, function(args, cb, msg)

where args = msg.pattern
cb as before
msg = full message (i.e. pattern + protocol)

mu.dispatch({...}, function(err, result, msg)

where result = msg.response
msg = full message (i.e. response + protocol)

this way the msg can be ignored in most cases but is available for inspection if required.

Thoughts?

integration with streams (Node streams and pull-streams) should be a core concept, and streaming API's should be first class citizens.

I think the earlier we can deal with this the better

mu.define.stream({role: 'realtime-thing', cmd: 'process-data'}, function (args, stream) {
  // no cb, errors handled on stream itself
  stream.pipe(createTransformFrom(args)).pipe(stream) // stream is a duplex, writable end is response hooked into transport
})

// elsewhere:

var stream = mu.dispatch.stream({role: 'realtime-thing', cmd: 'process-data'}) // no cb, events handled on stream (also maybe client should (optionally?) return pull-stream)
stream.on('error', handleIt)
stream.write('some data YOOOOO')
stream.write('Its all realtime in here init')
if (thereIsAProblem) { 
  stream.emit('error', mu.error('AHHHHHHH noooo'))
} else {
  stream.end('yay')
}

If a transport doesn't yet or can't support streaming, we can simply throw immediately when define.stream or dispatch.stream is called

cc @mcollina @pelger @mcdonnelldean

I think perhaps an explicit API is easier/cleaner from a user and implementation POV, than embedding streams in patterns i.e.:

mu.define({role: 'realtime-thing', cmd: 'process-data'}, function (args, cb) {
   if (invalid(args) { return cb(mu.error('invalid args for some reason')) }
   cb(null, {stream: args.data.pipe(someTransform)})
})

mu.dispatch({role: 'realtime-thing', cmd: 'process-data', data: myStream}, function (err, res) {
    if (err) { return handle(err) }
    // callback signifies that args were handled, and stream was instantiated
    var stream = res.stream
    stream.on('error', handleStreamErr)
    stream.pipe(someWhereElse)
})

could be useful, propagate cb for listen call up

See > https://github.com/mcdonnelldean/nanite/blob/master/.travis.yml#L4-L7

• Need a travis file
• Need to ensure npm commands are correctly set up
• Need to ensure precommit is active to stop bogus builds.

Something like https://github.com/mcollina/fastparallel, https://github.com/mcollina/steed or https://github.com/caolan/async needs to be used.

prime use case is hydrating local state from a remote source (particularly in a web app scenario) then replay from local

mu.outbound('*', hydrate(local(), ws.client(opts)))

@pelger @mcollina is there an analogous term for this (as with tee and balance)

We should replace it with https://www.npmjs.com/package/pino

currently we throw an assertion error if dispatch has no cb

we should allow for fire and forget - no cb required

@pelger @mcollina @mcdonnelldean

since accessing the pattern is definitely the main case, what if we make it the
args object, and make protocol accessible either through an additional parameter or on the proto of the args object. From the transport, given we have an object (o) that is composed {pattern, protocol} we can do

var args = o.pattern
pattern.__proto__ = {protocol: o.protocol}

Then we would use like so

mu.define({role: 'some', cmd: 'thing'}, function (args, cb) {
   console.log(args.role === 'some') // true
   console.log(args.protocol.ttl) // 10
})

since protocol is on the proto, it won't show up during serialization (or logging, but we can make a pino serializer)

The other approach would be

mu.define({role: 'some', cmd: 'thing'}, function (args, cb, protocol) {
   console.log(args.role === 'some') // true
   console.log(protocol.ttl) // 10
})

Or we could go polymorphic with (args, cb) and (args, protocol, cb) being appropriately handled

cc @pelger @mcollina

This can be done easily via travis + package.json

• Package.json > https://github.com/mcdonnelldean/nanite/blob/master/package.json#L15
• Travis > https://github.com/mcdonnelldean/nanite/blob/master/.travis.yml#L10

https://www.npmjs.com/package/es2040

This enforces a subset of useful ES6 features and prevents the others by matter of course

Transpiles this es6 subset for older browsers (obviously faster because it's more focused)

No need for a server side transpile step - all of es2040 is supported in v6

It could be useful to figure out a way to make it a secondary linter (after standard) - maybe start with something simple as attempting to transpile and then "oops that's not es2040"

ES2040 supported features:

• fat arrows - make inline functions cute-looking
• template strings / tagged templates - enable DSLs inside of JS
• const - using const by default makes it easy to spot where values are
  being redeclared
• object destructuring - const { a, b } = { a: 1, b: 2 }
• array destructuring - const [a, b] = [1, 2]
• default parameters - ({ a = 1, b = 2 } = {}) => a
• rest parameters - (a, b, ...args) => {}
• spread literals - f(a, b, ...args)
• short-hand properties - return { a, b }
• computed properties - return { [a]: b }

If we want to be even more strict, we could go with ES2020 is:

• fat arrows - make inline functions cute-looking
• template strings / tagged templates - enable DSLs inside of JS
• const - using const by default makes it easy to spot where values are
  being redeclared

Which would be probably be wicked fast at transpiling.

In either case, much faster than babel.

Thoughts? @mcdonnelldean @lucamaraschi @mcdonnelldean @mcollina

I like that we're going with the assert approach (apparently known as design by contract)

I think we should go all out with assert and just assert the crap out of everything

And then "unassert" strip all asserts (in a production server build and browserify build) using http://npm.im/unassert-cli and http://npm.im/unassertify

The only caveat I can think of, is care should be taken when defensive programming is required for production scenarios - in which case don't use assert, use an if statement

this module should be core

drivers and adapters should have their own modules

currently for in-process we have to do

var func = require('mu/drivers/func')
var createMu = require('mu')
var mu1 = createMu()
var mu2 = createMu()
mu1.inbound('*', func())
mu2.outbound('*', func({target: mu1})

I think this should be simpler, particularly for the browser case

var func = require('mu/drivers/func')
var mu = require('mu')()
mu.inbound('*', func())
mu.outbound('*', func({target: mu})) // currently causes max stack error on dispatch

or preferably, no need to specify target:

var func = require('mu/drivers/func')
var mu = require('mu')()
mu.inbound('*', func())
mu.outbound('*', func())

This allows for something like

var func = require('mu/drivers/func')
var http = require('mu/drivers/http')
var mu = require('mu')()
mu.inbound('*', http(opts))
mu.inbound({role: 'local-state'}, func())
mu.outbound({role: 'local-state'}, func())

http://npm.im/blocked is nice

@mcollina which was yours again?

there's also http://npm.im/toobusy but it's a native compile mod

let's use whichever adds minimum overhead (and maybe ignore native compile)

I just play something with the remote errors and I could not find a possibility to transfer more than just the timestamp and stack to the previous callee. I think those informations are good for logging but if you want to handle business cases it would be good to know what went wrong.

It would be nice to add also the message object to the stack item.

mu.error.wrapRemote(error, muCode, statusCode, message)

Performance should be a first level concern.

https://www.npmjs.com/package/fastbench

https://www.npmjs.com/package/tentacoli
https://github.com/mcollina/tentacoli/blob/master/benchmarks/bench.js

from a brief look its clear some pieces of mu are not browser friendly

for instance, requiring the entire lodash library to use _.isString when you can simply use typeof and _.cloneDeepwhen you can (and should) useObject.assign (polyfill for older browsers)

This would significantly reduce browser side payload

I think we should review the whole for browser side usage and keep the browser in mind
The first step being, no lodash.

as titled, in several places are strings or objects

Hi
why has a failed dispatch action the field err in the result payload?

function(err, result) {

}

err = <error>
result = { err: <error> }

if you pass an error and a result the payload looks like

result: { err: <error>, <additional-result-> }

the drivers look like C code wrapped in a JS closure (for instance, a scoped options object implicitly used in a listen function instead of being passed into listen)

if we rewrite the drivers in more typical js style - we can derive boilerplate that can be used for third part drivers to encourage ecosystem growth

Hi @davidmarkclements @mcollina I think we should create a roadmap to work on the right things.
At the moment there are lots of ideas. We should focus on the minimum to make it operational.

I suggest

• Writing down the api interface

• Implement a logging interface

• Clear concept of error handling + implementation

• Implement an interface for payload validation

• Make HTTP/TCP Transports stable.

Bloomrun is far faster (3x - 10x) than Patrun, it also supports index & depth first (which are equally valid).

The only concern I have right now is mu uses * over \*\ for regex. This means we either,

• Change mu to do the conversion
• Change bloomrun to do the conversion

@mcollina thoughts on this?

For now just the function headers and a quick example will do. The API is not clear if it isn't written down.

We need the possibility to validate the incoming payload.

Popular valdiation librarys:

https://github.com/rjrodger/parambulator
https://github.com/hapijs/joi

We could use the same approach as in seneca.

//parambulator
.define(
      {
        a: 1,
        b: {required$: true}
      },
      function (msg, done) {
        done(null, {c: msg.b})
      })
//Joi
.define(
      {
        a: 1,
        b: Joi.required()
      },
      function (msg, done) {
        done(null, {c: msg.b})
      })

currently, a service will log an error (premature close) when a client disconnects from server

handle this gracefully

Mu is an message based router for building distributed systems this include a concept which describes how to dealing with failure scenerios.

Netflix use this Approach I think we can get inspired to implement a simliar strategy for Mu.

• Request queue
• Circuit breaker
• Calculate Circuit Health
• Fallback solutions

Now, we can start the discussion.

References:
https://github.com/Netflix/Hystrix/wiki/Operations
http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html
https://msdn.microsoft.com/en-us/library/dn589784.aspx
https://github.com/Netflix/Hystrix/wiki/

It's possible that we do need some way of injecting errors into patterns (although I'd like to validate that thought, to be sure)
but shouldn't errors be on a meta data object (as agreed), rather than going down the "super private" double underscore route?

different but linked with #56

Unless you know always know exactly where the object came from (e.g. it was created in the same function that's stringifying it (instead of passed in))

use http://npm.im/fast-safe-stringify when you don't absolutely know the object schema

If you know the object schema, use @mcollina's http://npm.im/fast-json-stringify to get super fast serialization

Also there's way too much unnecessary serialisation going on (see #24)

soon we'll be going to monorepo - transports will be published to npm, so mu/drivers/tcp will be mu-tcp, mu/drivers/http would be mu-http

mu-func sounds like ambigious mu-local is clearer

(for these main cases, I would advocate not adding -driver or -transport to the end of each)

@mcollina @pelger

Here it is: https://github.com/mcollina/tinysonic.