Git Product home page Git Product logo

reshuffle's Introduction

Reshuffle Integration Framework

Reshuffle is a lightweight and open source integration framework. With Reshuffle, you can build integrations, workflows, and connect systems.

Here is a simple workflow that sends a Text message when an urgent Email arrives:

const { Reshuffle } = require('reshuffle')
const { IMAPConnector } = require('reshuffle-imap-connector')
const { TwilioConnector } = require('reshuffle-twilio-connector')

const app = new Reshuffle()
const imap = new IMAPConnector(app, emailServerOptions)
const twilioConnector = new TwilioConnector(app, twilioOptions)

imap.on({ mailbox: 'INBOX' }, (event, app) => {
  if (event.mail.headers.get('subject').startsWith('URGENT ALERT:')) {
    twilioConnector.sendSMS('We got urgent alert', '+16502224533')
  }
})

app.start()

Installation

$ npm install reshuffle

For a full step by step tutorial on how to install and use Reshuffle please visit this page

Reshuffle Core Features

  • Simple Express-style event handling model
  • Separation of configuration and execution
  • Connectors to many SaaS services
  • Highly extensible
  • Focus on high performance

Basic concepts

Event Based System

At its core Reshuffle is an event-based engine, very similar to programming with a web server. You only need to define a function that will be called when an event is triggered.

Events can be anything from a file change, S3 bucket update, a cron job timer, your own custom event, or even an HTTP call.

Here is an example of listening to an HTTP GET event on /test, using the HTTP connector:

const { Reshuffle, HttpConnector } = require('reshuffle')
const app = new Reshuffle()
const httpConnector = new HttpConnector(app)

httpConnector.on({ method: 'GET', path: '/test' }, (event, app) => {
  event.res.end('Hello World!')
})

Connectors act like node eventEmitter, so when the event is emitted the function is called. We will discuss Connectors in the next section.

More examples are found here

Reshuffle Connectors

A critical aspect of building integrations is configuring how to connect to different services we need to integrate with. With Reshuffle, you can configure Connector objects and inject them into your code.

Let's expand the example above and send a message to a Slack ‘reports’ channel every time someone triggers the 'HTTP/GET/test' event:

const { Reshuffle, HttpConnector, SlackConnector } = require('reshuffle')
const app = new Reshuffle()

// the httpConnector does not require any config
const httpConnector = new HttpConnector(app)

// Slack connection configuration
const slackConnectionOptions = {
  APIToken: process.env.SLACK_AUTH_KEY,
  team: 'ourTeam',
}
// the 3rd parameter is used to identify the connector later on
new SlackConnector(app, slackConnectionOptions, 'connectors/Slack')

httpConnector.on({ method: 'GET', path: '/test' }, (event, app) => {
  app.getConnector('connectors/Slack').send('Somebody triggered this event!', '#reports')
})

app.start()

Connector objects expose the API and Events that the external connector (connecting to anything from a DB to an ERP) provides. You can specify a connector id by providing an identifier in the connector constructor, and then access that connector using the app.getConnector(connectorId) method.

You noticed in the code sample above important information on how to connect to the 3rd party system (Slack in this case). Connectors are a way to separate the connection configuration from your code, configure a connection to a connector once and use it anywhere.

You can use the Connector object to take action on a remote service (such as adding a row to a CRM) and configure events that trigger when something happens in that system. We will show you how to do that in the next section.

You can read more about the connector class here

A full list of Connectors, and how to create your own Connector can be found here

Events

As we saw, connectors are basically adapters that connect external systems, such as Slack, Database, CRM, or any other system. Connectors can be configured to emit a Reshuffle event, when a preconfigured thing happens in these systems. To configure an event, use the on(eventOptions, handler) method of the relevant connector.

Here is how you would configure a SlackConnector to listen to a message from Slack:

const { Reshuffle, SlackConnector } = require('reshuffle')
const app = new Reshuffle()

const connectionOptions = {
  APIToken: process.env.SLACK_AUTH_KEY,
  team: 'ourTeam',
}

const slackConnector = new SlackConnector(app, connectionOptions, 'connectors/Slack')

const eventOptions = {
  event_type: 'new_message',
  channel: 'C6646754636',
  type: 'new_message',
}

slackConnector.on(eventOptions, (event, app) => {
  app.getConnector('connectors/Slack').reply('Thank you for your message!')
})

app.start()

It is the responsibility of the SlackConnector to listen to the messages in Slack and emit corresponding events in Reshuffle. Your code can listen to these events and run business logic.

As you can see, both the event creation and the business logic, use the same Connector configuration. This makes configuration easier to manage.

You can read more about the Event class here

Custom Events

Sometimes you just want to emit an event in your script without any relation to an external system, for this purpose you can use the Custom Event Connector. You can read more about the Custom Event Connector here

Health check

Reshuffle can expose a health check path. This can be used to let automated tools check the availability of the system. To make use of this functionality, set the RESHUFFLE_HEALTH_PATH environment variable with your desired health check path (e.g. /reshuffle-healthcheck). Note that there is not default path for this capability, and therefore no automatic health check will be provisioned if the environment variable isn't configured.

The health check call returns a 200 Response with:

{
  ok: boolean
  uptime: Number //sec
}

Examples

Examples can be found in the /examples folder

Connectors in this npm package

reshuffle's People

Contributors

antonmarsden avatar arielshaqed avatar ashevat avatar bergundy avatar dependabot[bot] avatar foundingnimo avatar refarer avatar tomv avatar vogre avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

reshuffle's Issues

Cross platform development is not supported

Currently our projects can be built on linux

They usually can be built in osx but we don't make it a priority (ie. using cp -r instead of cp -R)

They usually can never be built in windows (things like cp and rm -rf doesn't work there)

We should make an effort to support all platforms, this includes:

  1. Changing all rm -rf to rimraf npm package.
  2. Use cross-env to set environment variables in npm scripts
  3. Find cross platform alternative to cp
  4. Probably other stuff, maybe jenkins can try to build on windows?

Difficult to generate docs reliably

We chose to use jsdoc style for the initial release. Unfortunately, I've had issues generating jsdocs from typescript code and from the generated JavaScript. Instead, I tried using Typedoc, and while this at least allows me to generate documentation, the documentation is not good for our end users.

For example, if I want to generate docs for the db package, I need to exclude a million specific things (such as the constructor because the end user never "constructs" a db instance). I've attempted to do this manually, but I believe it's a flawed approach for two main reasons:

  1. My process breaks whenever a change is made to the API
  2. It's easier for engineering to understand what is/isn't important as opposed to an external judge

We need a method that allows engineering to concisely expose the API (and docs) that will be directly relevant for the end user.

Add Host and Origin protections

The local-proxy https://github.com/binaris/shiftjs/blob/ebd09a38da23013216dab07e86faea98c4b56222/local-proxy/src/index.ts#L87 does not have any check from request source, this way it's possible for websites to embed code that attempts to invoke code on a developer's machine.

create-react-app enables the webpack-dev-server protections only in case proxy feature is enabled. When the setupProxy feature is used the feature is not enabled. Additional checks should be implemented instead (similar to https://github.com/webpack/webpack-dev-server/blob/30fe4718a044f88a0fb7315403bddb57cb190a04/lib/Server.js#L834).

Add babel-macro test cases

Asked by @michaeladda in #41

  • non exposed functions
  • exposed function calling another exposed function
  • exposed function calling non exposed function
  • non exposed function calling exposed function
  • expose with all kinds of bad exports
  • exports of not exposed functions (and classes, and primitives)
  • exposing nested function by import (a file which itself imports another)
  • importing exposed code without the babel macro

Add install script

[ ] create backend directory with backend template
[ ] create setupProxy.js
[ ] npm install relevant packages (macro, proxy, runtime, db, ...?)
[ ] add shift stuff to .gitignore

Imported backend code does not trigger reload in some cases

This is difficult because Amir and I tried our best to consistently reproduce, but were not able to. Amir was developing a very basic MV(-1) app, and was importing backend code using the expected destructuring syntax.

import { getBasic, getOtherBasic, getFinal } from '../backend/backend.js';

When we could produce the bug, it appeared that adding a new key to destructure, was not enough to trigger a backend update.

- import { getBasic, getOtherBasic, getFinal } from '../backend/backend.js';
+ import { newAddition, getBasic, getOtherBasic, getFinal } from '../backend/backend.js';

I suspect that he used the imported function before importing it, saved the file. This clearly did not produce a valid build because he was relying on an unimported function. Afterwards, he did import it, but still received the same error.

Could not invoke /newAddition

Removing the specific imported function immediately removed the issue and the app worked correctly (including other backend functions). After messing around with it, we could not reproduce the generic bug.

Re-transpile of front-end code is not triggered by changes to backend code

For example after fixing an error (like a forgotten @expose) and saving the file the re-transpile isn't triggered until saving a front-end file.

A hacky workaround is the dev-server detecting this situation and triggering a touch src/index.js (expected to always exist as the entry-point), but not good for the long run.

@expose leading comment seems risky

consider this code:

// @expose
export function foo() {
}

// @expose
export function bar() {
}

export function internalFunction() {
}

turning to this cause function was commented out:

// @expose
export function foo() {
}

// @expose
// export function bar() {
// }
//
export function internalFunction() {
}

suddenly internalFunction is exposed!

Documentation should be more verbose

I've generated some documentation using Typedoc (specifically for the db). After I generated the docs, I immediately noticed that there was nearly 0 description for the individual methods of the DB. I don't think this contradicts the spec, but the methods already have less information than the spec does. Can we please add more verbose documentation to each of DB methods such that it's informative and insightful for our users.

Proper package publication (please)

Following #46, consider the following in each sub-package:

  • author and description fields in package.json.
  • README.md (you know, for NpmJS!)
  • Distribute only tsc outputs, no *.ts.

None of these is a current blocker.

Babel macro creates wrong output when importing from multiple files

expecting

import "../../macro";
import { foo, bar } from '../backend/mockBackend';
import { foo2, bar2 } from '../backend/mockBackend2';

to transpile to:

import { createRuntime } from "@binaris/shift-fetch-runtime";
const {
  foo,
  bar
} = createRuntime(["foo", "bar"], {
  "filename": "mockBackend.js"
});
const {
  foo2,
  bar2
} = createRuntime(["foo2", "bar2"], {
  "filename": "mockBackend2.js"
});

but actual result is:

import { createRuntime } from "@binaris/shift-fetch-runtime";
import { createRuntime } from "@binaris/shift-fetch-runtime";
const {
  foo2,
  bar2
} = createRuntime(["foo2", "bar2"], {
  "filename": "mockBackend2.js"
});
import { foo2, bar2 } from '../backend/mockBackend2';

multiple shift_local_proxy folders on windows

A new folder is created on every npm start, without removing the old one.

d----- 8/7/2019 3:03 PM .shift_local_proxy_XPMhye
d----- 8/2/2019 12:59 PM .shift_local_proxy_xPoKzI
d----- 8/6/2019 10:17 PM .shift_local_proxy_xtBJsu
d----- 8/2/2019 3:47 PM .shift_local_proxy_Xx00XP
d----- 8/2/2019 12:54 PM .shift_local_proxy_XZw81G
d----- 8/7/2019 12:58 AM .shift_local_proxy_yd6433
d----- 8/2/2019 3:43 PM .shift_local_proxy_yEph4q
d----- 8/6/2019 5:43 PM .shift_local_proxy_yFEtKG
d----- 8/2/2019 3:02 PM .shift_local_proxy_yIx3Yq
d----- 8/2/2019 12:55 PM .shift_local_proxy_YKooti
d----- 8/2/2019 11:26 PM .shift_local_proxy_yOFNbK
d----- 8/6/2019 11:38 PM .shift_local_proxy_YOwEnQ
d----- 8/2/2019 12:50 PM .shift_local_proxy_ypn612
d----- 8/7/2019 12:44 AM .shift_local_proxy_Yq9k0r
d----- 8/7/2019 12:15 AM .shift_local_proxy_YRYbIf
d----- 8/7/2019 12:48 AM .shift_local_proxy_YT9Ef2
d----- 8/1/2019 10:09 AM .shift_local_proxy_Z9Gll6
d----- 8/3/2019 12:19 AM .shift_local_proxy_ZaPdvo
d----- 8/7/2019 12:49 AM .shift_local_proxy_ZBddS2
d----- 8/7/2019 12:27 AM .shift_local_proxy_zEY9N7
d----- 8/6/2019 11:41 PM .shift_local_proxy_zFVzz1
d----- 8/2/2019 12:59 PM .shift_local_proxy_ZG6fwl
d----- 8/2/2019 11:09 PM .shift_local_proxy_ZJ9bXJ
d----- 8/1/2019 11:02 PM .shift_local_proxy_Zmrr1I
d----- 8/2/2019 12:50 PM .shift_local_proxy_Zn10c4
d----- 8/7/2019 12:50 AM .shift_local_proxy_zo8RVm
d----- 8/6/2019 11:44 PM .shift_local_proxy_ZpoksV
d----- 8/2/2019 11:11 PM .shift_local_proxy_zpyRMx
d----- 8/7/2019 12:49 AM .shift_local_proxy_Zuyjfs
d----- 8/7/2019 9:36 PM .shift_local_proxy_zXyGuh
d----- 8/5/2019 9:24 AM .shift_local_proxy_zzOu90

local server should catch unhandled rejections

It's not enough to

try {
  // ...
  await userFn();
} catch (err) {
  // Report err to console.error and return 5xx
}

because the user code can leak exceptions.

We should use project.unhandledRejection and report those nicely with console.error, too: these errors are most confusing to the users.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.