eeiver55555 / pipedream Goto Github PK

Pipedream is a serverless integration and compute platform.

We provide a free, hosted platform that makes it easy to connect apps and develop, execute and maintain event-driven workflows. The platform has over 300 fully integrated applications with managed authentication and support for over 1M npm packages.

Key Features:

• Event Sources - Open source components that emit events from services (Github, Slack, Airtable, RSS & more)
• Workflows - A sequence of linear steps - just Node.js code - triggered by an event (via event source, HTTP or timer)
• Actions - Pre-built code steps that you can use in a workflow to perform common operations across Pipedream's 300+ API integrations, for example: sending email, adding a row to a Google Sheet, and more.
• Destinations - Deliver events asynchronously to common destinations like Amazon S3, Snowflake, HTTP and email
• Serverless - No server or cloud resources to manage
• Free - No fees for individual developers (see limits)

Product Demo: YouTube (5 minutes)

You can also get support, raise a bug or feature request, or file a security disclosure.

Pipedream receives data via event sources. Event sources are open source, run on Pipedream's infrastructure and collect data from your own application and/or services like Github, DropBox, Zoom, RSS feeds, and more.

Event sources emit new events produced by the service, which can trigger Pipedream workflows, or which you can consume using Pipedream's REST API or a private, real-time SSE stream.

Here is the simplest event source possible, an HTTP event source:

module.exports = {
  name: "http",
  version: "0.0.1",
  props: {
    http: "$.interface.http"
  },
  run(event) {
    console.log(event); // event contains the method, payload, etc.
  }
};

Popular Event Sources:

• Airtable (deploy)
• AWS (deploy)
• Dropbox (deploy)
• Github (deploy)
• Google Calendar (deploy)
• Google Drive (deploy)
• RSS (deploy)
• Twitter (deploy)

Event sources can also be deployed via the Pipedream CLI:

curl https://cli.pipedream.com/install | sh

Once installed, you can deploy an event source by running:

pd deploy   # prompts you to select a component and pass required options

You can also create your own event sources for your own personal use. If you think others would benefit from your source, you can publish them to all Pipedream users by opening a PR in this repo. See these docs to get started:

• Component API
• HTTP Event Sources Quickstart
• Timer-based Event Sources Quickstart

Workflows are a sequence of linear steps - just Node.js code - triggered by an event (via event source, HTTP endpoint, or timer). Workflows make it easy to transform data and integrate with 300+ APIs from various apps and services.

• Trigger your workflow on any event (e.g. HTTP requests or a schedule).
• Add steps to run Node.js code (using virtually any npm package) and pre-built actions.
• Steps are executed in the order they appear in your workflow.
• Data is shared between steps via step exports.

Workflow code is public by default so the community can discover and copy them. Your workflow execution and event data is private.

You can copy this example workflow to get started, or review some community-developed workflows to see what others are building.

As you build more advanced workflows, you may also find these docs helpful:

• What are events? - events trigger workflow executions
• What are steps? - building blocks you use to create workflows
• Managing workflow state - how to store state in one execution of a workflow that you can read in subsequent executions
• Passing data to steps - steps are just Node functions, and can accept input via step parameters.
• Connected Accounts - how to authenticate to APIs within code steps.
• Error Handling - how to use the Global Error workflow to manage errors raised by workflows.

Actions are pre-built code steps that you can use to perform common operations across Pipedream's 300+ API integrations, for example: sending email, adding a row to a Google Sheet, and more. Pipedream currently supports over 1000+ actions.

Typically, integrating with these services requires a custom code to manage authentication, error handling, etc. Actions abstract that for you - you just pass the necessary params as input and the action handles the rest. For example, the Send HTTP Request action accepts the data you want to send and the URL you want to send it to, returning the HTTP response for use in future steps.

Actions come pre-built to solve a common use case, but you can modify them in any way you'd like. Actions are just Node.js functions. When you add an action, you'll see its code in your workflow - just click into the code and start editing to modify it.

Finally, you can create your own actions, allowing you to re-use them across workflows in your account. You can even publish actions to the entire Pipedream community, making them available for anyone to use.

Here's the code for the Send HTTP Request action:

async params => {
  const config = {
    method: params.method ||  "post",
    url: params.url,
  }
  for (const { key, value } of params.query || []) {
    if (!config.params) config.params = {}
    config.params[key] = value
  }
  for (const { key, value } of params.headers || []) {
    if (!config.headers) config.headers = {}
    config.headers[key] = value
  }
  if (params.auth) {
    config.auth = {
      username: params.auth.username,
      password: params.auth.password,
    }
  }
  if (params.responseType) {
    config.responseType = params.responseType
  }
  if (params.payload) config.data = params.payload
  return await require("@pipedreamhq/platform").axios(this, config)
}  

Destinations, like Actions, abstract the connection, batching, and delivery logic required to send events to services like Amazon S3 and Snowflake, or targets like HTTP and email.

For example, sending data to an Amazon S3 bucket is as simple as calling $send.s3():

$send.s3({
  bucket: "your-bucket-here",
  prefix: "your-prefix/",
  payload: event.body
});

Pipedream supports the following destinations today:

• Amazon S3
• Snowflake
• HTTP
• Email
• Pipedream SQL Service
• SSE

Pipedream is currently free, subject to the limits noted below. Paid tiers for higher volumes are coming soon.

If you exceed any of these limits, please reach out.

The Pipedream platform imposes some runtime limits on sources and workflows. Read more about those in our docs.

You can get help on our public Slack or reach out to our team directly with any questions or feedback. We'd love to hear from you!

Before adding an issue, please search the existing issues or reach out to our team to see if a similar request already exists.

If an issue exists, please add a reaction or comment on your specific use case.

If an issue doesn't yet exist, please use these templates to create one:

New feature

Report a bug

New trigger / source

New API integration

New action

If you'd like to report a suspected vulnerability or security issue, or have any questions about the security of the product, please contact our security team at [email protected].