Git Product home page Git Product logo

multiwoven / multiwoven Goto Github PK

View Code? Open in Web Editor NEW
623.0 12.0 28.0 4.29 MB

🔥🔥🔥 Open Source Hightouch, Census, and RudderStack Alternative

Home Page: https://multiwoven.com

License: GNU Affero General Public License v3.0

Ruby 62.19% Shell 0.03% Dockerfile 0.44% HTML 0.21% JavaScript 0.42% TypeScript 36.08% SCSS 0.63%
data-engineering reverse-etl data-pipeline data-activation etl react ruby self-hosted open-source dbt

multiwoven's Introduction

Multiwoven

Open Source Reverse ETL & Customer Data Platform (CDP)

Multiwoven is an open-source alternative to HighTouch, Census, and RudderStack. With Multiwoven, you can easily sync data from your data warehouse to any business tool, turning your data warehouse into a Customer Data Platform (CDP)

GitHub stars GitHub release (latest) GitHub commit activity License
server-ci integrations-ci ui-ci


Explore the docs »

SlackWebsiteBlogRoadmap


Why Multiwoven?

Multiwoven makes it easy to self-host and deploy a fully secure, scalable, and customizable Reverse ETL platform on your own cloud infrastructure like AWS, Azure, GCP and more. With one click deployment you can instantly start syncing data from your data warehouse to business tools. Multiwoven also makes it easy to extend the platform with new connectors to support a wide range of data sources and destinations.

Example Image

Example Image

Example Image

Consider giving us a star! Your support helps us continue innovating and adding new, exciting features.

Table of Contents

Getting Started

Multiwoven is a monorepo that consists of three main services:

  • server - The backend service that acts as a control plane for managing data sources, models, and syncs.

  • ui - The frontend react application that provides a user interface to manage data sources, destinations, and confgure syncs.

  • integrations - A Ruby Gem that provides a framework to build connectors to support a wide range of data sources and destinations.

Local Setup

To get started with Multiwoven, you can deploy the entire stack using Docker Compose.

  1. Clone the repository:
git clone [email protected]:Multiwoven/multiwoven.git
  1. Go inside multiwoven folder:
cd multiwoven
  1. Initialize .env file:
mv .env.example .env
  1. Start the services:
docker-compose build && docker-compose up

UI can be accessed at the PORT 8000 :

http://localhost:8000

For more details, check out the local deployment guide in the documentation.

Self-hosted Options

Multiwoven can be deployed in a variety of environments, from fully managed cloud services to self-hosted solutions. Refer to the deployment guides below to deploy Multiwoven on your preferred cloud provider.

Provider Documentation
Docker Deployment Guide
AWS EC2 Deployment Guide
AWS ECS Deployment Guide
AWS EKS (Kubernetes) Deployment Guide
Azure VMs Deployment Guide
Azure AKS (Kubernetes) Deployment Guide
Google Cloud GKE (Kubernetes) Deployment Guide
Google Cloud Compute Engine Deployment Guide
Digital Ocean Droplets Deployment Guide
Digital Ocean Kubernetes Deployment Guide
OpenShift Deployment Guide
Helm Charts Deployment Guide

Connectors

🔥 Multiwoven is rapidly expanding its list of connectors to support a wide range of data sources and destinations. Head over to the Integrations directory to explore the available connectors. If you don't see the connector you need, please open an issue to request it.

Sources

Destinations

CRM

Marketing Automation

  • Klaviyo
  • Braze
  • Salesforce Marketing Cloud
  • Klaviyo

Customer Support

  • Zendesk
  • Freshdesk
  • Intercom

Advertising

Collaboration

Analytics

  • Google Analytics
  • Mixpanel
  • Amplitude

Others

🧵...Weaving in more connectors to support a wide range of destinations.

Contributing

We ❤️ contributions and feedback! Help make Multiwoven better for everyone!

Before contributing to Multiwoven, please read our Code of Conduct and Contributing Guidelines. As a contributor, you are expected to adhere to these guidelines and follow the best practices.

Need Help?

We are always here to help you. If you have any questions or need help with Multiwoven, please feel free to reach out to us on Slack. We are open to discuss new ideas, features, and improvements.

⚠️ Development Status: Under Active Development

This project is under active development, As we work towards stabilizing the project, you might encounter some bugs or incomplete features. We greatly value your contributions and patience during this phase. Thank you for your support!

License

Multiwoven is licensed under the AGPLv3 License. See the LICENSE file for details.

Contributors

multiwoven's People

Contributors

afthabvp avatar akhilgkrishnan avatar amitabh-blade avatar blade365z avatar karthik-sivadas avatar lallenfrancisl avatar macintushar avatar nagstler avatar sagarbajpai avatar subintp avatar sujoygolan avatar sumitd94 avatar vatan588 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

multiwoven's Issues

Enable/disable sync from UI

**Is your feature request related to a problem? **
I would like to disable sync from source to destination for some time when especially when there schema changes going on the source/destination

Describe the solution you'd like
Frontend should give an option to disable or enable a sync

Test bug issue

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Smartphone (please complete the following information):

  • Device: [e.g. iPhone6]
  • OS: [e.g. iOS8.1]
  • Browser [e.g. stock browser, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

Show skipped rows count for sync runs

**Is your feature request related to a problem?
We are showing queried, success and failed count in the sync run listing. In incremental refresh we skip the rows based on the primary key and creates confusion for user eg:total queried is 1000, success =2, failed = 0. The question is where the other 998 records are. why it is not showing in the sync run list metrics.

Describe the solution you'd like
We need to show a new metrics called skipped which states how many records in the sync due to duplicate primary of the model.

Good to have
We need to show skipped rowns count in the reporting dashboard.

[BUG]: Add validation for primary_key in model ActiveRecord

validation for the primary key in the ActiveRecord model involves checking if the primary key exists for the source connector

model create payload

{
  "model": {
    "connector_id": 25,       // Replace with the appropriate connector_id
    "name": "users",          // Replace with the desired model name
    "query": "SELECT Id,User.Username, RecordType.Name, Account.AccountNumber, Account.OnboardedAccountNumber__c, PlannedVisitStartTime, PlannedVisitEndTime FROM Visit",  // Replace with the desired query
    "query_type": "soql",     // Replace with the desired query type
    "primary_key": "Id"
  }
}

Custom Destination Connector

Hi !

Congratulation for this project, it will be awesome!

I just want to open this feature request to allow developers building custom destination connectors.
Feel free to choose the best framework for this feature ;)

Regards

Setup release automation github action

Is your feature request related to a problem? Please describe.
Releasing an update for our repo requires manual intervention to release with a semver version update. It also takes some time to create a change log. Because these are manually done we have a discrepancy in our application release. Also when we update the docker images with the tag, the tags created are commit sha which should ideally be release versions. All these issues will be solved by automating the release process.

Describe the solution you'd like
We will cut a weekly release with a minor version. We will also set up a workflow which can be used to cut patches or major releases by manually initiating the GitHub action. This GitHub action will create a tag and push it to the repo. This tag will push a docker image to the docker hub with the same tag. We will also create PR with release notes using Git Cliff which will create a release not on merge.

Mysql connector in source

Is your feature request related to a problem? Please describe.
Add MySQL connecter or Give the option to add a configurable connector on user choice

In open source there is not option to create own connector thourgh UI please provide atleat mysql connector because most of the people use mysql as it there source .
Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Flush/Clean Sync Records for Sync Run in FullRefresh Sync Mode

In FullRefresh sync mode, for every sync run, we fetch all records from the source and send them to the destination. Since there's no need to preserve the sync records once the data is moved to the destination, we need to flush or clean the sync records after each completed run.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.