Topic: etl Goto Github

👇 Here are 3799 public repositories matching this topic...

2ndquadrant / pglogical
etl,Logical Replication extension for PostgreSQL 15, 14, 13, 12, 11, 10, 9.6, 9.5, 9.4 (Postgres), providing much faster replication than Slony, Bucardo or Londiste, as well as cross-version upgrades.
Organization: 2ndquadrant
Home Page: http://2ndquadrant.com/en/resources/pglogical/
postgresql replication logical-decoding database-replication subscription publish-subscribe data-transformation data-transport etl cdc

airbytehq / airbyte
etl,The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
Organization: airbytehq
Home Page: https://airbyte.com
data pipeline data-analysis data-engineering java python etl change-data-capture data-collection data-integration

alexioannides / pyspark-example-project
etl,Implementing best practices for PySpark ETL jobs and applications.
User: alexioannides
pyspark etl-job python data-engineering spark data-science etl etl-pipeline

apache / airflow
etl,Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Organization: apache
Home Page: https://airflow.apache.org/
airflow apache apache-airflow python scheduler workflow automation dag data-engineering data-integration

apache / doris
etl,Apache Doris is an easy-to-use, high performance and unified analytics database.
Organization: apache
Home Page: https://doris.apache.org
olap database hadoop hive hudi iceberg real-time sql bigquery dbt

apache / flink-cdc
etl,Flink CDC is a streaming data integration tool
Organization: apache
Home Page: https://nightlies.apache.org/flink/flink-cdc-docs-stable
change-data-capture cdc batch data-integration data-pipeline distributed elt etl flink kafka

apache / hop
etl,Hop Orchestration Platform
Organization: apache
Home Page: https://hop.apache.org/
java streaming hop apache data-integration etl orchestration pipeline workflow

apache / incubator-devlake
etl,Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
Organization: apache
Home Page: https://devlake.apache.org/
data data-analysis data-engineering data-integration data-transfers devops domain-layer etl golang integration

ariacom / seal-report
etl,Database Reporting Tool and Tasks (.Net)
User: ariacom
Home Page: https://sealreport.org
reporting-engine report-generator business-intelligence task-scheduler html-report dashboards chart linq sqlserver mysql

aws / aws-sdk-pandas
etl,pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
Organization: aws
Home Page: https://aws-sdk-pandas.readthedocs.io
python aws pandas apache-arrow apache-parquet data-engineering etl data-science redshift athena

blockchain-etl / ethereum-etl
etl,Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ
Organization: blockchain-etl
Home Page: https://t.me/BlockchainETL
ethereum csv export erc20 erc20-tokens etl transaction sql aws gcp

cinchoo / choetl
etl,ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)
User: cinchoo
csv parser writer reader flat xml json keyvalue etl cinchoo-etl

cloudquery / cloudquery
etl,The open source high performance ELT framework powered by Apache Arrow
Organization: cloudquery
Home Page: https://cloudquery.io
airbyte attack-surface-management aws azure bigquery cspm data data-analysis data-collection data-engineering data-integration elt etl etl-framework gcp github-api go google kubernetes sql

compose / transporter
etl,Sync data between persistence engines, like ETL only not stodgy
Organization: compose
Home Page: https://github.com/compose/transporter/issues/523
go etl mongodb elasticsearch rethinkdb postgresql rabbitmq mysql

dagster-io / dagster
etl,An orchestration platform for the development, production, and observation of data assets.
Organization: dagster-io
Home Page: https://dagster.io
data-pipelines dagster workflow data-science workflow-automation python scheduler data-orchestrator etl analytics

dagworks-inc / hamilton
etl,Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
Organization: dagworks-inc
Home Page: https://hamilton.dagworks.io/en/latest/
data-science python dag data-engineering dataframe etl etl-framework etl-pipeline feature-engineering featurization

data-engineering-community / data-engineering-wiki
etl,The best place to learn data engineering. Built and maintained by the data engineering community.
Organization: data-engineering-community
Home Page: https://dataengineering.wiki
data database data-engineering data-engineer sql data-modeling data-pipelines etl

dataform-co / dataform
etl,Dataform is a framework for managing SQL based data operations in BigQuery
Organization: dataform-co
Home Page: https://cloud.google.com/dataform/docs
data-pipelines elt data-engineering business-intelligence analytics etl hacktoberfest

datavane / tis
etl,Support agile DataOps Based on Flink, DataX and Flink-CDC, Chunjun with Web-UI
Organization: datavane
Home Page: https://tis.pub
java datax etl flink flink-streaming cdc dataops chunjun

getdozer / dozer
etl,Dozer is a real-time data movement tool that leverages CDC from various sources and moves data into various sinks.
Organization: getdozer
Home Page: https://getdozer.io
apis data rust sql realtime streaming etl low-code postgres snowflake

instill-ai / instill-core
etl,🔮 Instill Core is a full-stack AI infrastructure tool for data, model and pipeline orchestration, designed to streamline every aspect of building versatile AI-first applications
Organization: instill-ai
Home Page: https://www.instill.tech
unstructured-data low-code developer-tools etl no-code open-source hacktoberfest ai api cli

jf-tech / omniparser
etl,omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.
User: jf-tech
transform etl xml json csv fixed-length edi x12 edifact parser

kestra-io / kestra
etl,Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
Organization: kestra-io
Home Page: https://kestra.io
workflow workflow-engine orchestration scheduler data-pipeline elt etl data data-engineering data-orchestration

mage-ai / mage-ai
etl,🧙 Build, run, and manage data pipelines for integrating and transforming data.
Organization: mage-ai
Home Page: https://www.mage.ai/
machine-learning artificial-intelligence data data-engineering data-science python elt etl pipelines data-pipelines

mara / mara-pipelines
etl,A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
Organization: mara
etl data-integration python postgresql pipeline data

multiwoven / multiwoven
etl,🔥🔥🔥 Open Source Alternative to Hightouch, Census, and RudderStack - Reverse ETL & Customer Data Platform (CDP)
Organization: multiwoven
Home Page: https://squared.ai/multiwoven-reverse-etl
data-engineering reverse-etl data-pipeline data-activation etl react ruby self-hosted open-source dbt

nerevu / riko
etl,A Python stream processing engine modeled after Yahoo! Pipes
Organization: nerevu
stream-processing etl parallelism rss data asynchronous library cli functional-programming featured

neumtry / neumai
etl,Neum AI is a best-in-class framework to manage the creation and synchronization of vector embeddings at large scale.
Organization: neumtry
Home Page: https://neum.ai
ai data embeddings etl llm vector-database chatgpt data-engineering database pipeline

nucleuscloud / neosync
etl,Open source data anonymization and synthetic data orchestration for developers. Create high fidelity synthetic data and sync it across your environments.
Organization: nucleuscloud
Home Page: https://www.neosync.dev
docker golang protobuf synthetic-data benthos etl orchestration testing typescript open-source

orchest / orchest
etl,Build data pipelines, the easy way 🛠️
Organization: orchest
Home Page: https://orchest.readthedocs.io/en/stable/
data-science machine-learning pipelines ide jupyter cloud self-hosted jupyterlab notebooks docker

pathwaycom / pathway
etl,Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Organization: pathwaycom
Home Page: https://pathway.com
batch-processing kafka pathway python streaming machine-learning-algorithms real-time data-analytics data-pipelines data-processing

peerdb-io / peerdb
etl,Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
Organization: peerdb-io
Home Page: https://peerdb.io
etl sql bigquery cloud-native distributed-systems kafka postgres postgresql realtime rust

quadratichq / quadratic
etl,Quadratic | Technical Spreadsheet with Python, SQL, and AI
Organization: quadratichq
Home Page: https://QuadraticHQ.com
python data-science spreadsheet sql quadratic etl data-engineering data-analysis data wasm

raystack / optimus
etl,Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
Organization: raystack
Home Page: https://raystack.github.io/optimus
airflow etl workflows automation golang bigquery data-warehouse analytics data-modelling analytics-engineering

react-csv / react-csv
etl,React components to build CSV files on the fly basing on Array/literal object of data
Organization: react-csv
Home Page: http://react-csv.github.io/react-csv/
csv-document reactjs etl excel reporting

redpanda-data / connect
etl,Fancy stream processing made operationally mundane
Organization: redpanda-data
Home Page: https://docs.redpanda.com/redpanda-connect/about/
message-queue stream-processing streaming-data message-bus logs stream-processor cqrs event-sourcing go golang kafka amqp rabbitmq nats etl data-engineering data-ops

reugn / go-streams
etl,A lightweight stream processing library for Go
User: reugn
Home Page: https://pkg.go.dev/github.com/reugn/go-streams
stream-processing pipeline etl kafka data-stream kafka-streams streams redis pulsar data-pipeline

risingwavelabs / risingwave
etl,SQL engine for event-driven workloads. Perform streaming analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch processing. PostgreSQL compatible.
Organization: risingwavelabs
Home Page: https://www.risingwave.com/slack
database stream-processing cloud-native sql distributed-database rust serverless postgresql real-time postgres

rudderlabs / rudder-server
etl,Privacy and Security focused Segment-alternative, in Golang and React
Organization: rudderlabs
Home Page: https://www.rudderstack.com/
privacy warehouse-management data-warehouse customer-data customer-data-pipeline customer-data-platform customer-data-lake segment-alternative data-integration data-synchronization

rwynn / monstache
etl,a go daemon that syncs MongoDB to Elasticsearch in realtime. you know, for search.
User: rwynn
Home Page: https://rwynn.github.io/monstache-site/
elasticsearch go golang mongodb daemon sync oplog realtime river synchronization

singer-io / getting-started
etl,This repository is a getting started guide to Singer.
Organization: singer-io
Home Page: https://singer.io
etl etl-framework python singer data-analysis

stitchfix / hamilton
etl,A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton
Organization: stitchfix
Home Page: https://www.github.com/dagworks-inc/hamilton
python pandas dag data-science data-engineering numpy software-engineering etl-framework etl-pipeline etl

thbar / kiba
etl,Data processing & ETL framework for Ruby
User: thbar
Home Page: https://www.kiba-etl.org
etl etl-ruby ruby data rubydatascience

thenaturalist / awesome-business-intelligence
etl,Actively curated list of awesome BI tools. PRs welcome!
User: thenaturalist
business-intelligence etl data-visualization data-analysis data-science database sql awesome-list

tobikodata / sqlmesh
etl,Efficient data transformation and modeling framework that is backwards compatible with dbt.
Organization: tobikodata
Home Page: https://sqlmesh.com
dataops elt etl sql python dataengineering transformation dbt

toluaina / pgsync
etl,Postgres to Elasticsearch/OpenSearch sync
User: toluaina
Home Page: https://pgsync.com
elasticsearch elasticsearch-sync postgresql change-data-capture sql opensearch python etl kibana

turbot / steampipe
etl,Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.
Organization: turbot
Home Page: https://steampipe.io
steampipe sql postgresql postgresql-fdw cloud security aws azure cis cnapp

wgzhao / addax
etl,Addax is a versatile open-source ETL tool that can seamlessly transfer data between various RDBMS and NoSQL databases, making it an ideal solution for data migration.
User: wgzhao
Home Page: https://wgzhao.github.io/Addax/
hadoop hive database data-integrity clickhouse influxdb kudu mysql sqlserver trino

xyflow / awesome-node-based-uis
etl,A curated list with resources about node-based UIs
Organization: xyflow
node-based-ui awesome-list etl visual-programming workflow-editor

zinggai / zingg
etl,Scalable identity resolution, entity resolution, data mastering and deduplication using ML
Organization: zinggai
fuzzymatch fuzzy-matching deduplication dedupe masterdata dataengineering data-transformation analytics-engineering entity-resolution identity-resolution