Topic: data-pipeline Goto Github

👇 Here are 658 public repositories matching this topic...

adilkhash / data-engineering-howto
data-pipeline,A list of useful resources to learn Data Engineering from scratch
User: adilkhash
distributed-systems data-engineering data-pipeline cloud-providers scala

aeksco / aws-pdf-textract-pipeline
data-pipeline,:mag: Data pipeline for crawling PDFs from the Web and transforming their contents into structured data using AWS textract. Built with AWS CDK + TypeScript
User: aeksco
aws textract lambda dynamodb sns typescript pdf serverless webscraping s3

agnostiqhq / covalent
data-pipeline,Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.
Organization: agnostiqhq
Home Page: https://www.covalent.xyz
hpc workflow workflow-automation workflow-management quantum-computing hpc-applications python machine-learning pipelines covalent

airbytehq / airbyte
data-pipeline,The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
Organization: airbytehq
Home Page: https://airbyte.com
data pipeline data-analysis data-engineering java python etl change-data-capture data-collection data-integration

airscholar / e2e-data-engineering
data-pipeline,An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.
User: airscholar
Home Page: https://www.youtube.com/watch?v=GqAcTrqKcrY
apache-airflow apache-kafka apache-spark apache-zookeeper big-data cassandra containerization data-engineering data-pipeline data-processing data-storage docker etl-pipeline postgresql real-time-analytics

apache / flink-cdc
data-pipeline,Flink CDC is a streaming data integration tool
Organization: apache
Home Page: https://nightlies.apache.org/flink/flink-cdc-docs-stable
change-data-capture cdc batch data-integration data-pipeline distributed elt etl flink kafka

apache / seatunnel-web
data-pipeline,SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).
Organization: apache
Home Page: https://seatunnel.apache.org/
apache data-integration data-pipeline etl-framework high-performance offline real-time seatunnel sql-engine

bruin-data / ingestr
data-pipeline,ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
Organization: bruin-data
Home Page: https://bruin-data.github.io/ingestr/
bigquery copy-database data-ingestion data-integration data-pipeline duckdb ingestion-pipeline mssql postgresql snowflake

bytedance / bitsail
data-pipeline,BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.
Organization: bytedance
Home Page: https://bytedance.github.io/bitsail/
flink big-data data-integration data-lake data-pipeline data-synchronization high-performance real-time

conduitio / conduit
data-pipeline,Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
Organization: conduitio
Home Page: https://conduit.io
go data-integration etl data-pipeline data-engineering data-stream conduit kafka kafkaconnect

cuebook / cuelake
data-pipeline,Use SQL to build ELT pipelines on a data lakehouse.
Organization: cuebook
Home Page: https://cuelake.cuebook.ai
apache-iceberg delta lakehouse datalake data-lake elt etl data-engineering data-integration data-ingestion

damklis / dataengineeringproject
data-pipeline,Example end to end data engineering project.
User: damklis
big-data scraping mongodb elasticsearch data-engineering kafka kafka-connect debezium django-rest-framework redis airflow minio s3 python data-pipeline hacktoberfest

dataflint / spark
data-pipeline,Performance Observability for Apache Spark
Organization: dataflint
apache-spark big-data observability optimization data-pipeline data-pipelines databricks dataproc emr etl

digitalghost-dev / premier-league
data-pipeline,A Data Engineering project. Repository for backend infrastructure and Streamlit app files for a Premier League Dashboard.
User: digitalghost-dev
python go google-cloud streamlit data-visualization bigquery cloud-run docker data-engineer data-pipeline

elementary-data / elementary
data-pipeline,The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
Organization: elementary-data
Home Page: https://www.elementary-data.com/
data-lineage data-governance data-warehouse snowflake bigquery data-analysis data-pipelines data-pipeline lineage data-reliability

feldera / feldera
data-pipeline,The Feldera Incremental Computation Engine
Organization: feldera
Home Page: https://feldera.com
analytics data-analysis data-pipeline database kafka materialized-view rust sql streaming realtime

googlecloudplatform / data-science-on-gcp
data-pipeline,Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
Organization: googlecloudplatform
data-analysis data-visualization cloud-computing machine-learning data-pipeline data-processing data-science data-engineering

indexical-metrics-measure-advisory / watchmen-matryoshka-doll
data-pipeline,Watchmen Platform is a low code data platform for data pipeline, meta data management , analysis, and quality management
Organization: indexical-metrics-measure-advisory
Home Page: https://imma-watchmen.com/
visualization charts data-visualization data-pipeline etl-pipeline data-quality-monitoring pipeline

infoslack / awesome-kafka
data-pipeline,A list about Apache Kafka
User: infoslack
kafka streaming-data data-pipeline stream-processing apache-kafka apache-spark kafka-streams infrastructure data-processing

infuseai / piperider
data-pipeline,Code review for data in dbt
Organization: infuseai
Home Page: https://www.piperider.io/
data-pipeline data-profiling data-quality data-science data-exploration eda exploratory-data-analysis data-testing python data-observability

josephmachado / efficient_data_processing_spark
data-pipeline,Code for "Efficient Data Processing in Spark" Course
User: josephmachado
Home Page: https://josephmachado.podia.com/efficient-data-processing-in-spark
apache-spark data-engineering data-pipeline minio pyspark pyspark-notebook

jvalue / jayvee
data-pipeline,Jayvee is a domain-specific language and runtime for automated processing of data pipelines
Organization: jvalue
Home Page: https://jvalue.github.io/jayvee/
data-engineering data-pipeline data-science domain-specific-language etl-pipeline typescript

kestra-io / kestra
data-pipeline,Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
Organization: kestra-io
Home Page: https://kestra.io
workflow workflow-engine orchestration scheduler data-pipeline elt etl data data-engineering data-orchestration

msamogh / nonechucks
data-pipeline,Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!
User: msamogh
pytorch data-processing data-preprocessing data-pipeline data-cleaning preprocessing machine-learning torch

multiwoven / multiwoven
data-pipeline,🔥🔥🔥 Open Source Alternative to Hightouch, Census, and RudderStack - Reverse ETL & Customer Data Platform (CDP)
Organization: multiwoven
Home Page: https://squared.ai/multiwoven-reverse-etl
data-engineering reverse-etl data-pipeline data-activation etl react ruby self-hosted open-source dbt

olirice / flupy
data-pipeline,Fluent data pipelines for python and your shell
User: olirice
collections data-pipeline fluent python

openbridge / ob_bulkstash
data-pipeline,Bulk Stash is a docker rclone service to sync, or copy, files between different storage services. For example, you can copy files either to or from a remote storage services like Amazon S3 to Google Cloud Storage, or locally from your laptop to a remote storage.
Organization: openbridge
Home Page: https://www.openbridge.com
rclone data-pipeline amazon-web-services google-cloud sftp-synchronisation docker s3 oracle-cloud google-cloud-storage storage-service

patterns-app / patterns-devkit
data-pipeline,Data pipelines from re-usable components
Organization: patterns-app
data-science data-analysis pipelines etl etl-pipeline etl-framework functional-reactive-programming data-engineering sql immutability

pipeline-tools / gusty
data-pipeline,Making DAG construction easier
Organization: pipeline-tools
Home Page: https://pipeline-tools.github.io/gusty-docs/
airflow data-pipeline data-etl

pracdata / awesome-open-source-data-engineering
data-pipeline,A curated list of open source tools used in analytical stacks and data engineering ecosystem
Organization: pracdata
awesome awesome-list data-analytics data-engineering data-platform database self-hosted mlops data-storage data

pydoit / doit
data-pipeline,CLI task management & automation tool
Organization: pydoit
Home Page: http://pydoit.org
python build-tool build-automation task-runner build-system workflow-management data-pipeline workflow workflow-automation data-science hacktoberfest cli

remyxai / vqasynth
data-pipeline,Compose multimodal datasets 🎹
Organization: remyxai
Home Page: https://twitter.com/smellslikeml/status/1756723056675094726
data-pipeline data-processing dataset-generation multimodal-datasets multimodal-deep-learning synthetic-dataset-generation

reugn / go-streams
data-pipeline,A lightweight stream processing library for Go
User: reugn
Home Page: https://pkg.go.dev/github.com/reugn/go-streams
stream-processing pipeline etl kafka data-stream kafka-streams streams redis pulsar data-pipeline

rudderlabs / rudder-server
data-pipeline,Privacy and Security focused Segment-alternative, in Golang and React
Organization: rudderlabs
Home Page: https://www.rudderstack.com/
privacy warehouse-management data-warehouse customer-data customer-data-pipeline customer-data-platform customer-data-lake segment-alternative data-integration data-synchronization

scicloj / scicloj.ml
data-pipeline,A Clojure machine learning library
Organization: scicloj
machine-learning data-science data-pipeline hyperparameter-optimization classification regression scicloj nlp clojure clustering

snowplow / snowplow
data-pipeline,The leader in Next-Generation Customer Data Infrastructure
Organization: snowplow
Home Page: http://snowplowanalytics.com
snowplow analytics data data-pipeline data-collection product-analytics marketing-analytics snowplow-pipeline snowplow-events

sparkfish / augraphy
data-pipeline,Augmentation pipeline for rendering synthetic paper printing, faxing, scanning and copy machine processes
Organization: sparkfish
Home Page: https://github.com/sparkfish/augraphy
data-augmentation crappification deep-neural-networks training-data machine-learning data-pipeline image-processing augmentation-pipeline synthetic-data synthetic-dataset-generation

spotify / klio
data-pipeline,Smarter data pipelines for audio.
Organization: spotify
Home Page: https://docs.klio.io
audio-processing data-pipeline signal-processing media-processing

sspaeti-com / practical-data-engineering
data-pipeline,Practical Data Engineering: A Hands-On Real-Estate Project Guide
Organization: sspaeti-com
Home Page: https://ssp.sh/blog/data-engineering-project-in-twenty-minutes
dagster data-engineering data-pipeline

streamlet-dev / tributary
data-pipeline,Streaming reactive and dataflow graphs in Python
Organization: streamlet-dev
asynchronous data-pipeline kafka lazy-evaluation python python-data-streams python3 reactive-data-streams stream streaming websockets

superlinked / superlinked
data-pipeline,A compute framework for building Search, RAG, Recommendations and Analytics over complex (structured+unstructured) data, with ultra-modal vector embeddings.
Organization: superlinked
Home Page: https://superlinked.com
embeddings etl vector-search data-pipeline deep-learning information-retrieval llm ml mlops natural-language-processing

superstreamlabs / memphis
data-pipeline,Memphis.dev is a highly scalable and effortless data streaming platform
Organization: superstreamlabs
Home Page: https://docs.memphis.dev
data data-stream-processing data-streaming kubernetes messaging-queue data-engineering data-pipeline golang enrichment message-broker

taogeyt / pyetl
data-pipeline,python ETL framework
User: taogeyt
python etl es hive export csv excel mysql sqlserver db

thedataengineeringbook / thedataengineeringbook
data-pipeline,The Data Engineering Book - หนังสือวิศวกรรมข้อมูล ของคนไทย เพื่อคนไทย
Organization: thedataengineeringbook
Home Page: https://thedataengineeringbook.online
data-engineering data hacktoberfest book data-engineer data-pipeline data-integration data-ingestion data-infrastructure

tvdboom / atom
data-pipeline,Automated Tool for Optimized Modelling
User: tvdboom
Home Page: https://tvdboom.github.io/ATOM/
machine-learning data-pipeline modelling data-exploration data-science python visualization automl dagshub interactive-visualizations

ubisoft / mobydq
data-pipeline,:whale: Tool to automate data quality checks on data pipelines
Organization: ubisoft
Home Page: https://ubisoft.github.io/mobydq/
data-quality data-pipeline data-warehouse big-data data-quality-checks data-quality-monitoring

unnati-xyz / scalable-data-science-platform
data-pipeline,Content for architecting a data science platform for products using Luigi, Spark & Flask.
Organization: unnati-xyz
Home Page: http://www.unnati.xyz
data-science data-engineer machine-learning data-pipeline rest-api spark luigi

vincentclaes / datajob
data-pipeline,Build and deploy a serverless data pipeline on AWS with no effort.
User: vincentclaes
Home Page: https://pypi.org/project/datajob/
aws-cdk data-pipeline stepfunctions glue glue-job sagemaker aws pipeline serverless machine-learning

whylabs / whylogs
data-pipeline,An open-source data logging library for machine learning models and data pipelines. 📚 Provides visibility into data quality & model performance over time. 🛡️ Supports privacy-preserving data collection, ensuring safety & robustness. 📈
Organization: whylabs
Home Page: https://whylogs.readthedocs.io/
ai-pipelines approximate-statistics statistical-properties data-quality calculate-statistics python logging mlops dataops ml-pipelines

yc9701 / pansori
data-pipeline,Tools for ASR Corpus Generation from Online Video
User: yc9701
speech-recognition speech-corpus corpus dataset-generation data-pipeline online-video