Topic: data-engineering Goto Github

👇 Here are 2958 public repositories matching this topic...

adilkhash / data-engineering-howto
data-engineering,A list of useful resources to learn Data Engineering from scratch
User: adilkhash
distributed-systems data-engineering data-pipeline cloud-providers scala

airbytehq / airbyte
data-engineering,The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
Organization: airbytehq
Home Page: https://airbyte.com
data pipeline data-analysis data-engineering java python etl change-data-capture data-collection data-integration

alexioannides / pyspark-example-project
data-engineering,Implementing best practices for PySpark ETL jobs and applications.
User: alexioannides
pyspark etl-job python data-engineering spark data-science etl etl-pipeline

andkret / cookbook
data-engineering,The Data Engineering Cookbook
User: andkret
Home Page: https://learndataengineering.com/
data-engineer data-engineering big-data best-practices cookbook

apache / airflow
data-engineering,Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Organization: apache
Home Page: https://airflow.apache.org/
airflow apache apache-airflow python scheduler workflow automation dag data-engineering data-integration

apache / incubator-devlake
data-engineering,Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
Organization: apache
Home Page: https://devlake.apache.org/
data data-analysis data-engineering data-integration data-transfers devops domain-layer etl golang integration

apache / superset
data-engineering,Apache Superset is a Data Visualization and Data Exploration Platform
Organization: apache
Home Page: https://superset.apache.org/
superset apache apache-superset data-visualization data-viz analytics business-intelligence data-science data-engineering asf

argoproj / argo-workflows
data-engineering,Workflow Engine for Kubernetes
Organization: argoproj
Home Page: https://argo-workflows.readthedocs.io/
workflow kubernetes argo dag knative airflow machine-learning argo-workflows workflow-engine hacktoberfest

avaiga / taipy
data-engineering,Turns Data and AI algorithms into production-ready web applications in no time.
Organization: avaiga
Home Page: https://www.taipy.io
automation data-engineering data-ops data-visualization datascience developer-tools mlops orchestration pipeline pipelines

aws / aws-sdk-pandas
data-engineering,pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
Organization: aws
Home Page: https://aws-sdk-pandas.readthedocs.io
python aws pandas apache-arrow apache-parquet data-engineering etl data-science redshift athena

benthosdev / benthos
data-engineering,Fancy stream processing made operationally mundane
Organization: benthosdev
Home Page: https://www.benthos.dev
message-queue stream-processing streaming-data message-bus logs stream-processor cqrs event-sourcing go golang

cloudquery / cloudquery
data-engineering,The open source high performance ELT framework powered by Apache Arrow
Organization: cloudquery
Home Page: https://cloudquery.io
aws gcp azure sql data-integration elt etl etl-framework bigquery data-collection

dagster-io / dagster
data-engineering,An orchestration platform for the development, production, and observation of data assets.
Organization: dagster-io
Home Page: https://dagster.io
data-pipelines dagster workflow data-science workflow-automation python scheduler data-orchestrator etl analytics

dagworks-inc / hamilton
data-engineering,Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage and metadata. Runs and scales everywhere python does.
Organization: dagworks-inc
Home Page: https://hamilton.dagworks.io/en/latest/
data-science python dag data-engineering dataframe etl etl-framework etl-pipeline feature-engineering featurization

datafold / data-diff
data-engineering,Compare tables within or across databases
Organization: datafold
Home Page: https://docs.datafold.com
database mysql postgresql snowflake rdbms trino data-engineering data-quality data-science data-quality-monitoring

datastacktv / data-engineer-roadmap
data-engineering,Roadmap to becoming a data engineer in 2021
Organization: datastacktv
Home Page: https://datastack.tv
data-engineer-roadmap data-engineering cloud roadmap

datatalksclub / data-engineering-zoomcamp
data-engineering,Free Data Engineering course!
Organization: datatalksclub
data-engineering kafka spark dbt docker prefect

dlt-hub / dlt
data-engineering,data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
Organization: dlt-hub
Home Page: https://dlthub.com/docs
data python data-engineering data-lake data-loading data-warehouse elt extract load transform

eugeneyan / applied-ml
data-engineering,📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.
User: eugeneyan
applied-machine-learning production applied-data-science machine-learning data-science reinforcement-learning data-engineering recsys search deep-learning

eventual-inc / daft
data-engineering,Distributed DataFrame for Python designed for the cloud, powered by Rust
Organization: eventual-inc
Home Page: https://getdaft.io
machine-learning python data-engineering data-science dataframe distributed-computing rust big-data

evidence-dev / evidence
data-engineering,Business intelligence as code: build fast, interactive data visualizations in pure SQL and markdown
Organization: evidence-dev
Home Page: https://evidence.dev
analytics sql business-intelligence data-visualization dbt duckdb tailwindcss self-hosted open-source svelte

feast-dev / feast
data-engineering,Feature Store for Machine Learning
Organization: feast-dev
Home Page: https://feast.dev
machine-learning features ml big-data feature-store python mlops data-engineering data-science data-quality

feathr-ai / feathr
data-engineering,Feathr – A scalable, unified data and AI engineering platform for enterprise
Organization: feathr-ai
Home Page: https://join.slack.com/t/feathrai/shared_invite/zt-1ffva5u6v-voq0Us7bbKAw873cEzHOSg
feature-engineering feature-store artificial-intelligence mlops data-engineering data-quality machine-learning apache-spark azure data-science

gokumohandas / made-with-ml
data-engineering,Learn how to design, develop, deploy and iterate on production-grade ML applications.
User: gokumohandas
Home Page: https://madewithml.com
machine-learning deep-learning pytorch natural-language-processing data-science python mlops data-engineering data-quality distributed-ml

gokumohandas / mlops-course
data-engineering,Learn how to design, develop, deploy and iterate on production-grade ML applications.
User: gokumohandas
Home Page: https://madewithml.com
machine-learning deep-learning pytorch mlops data-engineering data-quality data-science distributed-ml llms natural-language-processing

googlecloudplatform / data-science-on-gcp
data-engineering,Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
Organization: googlecloudplatform
data-analysis data-visualization cloud-computing machine-learning data-pipeline data-processing data-science data-engineering

great-expectations / great_expectations
data-engineering,Always know what to expect from your data.
Organization: great-expectations
Home Page: https://docs.greatexpectations.io/
pipeline-tests dataquality datacleaning datacleaner data-science data-profiling pipeline pipeline-testing cleandata dataunittest data-unit-tests eda exploratory-data-analysis exploratory-analysis exploratorydataanalysis data-quality data-engineering pipeline-debt data-profilers mlops

growthbook / growthbook
data-engineering,Open Source Feature Flagging and A/B Testing Platform
Organization: growthbook
Home Page: https://www.growthbook.io
abtesting statistics abtest experimentation split-testing mixpanel snowflake bigquery redshift clickhouse

gunnarmorling / awesome-opensource-data-engineering
data-engineering,An Awesome List of Open-Source Data Engineering Projects
User: gunnarmorling
awesome-list data-engineering

hemansnation / god-level-data-science-ml-full-stack
data-engineering,A collection of scientific methods, processes, algorithms, and systems to build stories & models. Whether you are a fresher in the field or an experienced professional who wants to transition into Data Science & AI
User: hemansnation
Home Page: https://www.himanshuramchandani.co/
python datastructures numpy pandas matplotlib scikit-learn machine-learning deep-learning pytorch data-science

jqnatividad / qsv
data-engineering,CSVs sliced, diced & analyzed.
User: jqnatividad
csv data-wrangling tsv cli opendata data-engineering ckan excel luau datapackage

kantord / just-dashboard
data-engineering,:bar_chart: :clipboard: Dashboards using YAML or JSON files
User: kantord
Home Page: https://kantord.github.io/just-dashboard/
data-science data-visualization dashboard data-engineering d3 d3js chart data yaml csv

kestra-io / kestra
data-engineering,Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
Organization: kestra-io
Home Page: https://kestra.io
workflow workflow-engine orchestration scheduler data-pipeline elt etl data data-engineering data-orchestration

mage-ai / mage-ai
data-engineering,🧙 Build, run, and manage data pipelines for integrating and transforming data.
Organization: mage-ai
Home Page: https://www.mage.ai/
machine-learning artificial-intelligence data data-engineering data-science python elt etl pipelines data-pipelines

meltano / meltano
data-engineering,Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
Organization: meltano
Home Page: https://meltano.com/
dataops dataops-platform elt open-source opensource data pipelines extract-data meltano meltano-sdk

memphisdev / memphis
data-engineering,Memphis.dev is a highly scalable and effortless data streaming platform
Organization: memphisdev
Home Page: https://memphis.dev
data data-stream-processing data-streaming kubernetes messaging-queue data-engineering data-pipeline golang enrichment message-broker

metarank / metarank
data-engineering,A low code Machine Learning personalized ranking service for articles, listings, search results, recommendations that boosts user engagement. A friendly Learn-to-Rank engine
Organization: metarank
Home Page: https://metarank.ai
ranking scala search personalization machine-learning deep-learning data-engineering feature-engineering feature-extraction kubernetes data-science automl neural-networks

moataz-elmesmary / data-science-roadmap
data-engineering,Data Science Roadmap from A to Z
User: moataz-elmesmary
Home Page: https://www.linkedin.com/posts/moatazelmesmary_github-moataz-elmesmarydata-science-roadmap-activity-6974293994960769024-yCdi?utm_source=share&utm_medium=member_desktop
data-analysis data-engineering data-science data-visualization deep-learning machine-learning mathematics probability python sql

ploomber / ploomber
data-engineering,The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️
Organization: ploomber
Home Page: https://docs.ploomber.io
workflow machine-learning data-science data-engineering mlops papermill jupyter jupyter-notebooks pipelines vscode

prefecthq / prefect
data-engineering,Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines
Organization: prefecthq
Home Page: https://prefect.io
python workflow data-engineering data-science workflow-engine prefect infrastructure ml-ops data-ops automation

pyjanitor-devs / pyjanitor
data-engineering,Clean APIs for data cleaning. Python implementation of R package Janitor
Organization: pyjanitor-devs
Home Page: https://pyjanitor-devs.github.io/pyjanitor
pandas dataframe data cleaning-data data-engineering pydata hacktoberfest

quadratichq / quadratic
data-engineering,Quadratic | Data Science Spreadsheet with Python & SQL
Organization: quadratichq
Home Page: https://QuadraticHQ.com
python data-science spreadsheet sql quadratic etl data-engineering data-analysis data wasm

quiltdata / quilt
data-engineering,Quilt is a data mesh for connecting people with actionable data
Organization: quiltdata
Home Page: https://quiltdata.com
data data-engineering data-version-control data-versioning python serialization parquet

risingwavelabs / risingwave
data-engineering,Cloud-native SQL stream processing, analytics, and management. KsqlDB and Apache Flink alternative. 🚀 10x more productive. 🚀 10x more cost-efficient.
Organization: risingwavelabs
Home Page: https://www.risingwave.com/slack
database stream-processing cloud-native sql distributed-database rust serverless postgresql real-time postgres

running-elephant / datart
data-engineering,Datart is a next generation Data Visualization Open Platform
Organization: running-elephant
Home Page: https://running-elephant.github.io/datart-docs/
datart davinci business-intelligence data-visualization typescript dashboard chart react bi analytics

san089 / udacity-data-engineering-projects
data-engineering,Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
User: san089
data data-engineering data-engineering-pipeline etl-pipeline cassandra-database postgresql-database data-modeling data-warehouse data-lake airflow

sodadata / soda-core
data-engineering,:zap: Data quality testing for the modern data stack (SQL, Spark, and Pandas) https://www.soda.io
Organization: sodadata
Home Page: https://go.soda.io/core-docs
python data-engineering data-governance data-monitoring data-observability data-profiling data-quality data-quality-checks data-quality-monitoring data-reliability

treeverse / lakefs
data-engineering,lakeFS - Data version control for your data lake | Git for data
Organization: treeverse
Home Page: https://docs.lakefs.io
data-engineering data-versioning go object-storage data-lake aws-s3 data-quality azure-blob-storage google-cloud-storage golang

whoiskatrin / sql-translator
data-engineering,SQL Translator is a tool for converting natural language queries into SQL code using artificial intelligence. This project is 100% free and open source.
User: whoiskatrin
Home Page: https://www.sqltranslate.app/
data-analysis data-engineering dataquery datascience dataset openai postgresql query sql

xonsh / xonsh
data-engineering,:shell: Python-powered, cross-platform, Unix-gazing shell.
Organization: xonsh
Home Page: http://xon.sh
python xonsh prompt shell cli command-line hacktoberfest console terminal bash