Git Product home page Git Product logo

data-machinelearning-the-boring-way's Introduction

Data & Machine Learning - The Boring Way

This tutorial walks you through setting up and building a Data Engineering & Machine Learning Platform. The tutorial is designed to explore many different technologies for the similar problems without any bias.

This is not a Production Ready Setup

Target Audience

Data Engineers, Machine Learning Engineer, Data Scientist, SRE, Infrastructure Engineer, Data Analysts, Data Analytics Engineer

Expected Technologies & Workflow

Data Engineering & Analytics

  • Kubernetes Kind Installation link
  • MinIO Integrate Object Storage on top of Kubernetes and use minio interface for simulating the s3 link
  • Apache Airflow on top of Kubernetes & Running an end to end Airflow Workflow using Kubernetes Executor link
  • Apache Spark Deploy Apache Spark on Kubernetes and run an example link
  • Prefect Setup & Running an end to end Workflow
  • Dagster Setup & Running an end to end Workflow
  • Set up an ETL job running end-2-end on apache airflow. This job contains Spark & Python Operator
  • Apache Hive Setting up Hive & Hive Metastore
  • Deploy Trino & Open Source Presto and run dana Analytics queries.
  • Integrate Superset & Metabase to run visualization. Integrate Presto with the visualization system.
  • Open Table Format using Delta
  • Open Table Format using Apache Iceberg
  • Open Table Format using Apache Hudi
  • Metadata Management using Amundsen
  • Metadata Management using Datahub
  • Setting up Apache Kafka distributed event streaming platform
  • Using Spark Structered Streaming to run an end-2-end pipeline over any realtime data sources
  • Using Apache Flink to run an end-2-end pipeline over any realtime data sources
  • Redpanda, streaming data platform to run similar workflow
  • Airbyte Data Integration platform
  • Talend UI based Data Integration
  • DBT DBT Sql Pipeline to compare with Spark and other tech
  • Debezium Change Data Capture using Debezium to sync multiple databases

Monitoring & Observability

  • Grafana Setting Up Grafana for Monitoring components. Start with Monitoring Pods
  • FluentD logging metrics from pods & interact the same with Monitoring layer
  • Setting up a full Monitoring and Alerting Platform & integrate minitoring across other technologies
  • Setting up an Observability system

Machine Learning

  • Setup Ray for Data Transformations
  • Use Scikit-learn for an example ML training
  • Setup Argo Pipeline for deploying ML Jobs
  • Setup Flyte Orchestrator for pythonic Deployment
  • Use Pytorch Lightening for runing ML training
  • Use Tensorflow for running ML training
  • Setup ML End-2-End Workflow on Flyte
  • Deploy MLFlow for ML Model Tracking & Experimentation
  • Deploy BentoML For deploying ML Model
  • Deploy Sendon Core for ML Model Management
  • Integrate MLflow with Seldon Core

Prerequisites

  • ๐Ÿณ Docker Installed
  • kubectl Installed, The Kubernetes command-line tool, kubectl, allows you to run commands against Kubernetes clusters
  • Lens Installed, UI for Kubernetes.
    This is optional, kubectl is enough for getting all relevant stats from kubernetes cluster
  • Helm The package manager for Kubernetes

Lab Basic Setup

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.