Git Product home page Git Product logo

ydepledt / de Goto Github PK

View Code? Open in Web Editor NEW

This project forked from supaerodatascience/de

0.0 0.0 0.0 63.4 MB

Data Engineering class in the Supaero Data and Decision Sciences program

Home Page: https://supaerodatascience.github.io/DE/

License: MIT License

JavaScript 18.30% Python 1.68% Julia 0.05% CSS 12.87% Makefile 0.16% HTML 4.48% PLpgSQL 0.06% Jupyter Notebook 56.79% Dockerfile 0.06% SCSS 5.56%

de's Introduction

Data Engineering

The amount of data in the world, the form these data take, and the ways to interact with data have all increased exponentially in recent years. The extraction of useful knowledge from data has long been one of the grand challenges of computer science, and the dawn of "big data" has transformed the landscape of data storage, manipulation, and analysis. In this module, we will look at the tools used to store and interact with data.

The objective of this class is that students gain:

  • First hand experience with and detailed knowledge of computing models, notably cloud computing
  • An understanding of distributed programming models and data distribution
  • Broad knowledge of many databases and their respective strengths

As a part of the Data and Decision Sciences Master's program, this module aims specifically at providing the tool set students will use for data analysis and knowledge extraction using skills acquired in the Algorithms of Machine Learning and Digital Economy and Data Uses classes.

Class structure

The class is structured in three parts:

Data storage

In the first 10 hours of the course, the history of data storage from single databased management systems to distributed filesystems will be presented. For evaluation, students will install and manipulate data in PostgreSQL.

Data computation

20 hours on the computing platforms used in the data ecosystem. We will briefly cover cluster computing and then go in depth on cloud computing, using Google Cloud Platform as an example. Finally, a class on GPU computing will be given in coordination with the deep learning section of the AML class.

Data distribution

20 hours on the distribution of data, with a focus on distributed programming models. We will introduce functional programming and MapReduce, then use these concepts in a practical session on Spark. Finally, students will do a graded exercise with Dask.

Class schedule

Data Storage Readings
SQL 3h 18/09/2023 Databases introduction (fr)
PostgeSQL 3h 25/09/2023 PostgeSQL
Parallel DBMS 4h 04/10/2023
Data Computation Readings
Cloud Computing 3h 21/11/2023 Readings
Containers 3h 28/11/2023 Readings
Cloud Compute BE 3h 29/11/2023
Distributed DBMS 3h 04/12/2023
GPU computing 6h 13/12/2023
Exam 2h 19/12/2023
Data Distribution Readings
Orchestration 3h 08/01/2024 Readings
Deployment TP 3h 09/01/2024 Readings
Hadoop and MapReduce 3h 16/01/2024 MapReduce
Spark 4h 17/01/2024 Spark PySpark
Cloud DBMS 3h 04/12/2023
Dask 3h 13/02/2024 Dask documentation
Dask project 3h 13/02/2024 Dask

de's People

Contributors

d9w avatar fchouteau avatar guillaumeeb avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.