Light

mysterioussonofgod / xjob Goto Github PK

View Code? Open in Web Editor NEW

This project forked from yennanliu/airflowjob

1.0 0.0 0.0 635 KB

Airflow POC demo : 1) env set up 2) airflow DAG 3) Spark/ML pipeline | #DE

Dockerfile 0.52% Python 91.99% Shell 6.03% Java 1.46%

xjob's Introduction

XJob

As ETL build part of the "Daas (Data as a service) repo", we demo how to build a ETL system running for data engineering/science via Airflow in this POC project. Main focus: 1) ETL/ELT (extract, transform, load) env setting up 2) ETL code base development 3) ETL code test 4) 3rd party API intergration (Instagram, Slack..) 4) dev-op tools (Travis CI).

Daas (Data as a service) repo : Data infra build -> ETL build -> DS application demo
Airflow Heroku demo : airflow-heroku-dev
Mlflow Heroku demo : mlflow-heroku-dev

Tech

- Programming : Python 3, Java, Shell 
- Framework   : Airflow, Spark, InstaPy, scikit-learn, Keras 
- dev-op      : Docker, Travis

File structure

# .
# ├── Dockerfile             : Dockerfile define astro airflow env 
# ├── Dockerfile_dev         : Dockerfile dev file 
# ├── README.md
# ├── airflow_quick_start.sh : commands help start airflow 
# ├── clean_airflow_log.sh   : clean airflow job log / config before reboost airflow
# ├── dags                   : airflow job main scripts 
# ├── ig                     : IG job scripts 
# ├── install_pyspark.sh     : script help install pyspark local 
# ├── packages.txt           : packages for astro airflow in system level 
# ├── plugins                : plugins help run airflow jobs 
# ├── populate_creds.py      : script help populate credential (.creds.yml) to airflow 
# ├── requirements.txt       : packages for astro airflow in python  level 
# ├── .creds.yml             : yml save creds access services (slack/s3/...)

Installation

STEP 1) Prerequisites

Copy .creds.yml.dev as .creds.yml file then input your credential in it.
Please have a look on Airflow documentation before start

STEP 2') Quick Start ( via `Airflow`)

Airflow quick start

STEP 2'') Quick Start (via `Astronomer Airflow`)

There is an issue when run Spark job via Astro airflow, feel free to leave a PR for that 🙏
Astro Airflow quick start

Development

Docker image

Docker hub

CI/CD

Travis

xjob's People

Contributors

Stargazers

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.