kube-airflow

This repository contains:

• Dockerfile(.template) of airflow for Docker images published to the public Docker Hub Registry.
• airflow.all.yaml for creating Kubernetes services and deployments to run Airflow on Kubernetes

Informations

• Highly inspired by the great work puckel/docker-airflow
• Based on Debian Jessie official Image debian:jessie and uses the official Postgres as backend and RabbitMQ as queue
• Following the Airflow release from Python Package Index

Installation

Create all the deployments and services for Airflow:

    kubectl create -f airflow.all.yaml

Build

git clone this repository and then just run:

    make build

Usage

Create all the deployments and services to run Airflow on Kubernetse:

   kubectl create -f airflow.all.yaml

It will create deployments for:

• postgres
• rabbitmq
• airflow-webserver
• airflow-scheduler
• airflow-flower
• airflow-worker

and services for:

• postgres
• rabbitmq
• airflow-webserver
• airflow-flower

You can browse the Airflow dashboard via running:

make browse-web

the Flower dashboard via running:

make browse-flower

If you want to use Ad hoc query, make sure you've configured connections: Go to Admin -> Connections and Edit "mysql_default" set this values (equivalent to values in config/airflow.cfg) :

• Host : mysql
• Schema : airflow
• Login : airflow
• Password : airflow

Check Airflow Documentation

Run the test "tutorial"

    kubectl exec web-<id> --namespace airflow-dev airflow backfill tutorial -s 2015-05-01 -e 2015-06-01

Scale the number of workers

For now, update the value for the replicas field of the deployment you want to scale and then:

    make apply

Wanna help?

Fork, improve and PR. ;-)