Practice of the course: The Complete Hands-On Introduction to Apache Airflow.
- Download Docker.
- Download PyCharm Community Edition or Visual Studio Code.
Docker needs privilege rights to work, make sure you have them. If you have troubles to install these tools, here are some videos to help you
-
Open your terminal or CMD and go into airflow-materials
-
Create .env with:
AIRFLOW_IMAGE_NAME=apache/airflow:2.4.2
AIRFLOW_UID=50000- In terminal run:
docker-compose up -d- Open AirFlow in the browser: localhost:8080
Username: airflow
Password: airflow
pip install apache-airflowpip install apache-airflow-providers-postgrespip install pandas - Postgres
Connection Id: postgres
Connection Type: Postgres
Host: postgres
Login: airflow
Password: airflow
Port: 5432
- HTTP
Connection Id: user_api
Connection Type: HTTP
Host: https://randomuser.me/
Login: airflow
Password: airflow
List your docker containers
docker-compose psAccess to the scheduler bash
docker exec -it airflow-materials-airflow-scheduler-1 /bin/bashCheck the commands and execute task test
airflow -h
airflow tasks test user_processing create_table
airflow tasks test user_processing extract_user 2022-01-01
airflow tasks test user_processing process_user 2022-01-01Access to the worker
docker exec -it airflow-materials-airflow-worker-1 /bin/bashCheck worker files
ls /tmpAccess to the postgres sql
docker exec -it airflow-materials-postgres-1 /bin/bash Enter into psql
psql -UairflowThen show the data into users table
SELECT * FROM users;docker cp airflow-materials-airflow-scheduler-1:/opt/airflow/airflow.cfg .docker-compose downdocker-compose up -ddocker compose -f docker-compose-es.yaml up -ddocker compose -f docker-compose-es.yaml psdocker exec -it airflowmaterials-airflow-scheduler-1 /bin/bashcurl -X GET 'http://elastic:9200'- Elastic
Connection Id: elastic_default
Connection Type: HTTP
Host: elastic
Port: 9200
Restart docker to add ElasticHook
docker-compose -f docker-compose-es.yaml stopdocker compose -f docker-compose-es.yaml up -ddocker exec -it airflowmaterials-airflow-scheduler-1 /bin/bashairflow plugins
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent