This repository contains example DAGs showing features released in Apache Airflow 2.8.
Aside from core Apache Airflow this project uses:
- The Astro CLI to run Airflow locally (version 1.20.1).
- The Amazon Airflow provider with the
s3fs
extra installed. - The Slack Airflow provider.
- The Common IO Airflow provider.
- The scikit-learn package.
For pinned versions of the provider packages see the requirements.txt
file.
This section explains how to run this repository with Airflow.
Note
Some DAGs in this repository require additional connections or tools. The on_dataset_changed
listener uses a Slack Webhook connection with the ID slack_webhook_conn
Some of the ObjectStorage example DAGs (tag: ObjectStorage
) are using a connection to AWS S3 with the connection ID my_aws_conn
. If you use a different object provider you will need to adjust the connection name and URI in the relevant DAGs.
You can define these connection in the Airflow UI under Admin > Connections or by using the .env
file with the format shown in .env.example
.
See the Manage Connections in Apache Airflow guide for further instructions on Airflow connections.
DAGs with the tag core
work without any additional connections or tools.
Download the Astro CLI to run Airflow locally in Docker. astro
is the only package you will need to install.
- Run
git clone https://github.com/astronomer/2-8-example-dags.git
on your computer to create a local clone of this repository. - Install the Astro CLI by following the steps in the Astro CLI documentation. Docker Desktop/Docker Engine is a prerequisite, but you don't need in-depth Docker knowledge to run Airflow with the Astro CLI.
- Run
astro dev start
in your cloned repository. - After your Astro project has started. View the Airflow UI at
localhost:8080
.
The following sections list the DAGs shown sorted by the feature that they showcase. You can filter DAGs in the UI by their tags
.
object_storage_FileTransferOperator
: uses theFileTransferOperator
to transfer files from one location in S3 to another.object_storage_ftp_backend
: attaches a custom FTP backend to theObjectStoragePath
class to interact with an FTP server.object_storage_list_read_remote
: simple example of a pipeline using several methods of theObjectStoragePath
class to list, read and write files to an object storage.object_storage_load_file_to_duckdb
: loads a local parquet file into a DuckDB database using theObjectStoragePath
class.object_storage_showcase
: shows several methods and attributes of theObjectStoragePath
class.object_storage_use_case
: simple ML pipeline that trains a model and uses the Object Storage feature to move training data between different locations.
branching_example
: shows the new@task.branch_external_python
and@task.branch_virtualenv
decorators.html_example
: contains raw HTML in the DAG Docs and Airflow Params description, which is rendered in the UI if becauseAIRFLOW__WEBSERVER__ALLOW_RAW_HTML_DESCRIPTIONS
is set toTrue
in the Dockerfile.params_example
: a DAG using several Airflow params to show the newTrigger DAG
UI.pool_example
: a DAG using a pool calledmy_garden_pool
to limit the number of tasks running in parallel. You will need to create this pool in the Airflow UI under Admin > Pools to run this DAG.producer_example
: produces to a Dataset to show theon_dataset_changed
listener. Note that theon_dataset_changed
listener uses a Slack Webhook connection with the IDslack_webhook_conn
. You will need to create this connection in the Airflow UI under Admin > Connections to get Slack notifications from the listener.xcom_tab_example
: DAG that pushes several XComs to show the new XCom tab in the UI.deltalake_example
: Shows Delta Lake table serialization.literal_wrapper_example
: Shows theliteral
function used to disable Jinja templating in a task parameter.
DAGs that are here to support another DAG.
consumer_dag
: DAG that is triggered by the Dataset updated by theproducer_dag
.
- Object Storage Basic tutorial.
- Object Storage OSS Docs tutorial.
- Object Storage OSS Docs guide.
- Branching Guide.
- Listeners OSS Docs.
- Datasets guide.
- Airflow Config reference.
This repository contains the following files and folders:
.astro
: files necessary for Astro CLI commands.dags
: all DAGs in your Airflow environment. Files in this folder will be parsed by the Airflow scheduler when looking for DAGs to add to your environment. You can add your own dagfiles in this folder.include
: supporting files that will be included in the Airflow environment. Among other files contains the code for the listener plugin ininclude/listeners.py
.plugins
: folder to place Airflow plugins. Contains a listener plugin.tests
: folder to place pytests running on DAGs in the Airflow instance. Contains default tests..astro-registry.yaml
: file to configure DAGs being uploaded to the Astronomer registry. Can be ignored for local development..dockerignore
: list of files to ignore for Docker..env.example
: example environment variables for the DAGs in this repository. Copy this file to.env
and replace the values with your own credentials..gitignore
: list of files to ignore for git.Dockerfile
: the Dockerfile using the Astro CLI. Sets environment variables to change Airflow webserver settings.packages.txt
: system-level packages to be installed in the Airflow environment upon building of the Docker image. Empty.README.md
: this Readme.requirements.txt
: python packages to be installed to be used by DAGs upon building of the Docker image.