Proof of concept to integrate the Datadog API with callbacks and Airflow tasks to update Datadog objects (i.e. Logs, Metrics, etc.)
git clone [email protected]:astronomer/cs-tutorial-datadog.git
cd cs-tutorial-datadog
astro dev start
Note that the airflow_settings.yaml
file has the datadog_default
connection predefined and it should generate automatically. However, if it doesn't automatically populate when you use the astro d start
command, you can either add it manually using the steps below, or just try restarting your sandbox using astro dev stop && astro dev start
. Keep in mind that you'll need to replace the <INSERT-API-KEY>
and <INSERT-APP-KEY>
of the connection's extra field with your actual API key and APP key.
- Go to your sandbox http://localhost:8080/home
- Navigate to connections (i.e. Admin >> Connections)
- Add a new connection with the following parameters:
- Connection Id:
datadog_default
- Connection Type:
HTTP
- Host:
https://http-intake.logs.datadoghq.com
- Extra:
{"api_key": "<INSERT-API-KEY>", "app_key": "<INSERT-APP-KEY>"}
- Connection Id:
- Go to your DAGs by navigating to http://localhost:8080/home
- Unpause the DAG by toggling the switch to the left of the
datadog_poc
DAG - Click the trigger icon under the Actions section of the
datadog_poc
DAG
Check your Datadog log explorer, you should see logs begin to populate for each of the tasks in the DAG
You may have noticed that one of the tasks failed in your DAG run, this is by design as it will allow you to see what task failures would look like in the Datadog Logs UI
If you look at the code for the datadoc_poc
DAG, you'll notice that in the default_args
for the DAG, we've set the following parameters:
on_success_callback
on_failure_callback
on_retry_callback
These callback functions are defined in the /include/datadog.py
directory and can be customized to suite your organization's needs.
There is also another task in our DAG that is called send_log_to_datadog
this also uses the custom library located at /include/datadog.py
to pass sepecified parameters to the Datadog log. In this POC, we are using Jinja Template Variables. But these can also be customized to suite your organization's needs.