This repository contains example code for deploying Databricks Delta Live Tables pipelines with Terraform.

Delta Live Tables is a framework for building reliable, maintainable, and testable data processing pipelines. You define the transformations to perform on your data, and Delta Live Tables manages task orchestration, cluster management, monitoring, data quality, and error handling.

Terraform is an open-source infrastructure as code software tool that enables you to safely and predictably create, change, and improve infrastructure. The Terraform "databricks_pipeline" Resource is used to deploy Delta Live Tables.

• Terraform
• Delta Live Tables
• Python
• PySpark

• Install Terraform
• Install AWS CLI
• Install Databricks CLI

1. Clone the repo

$ git clone https://github.com/husqvarnagroup/terraform-dlt-public.git

2. Update the dev and live Terraform var-files in the config folder with your settings. The "backend-config" files contain settings for the Terraform S3 Backend and the "variables" files contain settings for the Delta Live Tables Pipeline.

• config/dev-backend-config.tfvars
• config/dev-variables.tfvars
• config/live-backend-config.tfvars
• config/live-variables.tfvars

1. Open a terminal and ensure you are logged into the correct AWS Profile:

$ aws sso login --profile <aws_profile_name>

2. Initialize the Terraform backend with the dev or live backend configuration file with terraform init:

$ terraform init -backend-config config/<dev-or-live>-backend-config.tfvars

If changing between dev and live configurations you must include the -reconfigure flag:

$ terraform init -reconfigure -backend-config config/<dev-or-live>-backend-config.tfvars

3. Show the Terraform Plan with the dev or live variables file to confirm everything works:

$ terraform plan -var-file=config/<dev-or-live>-variables.tfvars

4. After confirming everything works, apply the changes with the dev or live variables file:

$ terraform apply -var-file=config/<dev-or-live>-variables.tfvars

5. To delete the previously deployed resources:

$ terraform destroy -var-file=config/<dev-or-live>-variables.tfvars

Set the correct product edition based on features needed. Delta Live Tables Product Edition

Set mode depending on environment. Delta Live Tables Development or Production mode](https://docs.microsoft.com/en-us/azure/databricks/data-engineering/delta-live-tables/delta-live-tables-concepts#--development-and-production-modes)