This repository contains 3 main parts - strava/, weather/ and iac/

I chose to adopt a monorepo approach only because this is more of an exploratory/hobby work and did not want the hassle of maintaining multiple repositories.

strava/ contains all code around the strava pipeline.

• A batch ELT data pipeline in Python, connecting to Postgres DB, orchestrated by Airflow and dbt (through ECS).

weather/ contains all code around the weather pipeline.

• A realtime data pipeline in Golang utilizing Kafka on a Kubernetes Service, connecting to Cassandra, with Terraform as the IaC.

iac/ contains all Terraform (chosen IaC) code.

• All cloud resources with the exception of SSM Parameters are provisioned using Terraform.

• extract/ contains the logic of data extraction from STRAVA API.
• load/ contains the logic of loading data from landing buckets to database.
• transformation/ contains the transformation logic.
• orchestration/ contains the airflow code and DAGs.

1. Obtain the following credentials from STRAVA App Integration:

   • CLIENT_ID
   • CLIENT_SECRET
   • REFRESH_TOKEN

2. Store credentials in AWS SSM Parameter Store.

3. Implement logic of data extraction in main.py.

4. Pass credentials into container through ECS Task Definition.

5. Dockerise extract/ and push to ECR. CD through GHA has been implemented in cd.yaml to update image upon merging to main.

6. Compute will be orchestrated by Airflow through custom operator StravaToS3Operator which inherits from ECSRunTaskOperator.

Supabase is chosen as the postgres database for this project mostly because they recently went GA, and the UI looks pretty clean and most importantly I can keep within the free tier very comfortably.

• Data loaded from S3.

dbt is chosen to handle all data transformation work required.

A monorepo approach to dbt Project management is taken because there will be dependencies between strava dbt_project and weather dbt_project -- I'd prefer to have them all in 1 place just so the dependencies between can be captured by dbt.

Airflow is chosen to manage all orchestration work around extracting, loading and transforming of data.

• Extraction of data from STRAVA API to S3 bucket: StravaToS3Operator
• Loading of data from S3 bucket to Supabase: S3ToSupabaseOperator

• Custom StravaToS3Operator inherits EcsRunTaskOperator and is created to call the STRAVA API for extraction.
• Similarly, custom S3ToSupabaseOperator also inherits EcsRunTaskOperator and helps to load data from my S3 bucket to Supabase Postgres database.
• Lastly, DbtOperator which triggers dbt tasks through ECS to execute the transformation logic.
• All 3 logic (STRAVA extraction, Loading to Supabase & dbt Transformation) are managed in extract/, load/ and transformation/ respectively.

• Deployed to AWS S3 bucket through Github Actions aws s3 sync for MWAA cluster

• Local airflow development environment for testing
• Symlinked to orchestration/dags/
• Use docker-compose up to spin up local airflow

This was implemented in golang with Open-Meteo API.

This is also implemented in golang to consume events from a specified Kafka topic, processes the events, and then lands the data in an S3 bucket and a Cassandra instance.

Terraform is chosen to support the IaC for this entire strava pipeline project.

• ECS Task Definition
• ECR
• Cloudwatch Logs
• S3 Buckets
• Networking
  • VPC
  • Subnets
  • Security Group
• Identity Access Management
  • Service User for GHA
  • ECS Task Execution Role
  • ECS Task Role
  • Respective IAM policies required around authorisation management

• SSM Parameters

Scalr was chosen to support remote terraform operations. The free tier supports up to 50 terraform operations monthly.

terraform plan will execute upon raising a PR with commits from the declared directory -- iac/.

auto apply has been disabled and plans have to be manually approved on the Scalr UI, which can be navigated from the PR comments.

In this project, GitHub Workflows along with OIDC authentication are leveraged to automate the process of pushing/updating images to ECR on AWS.

Selected direnv along with a .envrc file to manage environment variables in the development environment. This automatically loads environment variables when entering the project directory.

For Terraform-related environment variables, they are prefixed with TF_VAR_. This ensures that the environment variables can be registered by .tf files.

In Python code, a straightforward method of retrieval is used (os.environ.get) to access environment variables.

Confidential credentials, such as API keys and database passwords, are securely stored in the AWS Systems Manager (SSM) Parameter Store. They are retrieved securely at runtime and passed into the ECS containers.

Jira