Data Engineering End-to-End with PostgreSQL, Airflow, Docker, Pandas
Overview: This project demonstrates an end-to-end data engineering process using PostgreSQL, Apache Airflow, Docker, and Pandas. The primary goal is to establish a robust data pipeline for seamlessly transferring CSV data into a PostgreSQL database.
Tech Stack:
- Apache Airflow: Task orchestration
- PostgreSQL: Database management
- Pandas: Efficient data processing
- Docker: Containerized deployment
- Python: Programming language
- SQL: Query language
Workflow:
-
Get Remote CSV Data:
- Download a CSV file from a remote repository.
- Configure PostgreSQL connection parameters.
-
Data Import and Table Creation:
- Create a local PostgreSQL table.
- Write CSV data to the PostgreSQL table using a Python script.
-
Data Modification and DataFrame Creation:
- Retrieve data from the table.
- Perform data modifications and Pandas practices.
- Create three separate data frames.
-
Write Data Frames to PostgreSQL:
- Create related tables in the PostgreSQL database.
- Insert data frames into these tables.
-
Automate with Airflow DAGs:
- Utilize Airflow for task automation.
- Schedule daily execution of tasks.
Usage:
- Configure PostgreSQL connection parameters.
- Install necessary libraries using
pip install -r requirements.txt
. - Run Python scripts for CSV import, table creation, data modification, and DataFrame creation.
- Execute Airflow DAG for automated task execution.