wonderakwei / robust-data-pipeline-csv-to-postgresql-with-airflow-docker-and-pandas Goto Github PK

Data Engineering End-to-End with PostgreSQL, Airflow, Docker, Pandas

Overview: This project demonstrates an end-to-end data engineering process using PostgreSQL, Apache Airflow, Docker, and Pandas. The primary goal is to establish a robust data pipeline for seamlessly transferring CSV data into a PostgreSQL database.

Tech Stack:

• Apache Airflow: Task orchestration
• PostgreSQL: Database management
• Pandas: Efficient data processing
• Docker: Containerized deployment
• Python: Programming language
• SQL: Query language

Workflow:

1. Get Remote CSV Data:

   • Download a CSV file from a remote repository.
   • Configure PostgreSQL connection parameters.

2. Data Import and Table Creation:

   • Create a local PostgreSQL table.
   • Write CSV data to the PostgreSQL table using a Python script.

3. Data Modification and DataFrame Creation:

   • Retrieve data from the table.
   • Perform data modifications and Pandas practices.
   • Create three separate data frames.

4. Write Data Frames to PostgreSQL:

   • Create related tables in the PostgreSQL database.
   • Insert data frames into these tables.

5. Automate with Airflow DAGs:

   • Utilize Airflow for task automation.
   • Schedule daily execution of tasks.

Usage:

1. Configure PostgreSQL connection parameters.
2. Install necessary libraries using pip install -r requirements.txt.
3. Run Python scripts for CSV import, table creation, data modification, and DataFrame creation.
4. Execute Airflow DAG for automated task execution.