Aim: Develop an end-to-end data pipeline with an aim to ingest Airbnb data into storage, and data lake and then transform it.
Azure Data Factory, Data Lake Gen 2, and Azure Databricks
- Data Ingestion into Azure Data Factory
- Store Raw data in Data Lake
- Transform the data using Databricks and Sparks query
- Load the transformed data back into the lake