In this repository, there are 2 folders: AmazonDataExtraction and SpotifyETL

In this project, the goal was to extract data from the amazon website using BeautifulSoup library.

1. The rough basic work is shown in:

Click Here

2. After aggregating all the logic from the basic rough work, I have defined all the functions and extracted data from the website in:

Click Here

3. The extra file "amazon_etl.py" is a folder created to define all the functions in a python file and can be later form a pipeline using AirFlow.

1. Loading the extracted data to S3 storage using Airflow.

2. Then, Modeling the data into star schema and finally loading into Redshift for further Analytical Work

In this project, the goal was to extract data from a playlist of spotify using Spotify API (Spotipy Library)

1. Build a basic file to explore and extract the data:

Click Here

2. Build a python file that connects us to the spotify api:

Click Here

3. Build a python file to define the proper functions of extracting with the help of the above rough work file:

Click Here

4. Now, Create an EC2 instance in AWS Console.

5. After Connecting to the instance, Install all the dependencies needed for the server:

Click Here

6. Finally, create a DAG file to be used in Airflow:

Click Here

Then, Perform the functions through Airflow and the data will be loaded in S3 Storage.

• Modeling the data into star schema and finally loading into Redshift for further Analytical Work.

My name is WAREPAM RICHARD SINGH. In this Project, I have learned:

1. Data Extraction
2. Data Modelling: (draw.io/ Lucid)
3. Data Transformation
4. Data Loading
5. AWS Services: S3, Apache Airflow, Redshift

For more project Updates, You can find me on:

• Twitter
• Instagram
• LinkedIn