saurabhskulkarni / data-wrangling-project-udacity Goto Github PK

Data Wrangling Project

By Saurabh Kulkarni

Submitted as a part of Udacity's Data Analyst Nanodegree Program Coursework

Summary

This project is a hands on experience on gathering data from various sources in various formats and assessing it for quality and tidiness issues followed by cleaning.

The project has been completed using Jupyter notebook. Data wrangling and visualizations have been demonstrated here.

The dataset that was wrangled is a tweet archive for user @dog_rates (also known as WeRateDogs). The account rates people's dogs with a humorous comment about the dog. There ratings almost always have a denominator of 10 and numerators greater than 10. This account has more than 4 million followers and is widely popular.

WeRateDogs twitter archive has been exclusively supplied by Udacity for this project. It contains basic data on tweets like ID, timestamp, text, etc. This data is from August 1, 2017.

What was used?

To complete this project, Jupyter notebook along with the following packages was used:

pandas

numpy

requests

tweepy

json

Objective of Project

The objective of this project was to gather all data from different sources provided and wrangle it (analyze and clean). Finally, some interesting analysis and visualizations have also been developed.

The Data

The project details includes a link to the twitter archive mentioned above. There are 2356 tweets with ratings. This data was extracted programmatically however it contains some errors.

Twitter API was used to get other tweet related information like favorite and retweet count.

An image predictions file has been provided that was run on these images (in tweets) to identify the breed of each dog in the picture. The confidence levels of each prediction are also provided in the data.

Tasks of the Project

Data Wrangling which included:

    > Gathering data programmatically
    
    > Assessing data (8 quality issues and 2 tidiness issues were identified)
    
    > Cleaning data (cleaning for the above documented issues)

Storing cleaned data: Creating a master file

Analyzing cleaned data and visulization

Reporting wrangling efforts and visulizations

Finally, two reports each for wrangling efforts and visulization were developed