Data

Approximately 500 million tweets are sent everyday. Of those tweets, about 2% are geotagged. Only geotagged tweets are included in the dataset. In total, there are about 1.1 billion tweets in this dataset, all from 2020.

The tweets are stored in a zip file for each day, which hold 24 texts files, one for each hour of the day. Within those text files, tweets are stored in JSON format.

Objective

Use MapReduce to anaylze the dataset and produce graphs tracking the usage of pandemic related hashtags.

map.py processes the zip file for an indiviudal day. It tracks the usage of hashtags at the language and country level, creating a zip file for each day and level with all the tweets containing the desired hashtags.

reduce.py merges the outputs of map.py into a combined file. It produces one file for all tweets at the language and one at the country level.

visualize.py generates a bar chart with the country/language on the x-axis, and the number of hashtag uses on the y-axis.

For example, to generate a bar chart displaying #cornavirus across languages in 2020, you would run the following command:

$ ./src/visualize.py --input_path=reduced.lang --key='#coronavirus'

alternative_reduce.py generates a line chart displaying their usage over 2020. It can be run as follows:

$ ./src/alternative_reduce.py --hashtags="'#covid19', '#virus', '#corona', '#nurse'"

#coronavirus Usage by Language in 2020

#코로나바이러스 Usage by Language in 2020

#coronavirus Usage by Country in 2020

#코로나바이러스 Usage by Country in 2020

Hashtag Usage Over 2020