Script to aggregate data from our investigation with Black Lives Matter into Police Violence in 2014.

First install the dependences: pip install -r requirements.pip

If you haven't already, install Redis for Linux or Windows.

If you haven't already, follow the instructions in steps 1 and 2 here and download the indicated client_secret.json file to the 'mpv' subfolder.

Make a copy of app.config.template and fill it in with your info. Rename it app.config.

Run count-story-totals.py to create data/YEAR/mpv-total-story-counts.csv. This file inclues a list of:

• each victim's name
• the total number of stories published from the sources in the time period
• the number of stories from the sources that were about the victim in the time period
• the normalized number of stories from the sources that were about the victim in the time period
• the query terms that specified the victim's name
• the query filter that was used to specify the sources and the time period

It writes giant combined queries to files in data/YEAR/, which we used to create the unspidered topic. This takes about 10 minutes on my laptop.

Run mc.storyCount() on the giant query in the data/YEAR/query-with-names.txt file you created to make sure that the number of stories is reasonable. Then ask the core team to create a controversy by sending them that txt file. Once it is ready you'll need the numeric controversy id. We do this to capture bit.ly click counts so we can evaluate social sharing over time.

• controversy_id for 2013: 1421
• controversy_id for 2014: 1326
• controversy_id for 2015: 1394
• controversy_id for 2016: 1408

Run list-all-stories.py to generate a list of all the stories in the topic we created (data/YEAR/mpv-controversy-stories.csv) and totals by victim (data/YEAR/mpv-controversy-story-counts.csv). This took about 25 minutes for me, but it could take less or more time depending on the size of the topic. Manually check some random names in the spreadsheet to ensure that the stories are on-topic, and remove the necessary story ids from the topic in the Topic Mapper tool. Once the topic is regenerated, run list-all-stories.py again to generate clean story data.

Run count-coverage.py to generate the attention over time data, saved to data/YEAR/mpv-sentences-over-time.csv. This takes a few seconds.

Remember to flush your local redis cache if any of the controversies have been updated. Do this by running the __init__.py and then running cache.clean() in the Python console.