bwbaugh / dentonpolice Goto Github PK

See the Twitter docs for more details: https://dev.twitter.com/rest/reference/post/statuses/update_with_media

Right now it's a very manual process to verify that changes work as intended and don't break. Without automated tests, the only verification options are either an ad hoc simulation with a REPL, which is takes too much manual effort, or to verify on production, which can take time since we have to wait for new data.

The City Jail Report page often posts old mug shots when the arrest information is first posted, and then updates the mug shot image after a while with a newer mug shot. It would be worth saving these newer mug shots as well, which we do not currently since the app only retrieves images for the inmates that it wants to post to Twitter.

It should be a drop in replacement for the logging module: https://structlog.readthedocs.org/en/latest/standard-library.html

and has many benefits, including outputting JSON and binding values to the logger e.g., the S3 key name of the relevant jail-report.

It sounds like getting 'events' from a user stream would allow us to save things like mentions and favorites.

Switch to using a library built for configuration instead of rolling our own. For example: https://github.com/dnephin/PyStaticConfiguration

Getting the jail report or opening a mug shot URL sometimes blocks indefinitely, and the crawler needs to be restarted.

All dates should be in a standard format (ISO) and be set to a standard timezone (UTC).

Background

The date fields (e.g., seen, arrest, DOB) have different formats and even different timezones. For example the arrest timezone is always US/Central as it's from the City Jail Report website, however the seen timezone depends on the local machine's settings.

Motivation

If we want to support browsing, searching, and filtering of the logged data e.g., using ElasticSearch), then it needs to be in a consistent format.

As a follow up to GH-12, whenever a new mug shot is found for an inmate that has already been posted to Twitter, reply to the original tweet with the updated mug shot.

The original tweet is in the log file, but it might be necessary to use a DB or a separate log file in the meantime in order to look up the prior tweet and image for comparison.

Create an AppSpec file and any other needed scripts in order to allow automatic deployments using AWS CodeDeploy.