jatvoirabotnik / dailyplanet Goto Github PK

This automated service needs to unzip the data and then organise it in a logical folder structure, for further processing down the chain.

This webpage needs to simply provide a query over day and return the collection of DO data for that day. This is just a proof of concept, to check that all the data is there, ready for further processing.

The DO for the Sao Paulo municipality is kept in a 7zip file in the following address (for 2016):
http://devcolab.each.usp.br/do/arquivos-baixar/zips/diarios-2016.7z

This task is for creating a service which starts every day and downloads the whole zipped file into some location (S3 bucket?) for further processing by other scripts. See https://github.com/JaTvoiRabotnik/docsolr/tree/master/scripts