Remove masterlist download altogether; just use date to download about gdeltpyr HOT 5 CLOSED

Skip download via multiprocessing altogether. Rather, use the multiprocessing workers to read pandas dataframes directly from the url list I build based on the dates passed. This should significantly save time.

from gdeltpyr.

Sample

import pandas as pd
df = pd.read_csv('/Users/linwood/Downloads/20150218230000.export.CSV.zip', compression='zip',sep='\t',header=None)

from gdeltpyr.

Check if blaze or dask can support this to handle the big data solution.

from gdeltpyr.

multiprocessing alone has sped this up orders of magnitude; concepts in notebook. push before week is out and implement in core code.

from gdeltpyr.

All features included in 787f2f7

from gdeltpyr.