Automated Data Processing At Scale about hydrator HOT 2 CLOSED

Hi @JROliver3 -- thanks for the question.

If you want to automate hydration I suggest you take a look at our other utility twarc. It is a command line utility, but also a Python software library. Hydrator was really designed for people who are comfortable doing data analysis on CSV files, but aren't familiar with running things from the command line or doing systems programming.

For example, to hydrate a tweet id file with twarc you can:

twarc hydrate ids.txt > tweets.jsonl

You could achieve something like what you are talking about by automating a system call, which should be doable in your preferred programming language.

Or if you know Python it would probably be cleaner to use twarc as a library. That would allow you to introduce logic for looking for files in a directory, and writing them out where you need to, maybe even to a database, or data pipeline if you wanted.

I actually helped write a little program for a group doing COVID-19 related Twitter research. It loops through a directory and hydrates the data, and might help illustrate a bit more about how you can use twarc programmatically:

https://github.com/echen102/COVID-19-TweetIDs/blob/master/hydrate.py

Please feel free to ask more questions here, or close this issue if you feel like it has been resolved.

from hydrator.

@edsu

Thanks a bunch for the reply. This will be a huge help and will prevent a lot of work for me in the future (considering that I don't know much about the hydration process on my own). Coincidentally (or maybe not) I'm also working on a COVID-19 related project using scraped data from Twitter, so it sounds like that project should be perfect. I'll check it out.

My problem is resolved, closing for now. Thanks again.

from hydrator.