data4democracy / discursive Goto Github PK
View Code? Open in Web Editor NEWTwitter topic search and indexing with Elasticsearch
Twitter topic search and indexing with Elasticsearch
Ultimately, we'll want to build analytical products using the Twitter data as a source. To do that, we'll identify things like distinct user handles, capture referenced URLs and common hashtags. Additionally, we'll look to utilize some NLP techniques to dig deeper into the Tweet texts and profile descriptions.
The Twitter data is stored in S3 in JSON so pulling that data into an analytical environment (maybe a Jupyter notebook?) could be a great task for beginners to tackle as there's plenty of documentation on the interwebs describing how to perform data analysis using Python with JSON data sources!
Related to my proposal on storage options #13, I think we are getting to the point where a full fledged CLI would be useful.
When designing, important to make sure we set sane defaults so it "just works" without arguments. This will help users unfamiliar with the tool get up and running quickly.
Brainstorming here but some arguments we may want to consider accepting via CLI this list is probably too large, whatever we do not support in CLI may be more appropriate for in a settings file:
Currently, the tweet mapping in essetup.py does not match what is collected in index_twitter_*.py. This needs to match.
Process that will accept a list of users and poll daily. Should save a time stamp and user attributes :
Allows us to see how a profile changes over time, fastest growing profiles etc.
Having recently "Dockerized" the Twitter collection process, one piece we're missing is the ability to run the Twitter collection from within the Docker container on a schedule (i.e. using cron). The OS is Alpine Linux, and here's a good example of what we're trying to do. Check out the branch here.
Please assign this to yourself if you are working on it so we're not stepping all over each other :)
Looking for someone to take a look at index_twitter_search.py
and get it working. Currently prints malformed data to console when run. Contact @nick on Slack to get access to the Elasticsearch index for your local testing if you pick this up!
This is a placeholder for the design (and associated discussion) of our foundational NLP capability to analyze collected Tweets. Divya (@divya on Slack) and Wendy (@wwymak on Slack) will be taking the lead with support from anyone/everyone else who wants to help! The goal will be to publish a proposed design for the implementation to this issue later this week and get feedback from the community. Anyone else who's interested in participating please don't hesitate to contact them.
To enable analysts we need to ensure we can get valid JSON written to S3 from the index_twitter_stream.py class StreamListener
The write to S3 should create a key (filename) with a timestamp as this stream runs every 15 minutes. Bonus points for creating a way to zip/concatenate files with filenames as arguments (so analysts could, for instance, take all Tweets from a given day, week, etc.).
There are lots of ways to do community detection using Twitter data. We'll want to discuss the nuts-and-bolts on Slack but once we select an implementation we like we can track progress here. There's lots of neat emerging research we could try out, too (i.e. https://arxiv.org/pdf/1608.01771v1.pdf)!
In order to generalize the tool to make it useful to the widest audience I think it would be beneficial to support multiple options for saving tweets with the ability to configure 1 or more option. In order to accomplish this I think we should refactor any of the storage related work to a separate process that uses config to determine how/where to save tweet. If we agree on the approach to do this, we can open new issues for each back end we want to add.
For starters:
This issue pertains to the discursive-graph-data branch
build_status_attr.py
is currently a mess. It needs to return the entire Tweet object from Tweepy (so, include "entities"). The returned object will contain important information about a given Tweet (status) that we can use for analysis.
Currently, it throws TypeError: 'int' object is not iterable
If you would like to work on this issue and need help setting up access to the Elasticsearch resource for your local testing please contact @nick on slack. This is a relatively painless process as long as you have an AWS account. Conversely, if you want to test disconnected from AWS resources, you can do so by commenting-out any of the AWS or Elasticsearch affiliated references (you'll still need Twitter, though).
We need to update the README to reflect:
Rather than having the tracking topics we want to use against the Twitter streaming API stored in our codebase let's move them to an input file. So, as a user, I can point to a file (i.e. on S3 or locally) to use as a source for my keywords!
In conversation with @ASRagab, @bstarling and @nataliaking over the past several weeks we've contemplated migrating the Discursive application to a 'serverless' architecture supported by Kinesis and Lambda. In terms of desired functionality (delivering Twitter data to researchers) the master
branch works just fine, albeit requiring a level of infrastructure expertise our research colleagues may not possess. To that end, we would like to gather community feedback regarding whether this migration to a serverless architecture is something we should pursue. Please provide feedback if you have it, we'd love to hear from you!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.