emojitracker / emojitrack-feeder Goto Github PK
View Code? Open in Web Editor NEW:dizzy: consumes Twitter Streaming API for Emojitracker
:dizzy: consumes Twitter Streaming API for Emojitracker
Last summer, Instagram devs did some initial analysis of emojis on Instagram - http://instagram-engineering.tumblr.com/post/117889701472/emojineering-part-1-machine-learning-for-emoji
The techniques in that article might serve as a nice starting place for exploring emoji meanings, perhaps to be incorporated on the server side with the "ensmallened" tweet text.
For example, Independent of their original purpose and definition, an emoji's meaning can be altered based on other factors that result in their usage. The parking symbol P is one instance I've noticed in looking at new trends for previously rarely used emoji.
A Redis script is transactional by definition, so everything you can do with a Redis transaction, you can also do with a script, and usually the script will be both simpler and faster.
http://redis.io/topics/transactions
Look into replacing multiple Redis calls per transaction with a single script EVAL. This will make development slightly more complex but could radically decrease amount of Redis writes necessary from the client side (and since most client libs seem to top out in the 20K ops/sec range, we could really use this.)
As per some stats I just ran, the average tweet that contains emoji contains 1.4 emoji characters.
Currently our event loop does a single redis EVALSHA
for each emoji. It would be possible to modify the script to take an array of affected emoji codepoints, and do the forEach logic on the Redis server side. The tinyjson
blob applies to each equally, so outgoing bandwidth would be spared.
We already cut down outgoing Redis traffic almost 10x via #5, but this could result in another ~30% reduction potentially, not nearly as dramatic, but cataloging here as a potential improvement in the future!
If official twitter gem adds support for delimited: length
(see discussion in sferik/twitter-ruby#631), the performance characteristics could improve sufficiently that it would be beneficial to replace tweetstream, and could buy is some additional time without having to migrate off Ruby or to big-ass servers.
Oj is definitely faster, but how much faster in our actual use case? Useful to know because we would not be able to use it in Jruby due to C extensions.
Don't know if would be interesting or not yet, but was wondering if you had considered storing emoji bigrams. Seems like more than one usually appears in a tweet anyway.
Currently only being used as a Tweet entity provider to TweetStream (although we may eventually change that via #6).
This should be done along with deprecation of kiosk mode in #7 since we will be using it in less places then. Note, will need to run some tests to make sure nothing has changed, and DEFINITELY performance profile to guard against any regressions.
Introduced a breaking change to production by testing #2 live (whoops), which changed the format of the profile_image_url
without my realizing.
Using this issue as a way to track mitigation.
This would add a small amount of size to our ensmallen()
-ed tweets, but people seem to be very curious about language distribution, so this is a first step towards making sure that information makes it into Redis.
Note this is different from actually aggregating based on language locally, which would be a separate issue if we decide to do it.
since this is not being currently used, try to isolate it a bit better from the main codebase to avoid maintenance burden until it is needed again.
I am not sure of impact ... except it could probably take up a lot of space quickly... but it would be nice to have timestamps logged as part of the redis updates to facilitate trend analysis.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.