emojitracker / emojitrack-feeder Goto Github PK

Last summer, Instagram devs did some initial analysis of emojis on Instagram - http://instagram-engineering.tumblr.com/post/117889701472/emojineering-part-1-machine-learning-for-emoji

The techniques in that article might serve as a nice starting place for exploring emoji meanings, perhaps to be incorporated on the server side with the "ensmallened" tweet text.

For example, Independent of their original purpose and definition, an emoji's meaning can be altered based on other factors that result in their usage. The parking symbol P is one instance I've noticed in looking at new trends for previously rarely used emoji.

A Redis script is transactional by definition, so everything you can do with a Redis transaction, you can also do with a script, and usually the script will be both simpler and faster.
http://redis.io/topics/transactions

Look into replacing multiple Redis calls per transaction with a single script EVAL. This will make development slightly more complex but could radically decrease amount of Redis writes necessary from the client side (and since most client libs seem to top out in the 20K ops/sec range, we could really use this.)

As per some stats I just ran, the average tweet that contains emoji contains 1.4 emoji characters.

Currently our event loop does a single redis EVALSHA for each emoji. It would be possible to modify the script to take an array of affected emoji codepoints, and do the forEach logic on the Redis server side. The tinyjson blob applies to each equally, so outgoing bandwidth would be spared.

We already cut down outgoing Redis traffic almost 10x via #5, but this could result in another ~30% reduction potentially, not nearly as dramatic, but cataloging here as a potential improvement in the future!

If official twitter gem adds support for delimited: length (see discussion in sferik/twitter-ruby#631), the performance characteristics could improve sufficiently that it would be beneficial to replace tweetstream, and could buy is some additional time without having to migrate off Ruby or to big-ass servers.

Oj is definitely faster, but how much faster in our actual use case? Useful to know because we would not be able to use it in Jruby due to C extensions.

Don't know if would be interesting or not yet, but was wondering if you had considered storing emoji bigrams. Seems like more than one usually appears in a tweet anyway.

Currently only being used as a Tweet entity provider to TweetStream (although we may eventually change that via #6).

This should be done along with deprecation of kiosk mode in #7 since we will be using it in less places then. Note, will need to run some tests to make sure nothing has changed, and DEFINITELY performance profile to guard against any regressions.

Introduced a breaking change to production by testing #2 live (whoops), which changed the format of the profile_image_url without my realizing.

Using this issue as a way to track mitigation.

This would add a small amount of size to our ensmallen()-ed tweets, but people seem to be very curious about language distribution, so this is a first step towards making sure that information makes it into Redis.

Note this is different from actually aggregating based on language locally, which would be a separate issue if we decide to do it.

since this is not being currently used, try to isolate it a bit better from the main codebase to avoid maintenance burden until it is needed again.

I am not sure of impact ... except it could probably take up a lot of space quickly... but it would be nice to have timestamps logged as part of the redis updates to facilitate trend analysis.