Git Product home page Git Product logo

gobo's Introduction

Gobo Build Status

Gobo is a responsive web-based social media aggregator with filters you can control. You can use Gobo to control what’s edited out of your feed, or configure it to include news and points of view from outside your usual orbit. Gobo aims to be completely transparent, showing you why each post was included in your feed and inviting you to explore what was filtered out by your current filter settings.

Try it out at https://gobo.social.

Gobo is a project of the MIT Center for Civic Media, at the MIT Media Lab. It was created by Jasmin Rubinovitz, Alexis Hope, Rahul Bhargava and Ethan Zuckerman, with generous support from the Knight Foundation.

Installation

Gobo is a Flask-based server side, which uses React & Redux in the browser to render the UI.

Backend

Gobo uses Python 3.7.x.

Create config.py in server/config/ using the provided template to hold the right api keys and database url.

PyEnv

We manage different versions with pyenv. Install this with HomeBrew:

brew update
brew install pyenv

Then install the versions of Python we need:

pyenv install 3.7.3

PyEnv-VirtualEnv

For managing a virtual enviromnent with a specific version of python for our project, we use pyenv-virtualenv. Install this with homebrew as well

brew install pyenv-virtualenv

As noted in their readme, you'll need to add these two lines to your .bash_profile file (or you .profile file). Then open a new terminal session:

eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

And then create a virtualenv for this project. The name is important, because the .python-version file refers to it so it loads automatically when you enter the directory (if eval "$(pyenv virtualenv-init -)" is in your .profile):

pyenv virtualenv 3.7.3 gobo-3.7.3

Requirements and Database

Install all requirements:

$ make requirements-local.py

To set up the database run:

$ export FLASK_ENV=dev
$ make db-setup

Front-end

In another terminal window, cd to /client.

If you haven't already, install Node Version Manager.

Install and use version node version 10.13.0:

$ nvm install 10.13.0
$ nvm use 10.13.0

Install requirements and build static assets:

$ npm install
$ npm run build 

Running

In development mode Gobo has multiple pieces you need to run:

  1. The Flask server handles authentication and interactions between the client and the various APIs.
  2. The Redis queue holds jobs for analyzing content with the plug-in algorithms, and requests to fetch posts.
  3. Celery runs the workers to do things in the queue.
  4. We use npm to run the front-end React code that drives the UI.

Run the Flask server locally:

$ ./run.sh

In order to fetch posts from Facebook, Twitter, and Mastodon you need to run the redis-server and celery worker locally. Open 2 new shell terminals. Then run:

$ redis-server

And in the other one:

$ celery -A server.scripts.tasks worker

In another terminal window open cd to /client and then:

$ nvm use 10.13.0
$ npm start

After that you should be able to see Gobo at localhost:5000

Recurring Tasks

You need to set up three recurring tasks. The first adds tasks to the queue to fetch FB and Twitter posts for users that have been using the system recently. Run this every hour or so:

$ python -m server.scripts.queue_prioritized_user_posts

The second updates the posts from news organizations (used for the "perspectives" filter). Run this every 6 hours or so:

$ python -m server.scripts.queue_latest_news_posts

The third removes old posts (Gobo only tracks the posts within the last two weeks). Run this once a night:

$ python -m server.scripts.delete_old_posts

Manual Tasks

To delete a specific user:

$ python -m server.scripts.delete_user [user_id]

Documentation and tasks for creating and sharing rules found via the flask CLI:

$ flask 

Configuration

Beta Password

You can choose to only allow signup to people that have a special password. Add the following vars in config.py:

LOCK_WITH_PASSWORD = True
BETA_PASSWORD = 'password_you_want'

To remove the password just set LOCK_WITH_PASSWORD = False.

Set up Google Analytics:

Edit the GA ID in client/app/index.js

Development

When updating models that result in a table change (e.g. column added/removed), generate migrations with:

$ flask db migrate

This will generate a new migration file in migrations/versions that should be added to version control.

Deploying

Setup

Gobo is set up to deploy to containerized hosts like Heroku or Dokku. Typically configuration is done with environment variables. For now we've got a system that involves editing the config file on a local branch. We'll get around to changing this eventually.

  1. Create a new local branch called "deploy": git checkout -b deploy
  2. Create a new app on the Heroku website, or with the command line in Dokku
  3. Add the heroku/dokku remote to the GitHub repo
  4. In "deploy" branch, edit .gitignore to not ignore config.py (make sure to also save a copy of config.py somewhere else on your computer)
  5. On your host (Heroku/Dokku), add a database and a redis instance
  6. Update config.py in the deploy branch to match the database and redis url
  7. Push to that deploy remote: git push deploy deploy:master

!!! - Make sure to not push this branch anywhere else!! as this contains sensitive data! - !!!

Versioning

Edit client/app/constants/index.js and bump up the semantic version number before every release. This shows up at the bottom of the About page.

Contributing

A pre-commit hooks will run JavaScript linting (e.g. when you commit, linting will be run). You can try to automatically fix JavaScript linting errors by running:

$ npm run lint_fix

Not all errors can be fixed this way and for more details about the linting error see eslint.

gobo's People

Contributors

dsjen avatar jasrub avatar rahulbot avatar sarahkemi avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gobo's Issues

Facebook auth sometimes coming back without email address

Is this because the user didn't give us permission? What is the right next step - do we save the account still or is this an error mode? Also, why isn't this making it to Sentry (I only noticed because I was tailing server logs)

2017-11-28T19:35:49.153291814Z app[web.1]: [2017-11-28 19:35:49,150] ERROR in app: Exception on /api/handle_facebook_response [POST]
2017-11-28T19:35:49.153371211Z app[web.1]: Traceback (most recent call last):
2017-11-28T19:35:49.153377016Z app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
2017-11-28T19:35:49.153381787Z app[web.1]:     response = self.full_dispatch_request()
2017-11-28T19:35:49.153385546Z app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
2017-11-28T19:35:49.153389706Z app[web.1]:     rv = self.handle_user_exception(e)
2017-11-28T19:35:49.153393429Z app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
2017-11-28T19:35:49.153397906Z app[web.1]:     reraise(exc_type, exc_value, tb)
2017-11-28T19:35:49.153401539Z app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
2017-11-28T19:35:49.153405992Z app[web.1]:     rv = self.dispatch_request()
2017-11-28T19:35:49.153409519Z app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
2017-11-28T19:35:49.153414164Z app[web.1]:     return self.view_functions[rule.endpoint](**req.view_args)
2017-11-28T19:35:49.153418625Z app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/flask_login/utils.py", line 228, in decorated_view
2017-11-28T19:35:49.153422881Z app[web.1]:     return func(*args, **kwargs)
2017-11-28T19:35:49.153428226Z app[web.1]:   File "/app/server/views/social_auth.py", line 25, in handle_facebook_response
2017-11-28T19:35:49.153432874Z app[web.1]:     current_user.set_facebook_data(json_data['facebook_response'])
2017-11-28T19:35:49.153436680Z app[web.1]:   File "/app/server/models.py", line 72, in set_facebook_data
2017-11-28T19:35:49.153441447Z app[web.1]:     self.facebook_email = data['email']
2017-11-28T19:35:49.153467442Z app[web.1]: KeyError: 'email'

add guided tour

We should add an overlay-highlight tour to introduce the UI. This should probably run on the first time the user has feed content show up, and perhaps be a new button in the top toolbar (ask Alexis where?). First step is to write the text.

re-architect filters to be plugins

Right now they are bit too hard-coded. We need to pull apart the back-end implementation and the front-end UI so that they feel more like plugins. This is a design goal for the app.

feed scrolling too slow in browser

Are we using too many CSS transitions? Is there too much content? This needs a little JS debugging to see what is slow, and then we can create an approach to speed it up.

mobile twitter auth flow breaks

A number of users have said it doesn't work. They end up with Gobo saying Twitter is not connected even though Twitter says it is.

modularize filters on server side

The changes in #43 are a good start to this, but we need to do more.

I'd propose a design that has each filter get it's own module under server.filters - ie. server/filters/gender.py, etc. Then at start-time server.scripts.tasks would walk that server.filters package to discover any analyze_[modulename] methods (ie. analyze_gender in gender.py).

modularize filters on front-end

A filter's UI is made up of a number of components scattered across our front-end codebase right now:

  • the slider UI - configured in Settings.js right now, rendered by a generic SettingsItem react component
  • the explanation for the back of the content - in BackOfPost.js right now
  • the filter name - configured in Settings.js right now
  • the filter icon - specified in Settings.js right now, assigned somehow with the fontello-codes.scss
  • the filtering logic that accepts or rejects each piece of content's score - encoded in filtering.js

This makes adding a new filter a bit crazy! We need to come up with a proposal for how to put all this in one place for each filter. Perhaps an architecture where each filter lives in its own folder or .js file, and then registers itself with the front-end system to supply each of these via an agreed on API or naming convention? Or those could all be in one folder and be discovered. Not sure...

close main menu once done with it

Reproduce:

  1. log in
  2. click the main menu in the top right
  3. select an item (such as "About Gobo")

At this point the selected page should load and the menu should close. It doesn't close.

Note: it should also close if the user clicks outside of it anywhere.

text when no posts

When no post to show, present text:
"None of the posts in your feed match the filters you've set. Try changing the filters."

only show one filter at a time on mobile

In Safari if I make the window narrow on my desktop and then try to use the filters they overlap.
Ie. if I click one filter on the right to open in, and then click another then there are two filters open. It should close an open filters before showing the one I've clicked.

add quick way to disable new account creation

In case we get some media post-launch and too many people try to sign up, we need some way to quickly turn off new-user registration.

My quick proposal: can you add support for an optional ALLOW_REGISTRATION environment variable (default True) that changes the "register" button any where it shows up to a "not accepting new users right now" note.

Open to any other alternate low-hanging-fruit solutions.

modify explanation of political_quintile content

When you flip content over we should show a different message based on the distance of this content from the quintile you picked (ie. abs(my_quintile - post.political_quintile))

Here's the message to show, based on the result of that distance:

  • 0: similar to your political perspective
  • 1-2: slightly different from your political perspective
  • 3-4: very different from your political perspective

add last login and last post update columns

This issue refers to the first part of issue #31, specifically:

add a last_twitter_fetch and last_facebook_fetch timestamp columns to User

@rahulbot, does this mean that the last_twitter_fetch and last_facebook_fetch columns are condensed into one, or do we end up with 3 columns (as we will also need to include a column for the last_login)?

can't drag filter controls on iPhone

Reproduce:

  1. log in on iOS
  2. click an icon to open a filter (ie. politics)
  3. try to drag the slider

At this point the slider should move, but it doesn't :-( This works for me in Chrome and Safari on desktop (wide and narrow windows). A mobile-safari-only bug?

attempt together Facebook posts for user failed with 'friends' key error

2017-11-29T00:15:20.727941587Z app[worker.1]: [2017-11-29 00:15:20,727: ERROR/ForkPoolWorker-11] Task server.scripts.tasks.get_facebook_posts_per_user[1e8df8bd-219f-4877-96d6-09bc29b92f86] raised unexpected: KeyError('friends',)
2017-11-29T00:15:20.728015983Z app[worker.1]: Traceback (most recent call last):
2017-11-29T00:15:20.728022164Z app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 374, in trace_task
2017-11-29T00:15:20.728027732Z app[worker.1]:     R = retval = fun(*args, **kwargs)
2017-11-29T00:15:20.728032051Z app[worker.1]:   File "/app/server/factory.py", line 54, in __call__
2017-11-29T00:15:20.728036612Z app[worker.1]:     return TaskBase.__call__(self, *args, **kwargs)
2017-11-29T00:15:20.728041051Z app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 629, in __protected_call__
2017-11-29T00:15:20.728046087Z app[worker.1]:     return self.run(*args, **kwargs)
2017-11-29T00:15:20.728050935Z app[worker.1]:   File "/app/server/scripts/tasks.py", line 85, in get_facebook_posts_per_user
2017-11-29T00:15:20.728055996Z app[worker.1]:     posts = _get_facebook_posts(user)
2017-11-29T00:15:20.728061184Z app[worker.1]:   File "/app/server/scripts/tasks.py", line 98, in _get_facebook_posts
2017-11-29T00:15:20.728065980Z app[worker.1]:     friends_likes = _get_facebook_friends_and_likes(user)
2017-11-29T00:15:20.728070737Z app[worker.1]:   File "/app/server/scripts/tasks.py", line 134, in _get_facebook_friends_and_likes
2017-11-29T00:15:20.728075503Z app[worker.1]:     friends_likes[key].extend(initial_result[key]['data'])
2017-11-29T00:15:20.728079894Z app[worker.1]: KeyError: 'friends'

Gobo unable to retrieve author information for FB posts to page walls

Currently, FB posts to page walls look like this:
screenshot from 2018-01-09 11-52-33

Need to figure out how to grab the post author's username and photo for this specific post case.

Lines of interest are:

let source;
if (post.source === 'twitter') {
source = content.user.name;
}
else if (post.content.from) {
source = post.content.from.name;
}
if ((post.source === 'facebook') && (content.post_user) && (source !== content.post_user.name)) {
source += ` β–Ά ${post.content.post_user.name}`;
}
return source;

let picSrc = '';
if (source === 'twitter') {
picSrc = content.user.profile_image_url_https;
} else {
picSrc = (content.from && content.from.picture) ? content.from.picture.data.url : '';
}

re-architect content fetching

It is too slow the way we do it now (#29).
Perhaps something like this could work:

  • add a last_twitter_fetch and last_facebook_fetch timestamp columns to User
  • a new add_incremental_tasks_to_queue that queues up tasks for the last ~30 users, sorted by last_twitter_fetch DESC (ie. people who haven't been updated in a while) and filtered for users that haven't been updated in the last 4 hours (same for last_facebook_fetch)
  • run add_incremental_tasks_to_queue every 5 minutes on cron

This needs some thoughtful design and testing to see if it would work.

better signup political affiliation text

Lets tweak the screen where we ask folks their political affiliation. How does this sound?

"

Almost done...

News Preferences

To tailor your feed, tell us a little about what type of news you read. Scan the names of popular news sites below and click on the one you read most. This will help us tailor the news filter that we let you control.

[ Huffington Post, MSNBC, Vox ]

[ NYTimes, BuzzFeed, Time ]

[ The Hill, ABC News, Business Week ]

[ Examiner, National Review, US Chronicle ]

[ Breitbart, Daily Caller, Fox News ]

[ Back ]
"

Registration political affiliation

it's currently overwhelming and make the user believe they should choose more than one.

ideas:
slider - put yourself on a scale
change wording - write it more straightforward - actually say left/ right etc.

add access code in from of registration

for the closed beta Ethan wants it to be behind a magic password. Ie. when you click register it has a popup that says
"
Closed Beta
Gobo is currently testing with a select group of people (we called this a "closed beta"). If you have the access code that lets you create an account, enter it below and then you can register. If you don't have the access code, check back in early November!
"
That should have a text field and a "register" button under that. When they hit "register" it hits the server the access code they entered, verifies it is correct, and then shows them to the register page. If it is incorrect then they get shown a message next to the text field saying "incorrect access code".

I guess the access code should be in the deploy config file?

worker jobs not running frequently

I set up a flower monitor for our redis queue and the jobs are running super slowly - just one job every two minutes or so! :-(

flower

At first I thought maybe some service was slow - but this chart suggests the job ran quickly, but took forever to get pulled off the queue to be executed, right?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.