Git Product home page Git Product logo

scoreboard's Introduction

Scoreboard πŸ—ΊπŸŽ―

Engaging and encouraging mappers of OpenStreetMap by rewarding contributions, promoting campaigns, and reporting current mapper statistics

πŸ”¨ Development

Dependencies

  • Node 12.14.x
  • Postgres 9.5+
  • Yarn
  • Docker (for development work)

To set the Node version as specified above:

 $ brew install nvm
 $ nvm use

See this tutorial for details and instructions for installing PostgreSQL on your operating system

Services

Scoreboard depends on three external services/APIs for statistics and functionality:

  • OSMesa is a toolkit for processing large amounts of OSM vector data and turning them into stats (e.g: km of road added by user by country)
  • Scoreboard pulls campaigns from the Tasking Manager and uses the campaign hashtags to pull campaign specific aggregations from OSMesa
  • OSM Teams is a service that allows Scoreboard users to create teams and assign campaigns to a team.

Environment file

Create a .env file by copying the .env.sample file and filling in the required values. The sample file has comments above each possible environment variable.

Install JavaScript Dependencies

 $ yarn

Database Setup

Create and run the database

 $ docker-compose up  # wait for scoreboard_wait-for-db-ready_1 

This command will create and run the database on port 5433. The database files are stored under .tmp folder.

If you are running the command for the first time, you should wait until the databases are created. This can take a few minutes.

To start with a new database with no data, stop the command, remove the .tmp folder and run the command again.

Run database migrations

Set up the database schema by running:

 $ yarn migrate

You can rollback migrations with yarn rollback.

Populate database

For the frontend and the API to work, you must have the data loaded on the local database.

To generate fake user and campaign data run:

 $ yarn seed

Create an admin user

To be able to develop admin pages you'll need to have an admin user. Make your user an admin by running:

./api/bin/scoreboard create-admin-user --username {your osm username}

If you were already logged in, log out and log back in, then you'll see an "admin" link in the menu in the top right.

Known issue: before running create-admin-user for username, you first have to login to the web app with that username first.

Add some edits to your user

It could be useful to give some edits to your user in development.

You can run:

./api/bin/scoreboard create-edits --osm-id {your osm id}

This will select some random changesets from the OSMesa edits and assign them to your osm id

Start Development Server

This command will start both the frontend and the api together, with livereloading for both the front end and api

 $ yarn dev

This command will start both the frontend and the api together, with a static version of the api $ yarn dev-static

Tests

Tests for the API endpoints and related code are stored in api/tests/ and UI tests are stored in tests/. To run both:

$ yarn test

Test the UI and API separately using yarn test-ui and yarn test-api.

Linter

Lint the code using:

 $ yarn lint

You can automatically fix some lint errors using yarn lintfix.

Generating Docs

If you make changes to the API documentation make sure to update the README for the API which is auto-generated from the swagger YAML by running:

 $ yarn docs

Connecting to OSM Teams

In OSM-teams, choose "Connect a new app". For the callback URL use the path /auth/teams/accept

For example in development, Scoreboard runs on localhost:8181, so the callback URL would be http://localhost:8181/auth/teams/accept .

The client id and client secret should be saved to your Scoreboard .env file.

Scoreboard will work with a single organization at the moment. In osm-teams, record your organization's ID from the API and use the environment variable OSM_TEAMS_ORG_ID to have scoreboard read and write to the organization's teams.

If you are setting up local development for the first time, you may need to create an organization. There is currently no way to do this via the osm teams interfaces. Instead you can use the swagger UI to make the requests.

  1. check to see what organizations exist on your local osm-teams
  2. if there is none, create an organization (the id can be anything, but should match OSM_TEAMS_ORG_ID in your scoreboard .env)
  3. run yarn seed
  4. clear your cache and login

Administrator Settings

The Administrator Settings are found in the Admin user's dashboard, or via the Admin entry in the User menu.

  • Tasking Managers: Currently HOT Tasking Manager (TM2 - TM3) is supported.
  • Setup Badges
  • Manage Users
  • App Settings: OSMesa integration settings

Simulated osmesa database

A pg_dump of osm-fabricator is used to mock the osmesa db when developing locally.

To update the osmesa db, first develop on osm-fabricator, then pg_dump it into docker-context/osmesa-mock/init.sql. See scoreboard's docker-compose.yml for more details.

API docs

See api/README

Production

To use scoreboard in production follow these steps:

  1. Make sure you have all the requirements.
  2. Install dependencies: yarn
  3. Setup a production Postgres database
  4. Create the database by running these commands.
  5. Setup an .env file with correct values -> example.
    • make sure to set NODE_ENV=production in your .env file
  6. Run the migrations: npm run migrate
  7. Add initial data to the database (skip if the database already has data) npm run seed
  8. [OPTIONAL] Preload list of users npm run users
  9. Populate stats: npm run clocks
  10. [OPTIONAL] Assign admin users by running ./api/bin/scoreboard create-admin-user --osm-id {your-id}
    • Note:
  11. Build: npm run build
  12. Launch the server: node server.js

scoreboard's People

Contributors

alyssadelaine avatar auroranou avatar dependabot[bot] avatar jvntf avatar kamicut avatar lanesgood avatar maxgrossman avatar mojodna avatar scisco avatar sethvincent avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

scoreboard's Issues

Split into seperate repos

We'll likely need to create separate repos rather than have the api and frontend in one repo.

I'd probably split them into:

  • scoreboard-api
  • scoreboard-ui
  • scoreboard-deploy

Install on node v10 fails

It looks like the sqlite3 package isn't installing. We also are no longer using sqlite3 so this should be an easy fix, but not urgent as we're still targeting node v8.

Error from npm i:

``` node-pre-gyp ERR! Tried to download(403): https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v3.1.13/node-v64-darwin-x64.tar.gz node-pre-gyp ERR! Pre-built binaries not found for [email protected] and [email protected] (node-v64 ABI) (falling back to source compile with node-gyp) node-pre-gyp ERR! Tried to download(undefined): https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v3.1.13/node-v64-darwin-x64.tar.gz node-pre-gyp ERR! Pre-built binaries not found for [email protected] and [email protected] (node-v64 ABI) (falling back to source compile with node-gyp) gypgyp WARN downloadWARN NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR gyp gypWARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR ACTION deps_sqlite3_gyp_action_before_build_target_unpack_sqlite_dep Release/obj/gen/sqlite-autoconf-3150000/sqlite3.c ACTION deps_sqlite3_gyp_action_before_build_target_unpack_sqlite_dep Release/obj/gen/sqlite-autoconf-3150000/sqlite3.c TOUCH Release/obj.target/deps/action_before_build.stamp TOUCH Release/obj.target/deps/action_before_build.stamp CC(target) Release/obj.target/sqlite3/gen/sqlite-autoconf-3150000/sqlite3.o CC(target) Release/obj.target/sqlite3/gen/sqlite-autoconf-3150000/sqlite3.o rm: ./Release/.deps/Release/obj.target/sqlite3/gen/sqlite-autoconf-3150000/sqlite3.o.d.raw: No such file or directory make: *** [Release/obj.target/sqlite3/gen/sqlite-autoconf-3150000/sqlite3.o] Error 1 gyp ERR! build error gyp ERR! stack Error: `make` failed with exit code: 2 gyp ERR! stack at ChildProcess.onExit (/Users/sdv/.nvm/versions/node/v10.13.0/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:262:23) gyp ERR! stack at ChildProcess.emit (events.js:182:13) gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:240:12) gyp ERR! System Darwin 18.2.0 gyp ERR! command "/Users/sdv/.nvm/versions/node/v10.13.0/bin/node" "/Users/sdv/.nvm/versions/node/v10.13.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "build" "--fallback-to-build" "--module=/Users/sdv/tmp/scoreboard/node_modules/sqlite3/lib/binding/node-v64-darwin-x64/node_sqlite3.node" "--module_name=node_sqlite3" "--module_path=/Users/sdv/tmp/scoreboard/node_modules/sqlite3/lib/binding/node-v64-darwin-x64" gyp ERR! cwd /Users/sdv/tmp/scoreboard/node_modules/sqlite3 gyp ERR! node -v v10.13.0 gyp ERR! node-gyp -v v3.8.0 gyp ERR! not ok node-pre-gyp LIBTOOL-STATIC Release/sqlite3.a ERR! build error node-pre-gyp ERR! stack Error: Failed to execute '/Users/sdv/.nvm/versions/node/v10.13.0/bin/node /Users/sdv/.nvm/versions/node/v10.13.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --fallback-to-build --module=/Users/sdv/tmp/scoreboard/node_modules/sqlite3/lib/binding/node-v64-darwin-x64/node_sqlite3.node --module_name=node_sqlite3 --module_path=/Users/sdv/tmp/scoreboard/node_modules/sqlite3/lib/binding/node-v64-darwin-x64' (1) node-pre-gyp ERR! stack at ChildProcess. (/Users/sdv/tmp/scoreboard/node_modules/sqlite3/node_modules/node-pre-gyp/lib/util/compile.js:83:29) node-pre-gyp ERR! stack at ChildProcess.emit (events.js:182:13) node-pre-gyp ERR! stack at maybeClose (internal/child_process.js:962:16) node-pre-gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:251:5) node-pre-gyp ERR! System Darwin 18.2.0 node-pre-gyp ERR! command "/Users/sdv/.nvm/versions/node/v10.13.0/bin/node" "/Users/sdv/tmp/scoreboard/node_modules/sqlite3/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build" node-pre-gyp ERR! cwd /Users/sdv/tmp/scoreboard/node_modules/sqlite3 node-pre-gyp ERR! node -v v10.13.0 node-pre-gyp ERR! node-pre-gyp -v v0.6.38 node-pre-gyp ERR! not ok Failed to execute '/Users/sdv/.nvm/versions/node/v10.13.0/bin/node /Users/sdv/.nvm/versions/node/v10.13.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --fallback-to-build --module=/Users/sdv/tmp/scoreboard/node_modules/sqlite3/lib/binding/node-v64-darwin-x64/node_sqlite3.node --module_name=node_sqlite3 --module_path=/Users/sdv/tmp/scoreboard/node_modules/sqlite3/lib/binding/node-v64-darwin-x64' (1) CXX(target) Release/obj.target/node_sqlite3/src/database.o3~install: Failed to exec install script CXX(target) Release/obj.target/node_sqlite3/src/node_sqlite3.o CXX(target) Release/obj.target/node_sqlite3/src/statement.o~install: Failed to exec install script clangclang: error: no such file or directory: '../src/node_sqlite3.cc' Failed to exec install script : clang: errorerror: : no input filesno such file or directory: '../src/statement.cc'

clang: error: no input files
clang: error: no such file or directory: '../src/database.cc'
clang: error: no input files
make: *** [Release/obj.target/node_sqlite3/src/node_sqlite3.o] Error 1
make: *** Waiting for unfinished jobs....
make: *** [Release/obj.target/node_sqlite3/src/statement.o] Error 1
make: *** [Release/obj.target/node_sqlite3/src/database.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: make failed with exit code: 2
gyp ERR! stack at ChildProcess.onExit (/Users/sdv/.nvm/versions/node/v10.13.0/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:262:23)
gyp ERR! stack at ChildProcess.emit (events.js:182:13)
gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:240:12)
gyp ERR! System Darwin 18.2.0
gyp ERR! command "/Users/sdv/.nvm/versions/node/v10.13.0/bin/node" "/Users/sdv/.nvm/versions/node/v10.13.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "build" "--fallback-to-build" "--module=/Users/sdv/tmp/scoreboard/node_modules/sqlite3/lib/binding/node-v64-darwin-x64/node_sqlite3.node" "--module_name=node_sqlite3" "--module_path=/Users/sdv/tmp/scoreboard/node_modules/sqlite3/lib/binding/node-v64-darwin-x64"
gyp ERR! cwd /Users/sdv/tmp/scoreboard/node_modules/sqlite3
gyp ERR! node -v v10.13.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok
npm WARN [email protected] requires a peer of preact@* but none is installed. You must install peer dependencies yourself.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: node-pre-gyp install --fallback-to-build
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /Users/sdv/.npm/_logs/2018-11-28T20_41_26_434Z-debug.log

<details>

Intermittent db connection error on CI

I get this error on and off:

  Rejected promise returned by test. Reason:

  Error {
    message: 'ECONNRESET: Connection reset by peer',
  }

  Test.assert (node_modules/supertest/lib/test.js:165:15)
  localAssert (node_modules/supertest/lib/test.js:131:12)
  node_modules/supertest/lib/test.js:128:5
  Test.Request.callback (node_modules/superagent/lib/node/index.js:728:3)
  ClientRequest.req.once.err (node_modules/superagent/lib/node/index.js:647:10)

error Command failed with exit code 1.

here's an example: https://circleci.com/gh/developmentseed/scoreboard/259

Tests will run fine locally and it seems like 50% of the time will fail on CI.

Create tasks database table

Draft schema:

user ID of assignee
Campaign ID
user ID of assigner

This will have a many:1 relationship with the users table and a many:many relationship with the campaigns table

"Last Edit" granularity is misleading

getLastEdit in

function getLastEdit (edit_times) {
const days = edit_times.map(time => parse(time.day));
const lastEdit = head(days.sort(compareDesc));
return `${distanceInWordsToNow(lastEdit)} ago`
}
uses the list of edit_times to surface the most recent edit.

edit_times in the API response (api/users/<id>) only includes dates (no timestamps), so this will be misleading if the last edit is in the window for which distanceInWordsToNow renders dates in granularity smaller than days:

{
  // ...
  "records": {
    edit_times: [
      // ...
      {
        "day": "2018-09-21",
        "count": 2
      }
    ]
    // ...
  }
}

Compare to

<td>{user.edit_count > 0 ? `${distanceInWordsToNow(parse(user.last_edit))} ago`: "N/A"}</td>
, which uses the api/users endpoint and has access to a last_edit timestamp represented as epoch milliseconds.

Unlink Tasking Manager

I think that tasking manager should not be coupled to scoreboard tightly. We should move towards the concept of "Adding a task" and we could add tasks from multiple sources, such as a TM2 or a TM3 instance. These tasks could then be assigned to teams.

Should docker be required for development?

#20 adds docker as a dependency for development. While I understand using docker for less friction in deployment, my opinion is that it should be optional for development work.

Should the documentation contain instructions for setting up a database without docker?

Create tasks api route

With the following methods available:

  1. GET: given user ID, returns all tasks [self-]assigned to a user
  2. POST: given user ID and project ID, adds that project to that user's to-do list*
  3. DEL: given user ID and project ID, removes that project from that user's to-do list

* Later, this will also take an additional assigning-user ID so that we can allow team admins to assign tasks to team members

NOTE: Tasks here means assigned projects, not individual task squares

daysInRow returns the total number of mapping days

The calculation of daysInRow has two parts:

  1. sequentializeDates creates an array containing an array for every sequence of consecutive dates
  2. findLongestStreak finds the longest array and returns the length

An error is being introduced in step 1, because all days are being put into a single array. Thus the array of arrays passed to findLongestStreak is just an array containing a single array, and the length of that array is the the total number of days mapped.

So that, rather than the highest number of consecutive days mapped, is being returned as daysInRow

Move tasks out of the database without losing site responsiveness

I have started a branch which contains most of the code to move tasks out of the database, but it needs some improvements, namely

  1. Caching the TM 2 responses since the 100-item page size makes calls to that API slow
  2. Pulling all projects from TM 3. This code just pulls 14 or so results at a time -- though this will likely be fixed in another branch before we return to this one

Yarn test returns an error when running the database in docker

I am able to successfully run tests when I run the database locally. When I run it with docker, I get the following:

lerna ERR! yarn run test stderr:
createdb: could not connect to database template1: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
dropdb: could not connect to database template1: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
error Command failed with exit code 1.

lerna ERR! yarn run test exited 1 in 'scoreboard-api'
error Command failed with exit code 1.

API lacks CORS headers

After changing

baseURL: '/scoreboard'
to point to http://localhost:5000/scoreboard, the front-end is still unable to load any data due to a lack of CORS headers.

Proposed change:

diff --git i/api/package.json w/api/package.json
index b503fb1..f92bc3e 100644
--- i/api/package.json
+++ w/api/package.json
@@ -25,6 +25,7 @@
     "@heroku/foreman": "git+https://github.com/heroku/node-foreman.git",
     "body-parser": "^1.18.3",
     "compression": "^1.7.1",
+    "cors": "^2.8.4",
     "express": "^4.16.3",
     "express-boom": "^2.0.0",
     "express-promise-router": "^2.0.0",
diff --git i/api/src/index.js w/api/src/index.js
index d8b9cdf..7dad586 100644
--- i/api/src/index.js
+++ w/api/src/index.js
@@ -4,6 +4,7 @@
 const express = require('express')
 const bodyParser = require('body-parser')
 const boom = require('express-boom')
+const cors = require('cors')
 const session = require('express-session')
 const compression = require('compression')
 const YAML = require('yamljs')
@@ -24,6 +25,7 @@ const { passport, authRouter } = require('./passport')
 const swaggerDocument = YAML.load('./docs/api.yml')
 swaggerDocument.info.version = pckg.version
 
+app.use(cors())
 app.use(bodyParser.json())
 app.use(compression())
 app.use(boom())

Rewrite frontend APP_URL

I am reposting from #38

This line in https://github.com/developmentseed/scoreboard/blob/master/frontend/src/containers/Dashboard.js#L141 (and others around the app) does not make sense when hosting a public version of scoreboard. We should rewrite localhost:5000 either:

  • Rewrite APP_URL on the server and render the page on the server (this is the simplest logic)
  • Create a frontend REACT_APP_URL that replaces this at build time (this is the simplest fix, but requires two environment variables, APP_URL on the server and REACT_APP_URL on the frontend) cc @sethvincent

'yarn build' error

The following error results from trying to run yarn build, however yarn dev runs fine

SyntaxError: /scoreboard/pages/_app.js: Unexpected token (65:6)

return (
    <div className='App'>
    ^

use store & actions for api requests

I started using unistore for async actions in the dashbaord.

Let's start using it in other components/containers as well.

Sidenote: unistore is almost exactly the same as redux, but requires less boilerplate code and is a smaller file size. It's pretty cool.

Usage is the same, except actions and reducers are simplified down to just actions, and there's no need to use middleware to do async actions. Moving to redux would be easy, but probably not necessary.

Improve error reporting

We have a number of places in the UI where we need to provide better errors based on responses from APIs.

Tasks in Scoreboard

Tasks in Scoreboard

Task Attributes

For each page of tasks pulled from the API, the following will be displayed:

β€’ Name of task
β€’ Percent completed
β€’ Percent validated
β€’ Priority

The following will also be used and/or provided:

β€’ GeoJSON
β€’ Link to task

Note that for each of the first four attributes, we will need to make a separate API call since the only attributes provided in the basic search is the Project ID and priority.

For now, we will follow the task manager's example of:

β€’ Showing the most urgent projects by default
β€’ Displaying 14 results per page
β€’ Allowing users to search projects by name, priority, mapper level

Open Questions

β€’ What other data do we need to pull for the project page (as opposed to the card)?

User login

Right now scoreboard has a list of users with metadata and renders read-only users. We will be moving scoreboard to a login system. This has a few implications:

  • Can stats be viewed by non logged in users?
  • Are there different user permissions?
  • Do we need to calculate stats for users that have never logged in?
  • Do we have a public vs private profile where certain stats are only available to the logged in user?

It would be good to talk through these issues @ricardomestre @geohacker @scisco @alyssadelaine

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.