mozilla / makeapi Goto Github PK

View Code? Open in Web Editor NEW

30.0 43.0 33.0 1.15 MB

An API for Exposing User Created Web Content

Home Page: https://makeapi.webmaker.org

License: Mozilla Public License 2.0

JavaScript 88.59% HTML 7.62% CSS 3.79%

makeapi's Introduction

This project is no longer under active development

MakeAPI (core)

The MakeAPI is a node.js based service for storing and exposing metadata about user created web content, called "makes". It provides consumers with an API to Create, Update, Delete, and Search for metadata about a make.

NOTE: This README assumes that you have all the required external dependencies installed and have a working dev environment. New to Webmaker? Make sure to read our developer guide for everything you need in order to get started!

1. Installing & Running the Server

Dependencies

MongoDB
- Mac OS X: Run this with mongod
- Ubuntu-based linux systems.: Run this with sudo service mongodb start
ElasticSearch
- Mac OS X: Run this with elasticsearch -f
- Ubuntu-based linux systems.: Run this with sudo service elasticsearch start
Login server

Installation

Clone the git repository with git clone https://github.com/mozilla/MakeAPI.git
Execute npm install in the application directory.
In the root directory of the application, copy env.sample to a new file called .env

NOTE: The .env file contains settings for various aspects of the MakeAPI server's operation. The default settings should be enough for local development. For non-standard configurations, you may need to adjust where variables point to match the locations of your external dependancies. See the ENV file reference for more details.

Running the Node Server

Ensure all external dependencies are running
Run node server.js from the root of the MakeAPI application

By default the server will run at http://localhost:5000. You can change this by adding PORT= to your .env file.

Clearing make data

Clear elastic search:

curl -XDELETE "<URL of ElasticSearch Cluster>/<ElasticSearch Index Name>"

Find your mongo files and clear them.

In a terminal run mongo add the --host <YOUR_MONGO_HOST> flag to connect to a remote mongo service
From the mongo shell run use makeapi to switch to the makeapi index
Run db.dropDatabase() to remove all existing make data and API keys from mongoDB

2. API Documentation

API documentation can be found at https://mozilla.github.io/makeapi-docs/

The makeapi-client JavaScript library can be used to facilitate interaction with the API in web browser and node contexts. Documentation for the client library can be found at http://mozilla.github.io/makeapi-docs/client-docs/

The Make API Does not sanitize Data it receives or outputs, so it is up to consumer applications to sanitize data appropriately.

3. Resources

Env Variable Reference Section

All the environment variables are listed and detailed here: https://mozilla.github.io/makeapi-docs/#configuration

4. Testing

How to test

We use a combination of technologies to "lint" and test our CSS and JavaScript code. These tests must pass in order for a pull request to be merged into the Mozilla repository. To run them locally,

Navigate to the root folder of the MakeAPI server
Run npm test

TravisCI

When a pull request is made to the Mozilla repository, it is automatically scheduled for testing on the Travis-CI continuous-integration platform. This verifies that the code passes linting requirements as well as all of its unit tests. You can see the status of these tests on the Github page for the pull request, and on the MakeAPI travisCI page.

Updating tests

Most developers won't need to update the tests, but changes to certain parts of the MakeAPI require that the tests be revised. Keeping these tests accurate is essential for easy maintenence of this code base, and pull requests that change these parts will be rejected without proper unit tests.

If you need help understanding the unit tests, hop on the #webmaker IRC channel and we'll be happy to help! No idea what IRC is? Check out our IRC guide.

5. Accessing the MakeAPI with your service

The MakeAPI uses HAWK to authenticate MakeAPI calls. Hawk is an HTTP authentication scheme that can be used to verify the authenticity of messages using cryptographically generated message authentication codes (MACs). Hawk does not provide Transport Layer security, and only serves to identify the sender and verify message integrity.

All applications that wish to make authenticated calls must be issued a pair of keys to sign all requests. Keys are optional for search requests.

Authentication in a dev environment

For convenience of development and testing, the USE_DEV_KEY_LOOKUP variable can be set to true in the environment file. This flag will use a DEVELOPMENT ONLY strategy when verifying keys.

When development key mode is enabled, clients can sign their requests by passing hawk `"00000000-0000-0000-000000000000" as their public and private key. Any other key combination will fail to authenticate.

DO NOT USE DEV KEYS OUTSIDE OF A DEVELOPMENT ENVIRONMENT!

Generating keys for production or staging servers w/ `generateKeys.js`

The generation of keys can be done using the generateKeys script.

The script is called with two arguments: an email address to associate with the keys and a integer indicating the number of pairs to be generated. Generated keys are added to the database and then outputted to the console.

Generating keys in the admin console

There is a tool in the Admin Make Editor that generates keys. It can be reached by visiting the /admin page of the MakeAPI in your browser. You must have a Webmaker admin account on the login server that the MakeAPI is using for authentication.

6. Metrics and logging

The MakeAPI server uses a number of technologies, like STATSD and New Relic, to optionally collect and analyze useful performance data. For local development, this shouldn't be a concern.

For more information on configuring the MakeAPI server's New Relic module, see: https://github.com/newrelic/node-newrelic/#configuring-the-agent

makeapi's People

Contributors

Stargazers

Watchers

makeapi's Issues

PUT /api/make/:id Must only update if __v is the same as current version in DB

This implements optimistic locking.

Difficult to use

I'm finding it hard to consume the MakeAPI.

Here's what I'd like to do:

https://makeapi.webmaker.org/api/makes/search?q=card

The current API seems way overkill.

Documentation insufficient to understand the toolchain

There is no high level documentation that explains "really" what this tool can do and more important what it cannot do that you might expect from the description "An API for Exposing User Created Web Content" (either here on wiki or on http://cadecairos.github.io/makeapi-docs/ ) . Specifically what does "exposing" mean, and are there any limitations on the source location. What does it output. At the moment I can only guess what this might be useful for from the API docs.

Would be good to have a simple explanation (with diagrams) of the different parts of the solution and perhaps a "worked example" showing a source web and its output documentation.

Implement Automated Testing

Implement testing framework
Pick two recent or open bugs
- Bug: ____
- Bug: ____
implement tests that validate fixes for two open bugs
Document how to add a new test
Set up project on TravisCI
Add Travis status Image to README
Add mofo-style linter
Ensure all tests are passing

MozillaFoundation/plan#351

update make route is artificially inflating make likes

If the admin interface is used to edit a make, it accidentally adds a bogus like to the make.

STR:

visit '/admin'
select a make, open it in a new tab and view it's details. note the like count.
edit the description
check the make details again!

GET /api/20130724/make/search?id=MY_ID throws error

Sending a request like so:

GET /api/20130724/make/search?id=MY_ID

Will result in "Check your arguments" error in lib/queryBuilder/queryTypes/search.js

The error is from these lines:

module.exports = function (query, callback, authenticated) {
  if (!(query && typeof query.constructor === Object) || !(callback && typeof callback === "function")) {
    throw new Error("Check your arguments.");
  }
...

The query variable comes from req.query and is an object like this:

{ id: '55c4894253329be03f000003', [__proto__]: {} }

So typeof query == 'object' will be true, but query.Constructor === Object wil not.

Is this the expected behaviour or a bug?

I'm using express 3.4.5

Export user's makes into JSON hosted on S3

As a part of archiving user's makes:

For each user, create a JSON file with all of the user's makes, upload file to S3 in a user-space URL. (e.g. /archivedMakes/{user_id}.json

{
  makes: [
    {
      email: "[email protected]",
      url: "http://notarealurl.com/make_asdf.html",
      contentType: "application/x-applicationName",
      title: "title of make",
      contenturl: "http://realMakeURL.com/notbehindaproxy",
      locale: "en_CA",
      description: "description of make",
      thumbnail: "http://fakeimageURL.com/img123.gif",
      author: "anonymous person",
      tags: [
          "foo",
          "bar"
      ],
      remixedFrom: "OBJECT-ID-OF-ANOTHER-MAKE"
    },
    ...
  ]
  total: n
}

TODO: Add Mongoose Mongodb-connection error handling

If the mongoose's connection to the MongoDB fails, it should do as loudly as possible.

Mongoose error handling takes the form:

mongoose.connection.on("error", errorHandleCB);

Create a route for restoring makes

POST //make/restore/:id

in support of MozillaFoundation/plan#240

Create routes for adding/removing tags

The handling of app tags is insufficient for the new firehose: mozilla/webmaker-firehose#1

patch en route

also relevant: MozillaFoundation/plan#240

It should be possible to search for text in title OR description, ANDed with other fields

It appears to be possible to universally AND/OR search terms. However, in http://weblitmapper.webmakerprototypes.org/ I'd like to be able to search for the text foo in the title OR description, while ensuring that the resource also has specific web literacy tags. Is this possible?

CODE_OF_CONDUCT.md file missing

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please reach out to [email protected].

(Message COC001)

Support an optional "reason" text field when reporting a make.

Add support for a reason when reporting a make as per:
http://mozilla.github.io/makeapi-docs/api-docs/#report-make

Use case:
mozilla/webmaker-android#30

Allow for more granular searches of URLs

In toolness/weblitmapper#22, I'd like to show users a list of other resources tagged on the same domain, but there doesn't seem to be an easy way to do this with the MakeAPI. If there were a urlPrefix query term, analogous to the existing tagPrefix term, that would make it possible... Another alternative would be to actually parse the URLs and index them by full domain name and/or TLD.

Wiki changes

FYI: The following changes were made to this repository's wiki:

defacing spam has been removed
Restricting write access to contributors is strongly encouraged. Please make that change (documentation).

These were made as the result of a recent automated defacement of publically writeable wikis.