kinto / kinto Goto Github PK

View Code? Open in Web Editor NEW

4.3K 88.0 418.0 30.54 MB

A generic JSON document store with sharing and synchronisation capabilities.

Home Page: http://docs.kinto-storage.org/

License: Other

Makefile 0.44% Python 96.01% Smarty 0.64% PLpgSQL 2.70% HTML 0.07% Dockerfile 0.10% Shell 0.04%

api storage web http decentralisation

kinto's Introduction

Kinto

Kinto is a minimalist JSON storage service with synchronisation and sharing abilities.

Requirements

Python: 3.8+
Backends: In-memory (development), PostgreSQL 9.5+ (production)

kinto's People

Contributors

Stargazers

Watchers

Forkers

fil jotes magopian nopressurelabs phrawzty algogator s-utsch ebrehault jschneiders oak11 clouserw smorele crudbug volksport rektide tinkerwell cgiogkarakis htk cdfpaz ezhangle weareburo gvdp-travix fnet123 fysoft2006 pirater jango2015 fengshao0907 407390911 williamren rohitpal rakesh-mohanta kelur-hawkingrei conlini fengweijp snehaparkar thefon duanshuaimin iexperi tomzhang sxhao johngian is00hcw qinwenshi shalomabitan tfroehlich82 aegaas chrisckchang luisj135 cscg viktor-evdokimov mikesavior perpetual-hydrofoil selfey rajularora taylorzane desmond0321 atteeela rlugojr williamhoang waprin kaiquewdev cooliotseng ceedee4u imjerrybao ahmedfawzy bradparks repos-parse-alternatives vfulco dieface matildahope matrixy neuroradiology johnhnorton nadeemkhan janstk michielbdejong whollacsek niagaraso tml yannski lguzzon-scratchbook acemaster mathiasbc fpauer puckjam mjsorribas khoden nyimbi ayusharma magic-coder jstoja lavish205 woovar rodnandes vsham20 sayli-karnik ipsha21 pokleh happy-tanuki alfiyazi

kinto's Issues

Allow underscore in object's id

Explicit settings available in hello view

As pointed out by @Natim in Kinto/kinto.js#55, it would be relevant to mention the settings available in the hello view, in Kinto documentation (not just Cliquet)

Do we want to force record's id to be UUID?

For now Cliquet ressources can be any kind of objects.

In the scope of kinto, do we want to say that a record id must be a UUID?

<ip>:<port>/v0/ is redirected to /v1/v0 on 1.1.0

Latest master successfully installed and running locally (python 2.7.6, osx):

$ curl -I http://0.0.0.0:8888/v0
HTTP/1.1 307 Temporary Redirect
Content-Length: 205
Content-Type: text/html; charset=UTF-8
Date: Tue, 30 Jun 2015 08:07:26 GMT
Location: http://localhost:8888/v1/v0
Server: waitress

Server logs:

$ make serve
/Users/niko/Sites/kinto/.env/bin/cliquet --ini config/kinto.ini migrate
INFO:venusian:cloud storage 1.2.0.dev0 starting.
/Users/niko/Sites/kinto/.env/bin/pserve config/kinto.ini --reload
Starting subprocess with file monitor
2015-06-30 10:05:04,876 INFO  [venusian][MainThread] cloud storage 1.2.0.dev0 starting.
Starting server in PID 18692.
serving on http://0.0.0.0:8888
2015-06-30 10:05:22,206 INFO  [venusian][Dummy-2] "GET   /v0/" 307 (2 ms) request.summary lang=en-US,en;q=0.7,fr-FR;q=0.3; uid=None; errno=None; agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:40.0) Gecko/20100101 Firefox/40.0; authn_type=None; time=2015-06-30T10:05:22
2015-06-30 10:05:22,236 INFO  [venusian][Dummy-3] "GET   /v1/v0/" 404 (1 ms) request.summary lang=en-US,en;q=0.7,fr-FR;q=0.3; uid=None; errno=111; agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:40.0) Gecko/20100101 Firefox/40.0; authn_type=None; time=2015-06-30T10:05:22

We should consider hidding resources permissions when we only have the read permission

This is related to #88

When we have the read permission to list existing buckets, we can ask for a bucket and in that case we can list its permissions. I think it is only something that somebody that can modify this permission should be able to see.

Should we return 409 instead of 200 for POST if specified id already exists ?

As per Kinto/kinto.js#14 I tried to raise a 409 on a POST.

The behavior is not the one I was expected:

$ echo '{"data": {"title": "Test"}}' | http --verify no POST https://kinto.dev.mozaws.net/v1/buckets/default/collections/tasks/records --auth natim:HTTP/1.1 201 Created
Access-Control-Expose-Headers: Backoff, Retry-After, Alert
Backoff: 10
Connection: keep-alive
Content-Length: 202
Content-Type: application/json; charset=UTF-8
Date: Wed, 15 Jul 2015 09:38:23 GMT
Server: nginx/1.4.6 (Ubuntu)

{
    "data": {
        "id": "1b719cfb-ff5d-4b86-baf6-1e204245a999", 
        "last_modified": 1436953103016, 
        "title": "Test"
    }, 
    "permissions": {
        "write": [
            "basicauth:df93ca0ecaeaa3126595f6785b39c408be2539173c991a7b2e3181a9826a69bc"
        ]
    }
}

$ echo '{"data": {"id": "1b719cfb-ff5d-4b86-baf6-1e204245a999", "title": "Toto"}}' | http --verify no POST https://kinto.dev.mozaws.net/v1/buckets/default/collections/tasks/records --auth natim:
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Backoff, Retry-After, Alert
Backoff: 10
Connection: keep-alive
Content-Length: 202
Content-Type: application/json; charset=UTF-8
Date: Wed, 15 Jul 2015 09:43:13 GMT
Server: nginx/1.4.6 (Ubuntu)

{
    "data": {
        "id": "1b719cfb-ff5d-4b86-baf6-1e204245a999", 
        "last_modified": 1436953103016, 
        "title": "Test"
    }, 
    "permissions": {
        "write": [
            "basicauth:df93ca0ecaeaa3126595f6785b39c408be2539173c991a7b2e3181a9826a69bc"
        ]
    }
}

As you can see when I try a POST on an existing ID, I get the existing record with a 200.

I was expecting getting a 409 Conflict error with the stored record in the detail field.

pyramid.route_path to build internal route_path

We are building URIs by hand at multiple places in the code (e.g. parent id, default subrequests) and we should probably use request.route_path() for that.

http://docs.pylonsproject.org/projects/pyramid/en/latest/api/request.html#pyramid.request.Request.route_path

Schemas validation

Let's start a discussion about schema validation !

I believe we want something similar to Daybed. Except that we won't
reinvent the wheel with custom schema formalism.

It looks like there is no way around JSON schema. We won't rewrite thousands
of lines of code using Colander (see https://github.com/Julian/jsonschema).
We will use existing validators, and frontends will build forms from schemas.

By default, collections could be schemaless.

PUT|PATCH|DELETE /collections/person/schema
{
    {
        "title": "Example Schema",
        "type": "object",
        "properties": {
            "firstName": {
                "type": "string"
            },
            "lastName": {
                "type": "string"
            },
            "age": {
                "description": "Age in years",
                "type": "integer",
                "minimum": 0
            }
        },
        "required": ["firstName", "lastName"]
    }
}

It is acceptable if the validation of the definition is done using Cornice/Colander.
Even though there are meta-schemas :)

If a schema exists for a collection, records that don't match the schema are
refused with 400 Bad Request (on PUT and after merge of PATCH).

Unlike Daybed, I propose that each user owns the schema of the collection.
Especially because the schema endpoint will probably be a resource :)
Schemas could be cached to avoid overhead of reading it from storage at each incoming record.

This implies that users can store heterogeneous data with the same collection name.

Hence, on first start, the JS application will have to check that the user did
not set any schema (e.g. at /collections/moz:readinglist:articles/schema).
If she did, then confirm to replace it. If she hacks it afterwards and stores
invalid records, the JS application may crash, but that's okay.

Open questions

Is the schema using the resource code of Cliquet ? If so, how do avoid overlap of stored collections ? We could use underscores prefix, and prevent public collections to have a name that starts with underscore :)
What happen to existing data when schema is changed ? Probably ignore and wait for next
PUT or PATCH ?
What happen when records are shared between users ? Do we let other users
records crash our application when fetching shared records ? We could probably
run client-side validation using JSON schema on shared records.
Do we want to provide a collection of custom formats or even types ?
I'm thinking of recurrent needs for uuid4, geohash, GeoJSON objects, phone, postal codes...

Managing cliquet extra requirements

Cliquet defines some extra requirements:

postgresql, for the backends
monitoring for monitoring tooling (statsd, newrelic)

Currently, we don't pull them in Kinto, because we want it to be straightforward to run (i.e. git clone, make serve). That's why we also switched to the memory backend by default.

But, as a consequence, if someone wants to use Kinto with postgresql, she has to explicit pull cliquet requirements using pip install "cliquet[postgresql]" (see docs).

Ideally, we would like to add a some extra requirements to Kinto to ease that (for example pip install kinto[production]).

Our attempts did not lead us anywhere so far. We need help!

See:

695dde8 : see [:-1]
a17da2a : repeats cliquet underlying deps (not ideal)

Sharing a record (adding read permission) doesn't work

Still following the tutorial, I made Alice add the read permission to Bob on a newly created record. The server answered with a 200 OK, but still the permission wasn't added:

echo '{
    "permissions": {
        "read": ["basicauth:a103c2e714a04615783de8a03fef1c7fee221214387dd07993bb9aed1f2f2148"]
    }
}' | http PATCH http://0.0.0.0:8888/v1/buckets/todo/collections/tasks/records/55793ad7-9387-43c0-a275-7023ba100d2a     -v --auth 'alice:alicepassword'
PATCH /v1/buckets/todo/collections/tasks/records/55793ad7-9387-43c0-a275-7023ba100d2a HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate
Authorization: Basic YWxpY2U6YWxpY2VwYXNzd29yZA==
Content-Length: 126
Content-Type: application/json; charset=utf-8
Host: 0.0.0.0:8888
User-Agent: HTTPie/0.8.0

{
    "permissions": {
        "read": [
            "basicauth:a103c2e714a04615783de8a03fef1c7fee221214387dd07993bb9aed1f2f2148"
        ]
    }
}

HTTP/1.1 200 OK
Access-Control-Expose-Headers: Backoff, Retry-After, Alert
Content-Length: 230
Content-Type: application/json; charset=UTF-8
Date: Sun, 05 Jul 2015 12:56:55 GMT
Server: waitress

{
    "data": {
        "description": "Alice task",
        "id": "55793ad7-9387-43c0-a275-7023ba100d2a",
        "last_modified": 1436100739056,
        "status": "todo"
    },
    "permissions": {
        "write": [
            "basicauth:9be2b51de8544fbed4539382d0885f8643c0185c90fb23201d7bbe86d70b4a44"
        ]
    }
}

Here's a GET on the record:

http GET http://0.0.0.0:8888/v1/buckets/todo/collections/tasks/records/55793ad7-9387-43c0-a275-7023ba100d2a     -v --auth 'alice:alicepassword'
GET /v1/buckets/todo/collections/tasks/records/55793ad7-9387-43c0-a275-7023ba100d2a HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Authorization: Basic YWxpY2U6YWxpY2VwYXNzd29yZA==
Host: 0.0.0.0:8888
User-Agent: HTTPie/0.8.0



HTTP/1.1 200 OK
Access-Control-Expose-Headers: Backoff, Retry-After, Alert, Last-Modified, ETag
Content-Length: 230
Content-Type: application/json; charset=UTF-8
Date: Sun, 05 Jul 2015 13:02:03 GMT
Etag: "1436100903483"
Last-Modified: Sun, 05 Jul 2015 12:55:03 GMT
Server: waitress

{
    "data": {
        "description": "Alice task",
        "id": "55793ad7-9387-43c0-a275-7023ba100d2a",
        "last_modified": 1436100739056,
        "status": "todo"
    },
    "permissions": {
        "write": [
            "basicauth:9be2b51de8544fbed4539382d0885f8643c0185c90fb23201d7bbe86d70b4a44"
        ]
    }
}

Include cliquet-fxa by default

Add it to setup.py deps (optional require ?)
Add it to config/kinto.ini

If we disagree on this, then clean-up config/kinto.ini and remove mentions of fxa

Kinto just needs cliquet as a dependency not cliquet[postgresql,monitoring]

By default Kinto now uses the memory backend.

Define users, organization, collection and permissions API

State of the API

Today, collections are linked to users

When you access /collections/<collection_id>/records you get record for the given collection_idand the connected user uid.

What does permissions implies?

When you start to talk about collection and record permission, it means you will share data between users, it means we need a way to access someone else collection. It also means that collection doesn't necessarily belongs to one user only but can have multiple owners.

In my opinion this is a big shift in the way we currently designed our APIs.

Collections access

Today when you access a collection it is automatically your personal collection: /collections/<collection_id>/records.

In order to share data with someone we should be able to access someone else collections: /users/<uid>/collections/<collection_id>/records

Also having two distinct URLs to access the same data doesn't looks right.

/collections/todo/records is the same as (with 1234 as my user id) /users/1234/collections/todo/records

Introducing buckets

My proposal is to introduce a bucket concept.

A bucket is a set of collections that belongs to one or more users.

{
  "id": "servicedenuages",
  "principals": ["user:[email protected]", "user:[email protected]"]
}

A collection belongs to a bucket and have got the following URI:

/buckets/<bucket_id>/collections/<collection_id>/records

ie: /buckets/servicedenuages/collections/todo/records

To manage bucket users you must be a bucket user and use this resource: PATCH /servicedenuages

New HTTP API

In order to do so, we can define the following resources:

/buckets The list of buckets
/buckets/<bucket_id> The bucket resource
/buckets/<bucket_id>/collections/<collection_id> The collection resource
/buckets/<bucket_id>/collections/<collection_id>/records The records list resource
/buckets/<bucket_id>/collections/<collection_id>/records/<record_id> The record resource
/buckets/<bucket_id>/collections/<collection_id>/permissions The permission resource
/buckets/<bucket_id>/collections/<collection_id>/schema The schema definition resource

The benefit from having bucket is that we let the user defines what does the bucket means.

Is it collection for an app, a user, an organization, a group of users in an app, a device?

Handling permissions on collections

With regards to the permission API, I would like to keep what we did for Daybed: http://daybed.readthedocs.org/en/latest/usage.html#get-back-the-model-permissions

Having a shortcut ALL to add all permissions + to add and - to remove.

Adding Everyone and Authenticated principals to handle Unauthenticated and Authenticated users access.

PATCH /buckets/servicedenuages/collections/example/permissions

{
  "Everyone": ["+create_record"],
  "Authenticated": ["+read_permissions"],
  "user:[email protected]": ["-update_permissions"]
}

Handling permissions on records

If I give this right on the permission:

PATCH /buckets/servicedenuages/collections/example/permissions

{
  "user:[email protected]": ["+read_shared_records"]
}

I also need to update the record:

PATCH /buckets/servicedenuages/collections/todo/records/task_12

{"shared_principals": ["+user:[email protected]"]}

Package with Docker Compose

Running a Kinto instance could be very smooth if underlying redis or postgresql would run as linked containers automatically :)

May require configuration via environment variables in Cliquet, see mozilla-services/cliquet#42

Missing CORS header on /v1/buckets/default/collections/tasks/records

09:19:55,733 Blocage d'une requête multi-origines (Cross-Origin Request) : la politique « Same Origin » ne permet pas de consulter la ressource distante située sur http://0.0.0.0:8888/v1/buckets/default/collections/tasks/records?_since=1436512795672. Raison : l'en-tête CORS « Access-Control-Allow-Origin » est manquant.1

How to remove permissions in child objects from parent?

We've got two things:

Inheritance
Auto write permission on update.

If I have the write permission on a bucket and that I update a records or a collection, my userid will be added to the write ACL on each record or collection I have updated.

If someone wants to remove my permission from this bucket, she will have to also update all the records I have been editing in between.

I propose that we add the write permission to an object only if after applying permission update on the object, we don't have the write permission anymore.

Which means:

We removed the permission we had and it is not a good idea
We only add the "*:create" permission and we need the write permission on this object
If we still have the write permission on the object, it means we inherited from the object permission and we are all set already (so we do not need to add the userid to the write ACL)

Give details about 404 when parent object is missing

With implementation of #48, the message of 404 provides no detail.

Error trying to install latest 1.3.1 from the git tag

Source script

…
.venv/bin/cliquet --ini config/kinto.ini migrate
Traceback (most recent call last):
  File ".venv/bin/cliquet", line 9, in <module>
    load_entry_point('cliquet==2.4.0.dev0', 'console_scripts', 'cliquet')()
  File "/var/tmp/kinto/.venv/local/lib/python2.7/site-packages/cliquet/scripts/cliquet.py", line 48, in main
    env = bootstrap(args.ini_file)
  File "/var/tmp/kinto/.venv/local/lib/python2.7/site-packages/pyramid/paster.py", line 131, in bootstrap
    app = get_app(config_uri, options=options)
  File "/var/tmp/kinto/.venv/local/lib/python2.7/site-packages/pyramid/paster.py", line 31, in get_app
    global_conf=options)
  File "/var/tmp/kinto/.venv/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
    return loadobj(APP, uri, name=name, **kw)
  File "/var/tmp/kinto/.venv/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 271, in loadobj
    global_conf=global_conf)
  File "/var/tmp/kinto/.venv/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
    global_conf=global_conf)
  File "/var/tmp/kinto/.venv/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 320, in _loadconfig
    return loader.get_context(object_type, name, global_conf)
  File "/var/tmp/kinto/.venv/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 454, in get_context
    section)
  File "/var/tmp/kinto/.venv/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 476, in _context_from_use
    object_type, name=use, global_conf=global_conf)
  File "/var/tmp/kinto/.venv/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 406, in get_context
    global_conf=global_conf)
  File "/var/tmp/kinto/.venv/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
    global_conf=global_conf)
  File "/var/tmp/kinto/.venv/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 328, in _loadegg
    return loader.get_context(object_type, name, global_conf)
  File "/var/tmp/kinto/.venv/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 620, in get_context
    object_type, name=name)
  File "/var/tmp/kinto/.venv/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 640, in find_egg_entry_point
    pkg_resources.require(self.spec)
  File "/var/tmp/kinto/.venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 952, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/var/tmp/kinto/.venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 844, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
    pkg_resources.ContextualVersionConflict: (cliquet 2.4.0.dev0 (/var/tmp/kinto/.venv/lib/python2.7/site-packages), Requirement.parse('cliquet==2.3.1'), set(['kinto']))

collections and records aren't ordered (by last modified?)

By default, they don't seems ordered at all.

I tried the following (inspired by what I found in http://cliquet.readthedocs.org/en/latest/api/resource.html?highlight=order#list-of-available-url-parameters), but none changed the ordering:

http GET http://0.0.0.0:8888/v1/buckets/default/collections?_sort=-last_modified -v --auth 'user:password'
http GET http://0.0.0.0:8888/v1/buckets/default/collections/tasks/records?_sort=last_modified -v --auth 'user:password'

Should there be a way to order responses?

Add postgres in the docker setup.

Add a create permission doesn't looks to work

echo '{"data": {}, "permissions": {"records:create": ["system.Authenticated"]}}' | http PUT http://localhost:8888/v1/buckets/mozilla/collections/payments -v --auth="$LOGIN:$PASSWORD"

The create permission is not set.

Create buckets/groups/collections using file/configuration

It would be nice to have a command that takes a file as an input and creates buckets/collections/groups, and permissions...

kinto initialize --server https://kinto.dev.mozaws.net/v1/ conf/payments.json

Finer grained permissions

Today, when we give the collection:write permission it implies the right to share this collection to anyone.

When you are logging in some application, you might want to refrain it to share your data.

A good way to do that is to explicitly ask for permission using scopes.

My proposal is to not give any permission with the kinto scope but to only allow permission to what defines in the scope:

todo:tasks:write
todo:tasks:permission
todo:tasks:schema

Create parent objects if missing ?

Currently, in order to post a record in a new collection, the client shall previously perform two PUT requests (on bucket and on collection).

I believe this is not very convenient for clients (@n1k0?).

What if, on record POST/PUT, we create the collection and the bucket ? (if they are missing and if the client is allowed to)

Untrusted connection on dev server instance

Error code: sec_error_expired_certificate

ETag is only returned on GET

Shouldn't it also be returned on other verbs, like POST/PATCH/PUT...?

URLs ending with a '/' return a 404

Should it allow it like Django does: https://docs.djangoproject.com/en/1.8/ref/settings/#append-slash?

Header value is not an string

With a redis backend, Python 3.4.0 and a GET /v0/collections/todo/records on an unexisting collectionm I get:

2015-03-10 17:37:57,990 ERROR [waitress][Dummy-5] Exception when serving /v0/collections/todo/records
Traceback (most recent call last):
  File "/Users/tarek/Dev/github.com/kinto/.venv/lib/python3.4/site-packages/waitress-0.8.9-py3.4.egg/waitress/channel.py", line 337, in service
    task.service()
  File "/Users/tarek/Dev/github.com/kinto/.venv/lib/python3.4/site-packages/waitress-0.8.9-py3.4.egg/waitress/task.py", line 173, in service
    self.execute()
  File "/Users/tarek/Dev/github.com/kinto/.venv/lib/python3.4/site-packages/waitress-0.8.9-py3.4.egg/waitress/task.py", line 392, in execute
    app_iter = self.channel.server.application(env, start_response)
  File "/Users/tarek/Dev/github.com/kinto/.venv/lib/python3.4/site-packages/pyramid-1.5.4-py3.4.egg/pyramid/router.py", line 243, in __call__
    return response(request.environ, start_response)
  File "/Users/tarek/Dev/github.com/kinto/.venv/lib/python3.4/site-packages/WebOb-1.4-py3.4.egg/webob/response.py", line 1021, in __call__
    start_response(self.status, headerlist)
  File "/Users/tarek/Dev/github.com/kinto/.venv/lib/python3.4/site-packages/waitress-0.8.9-py3.4.egg/waitress/task.py", line 376, in start_response
    'Header value %r is not a string in %r' % (v, (k, v))
AssertionError: Header value b'1426005477988' is not a string in ('Last-Modified', b'1426005477988')

There's a type confusion

List user data in a shared collection

http GET https://kinto.dev.mozaws.net/v1/buckets/todo/collections/tasks/records -v --auth 'bob:bobpassword'

GET /v1/buckets/todo/collections/tasks/records HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Authorization: Basic Ym9iOmJvYnBhc3N3b3Jk
Connection: keep-alive
Host: kinto.dev.mozaws.net
User-Agent: HTTPie/0.9.2

HTTP/1.1 403 Forbidden
Access-Control-Expose-Headers: Backoff, Retry-After, Alert, Next-Page, Total-Records, Last-Modified, ETag
Connection: keep-alive
Content-Length: 95
Content-Type: application/json; charset=UTF-8
Date: Thu, 18 Jun 2015 16:45:11 GMT
Server: nginx/1.4.6 (Ubuntu)

{
    "code": 403, 
    "errno": 121, 
    "error": "Forbidden", 
    "message": "This user cannot access this resource."
}

We should have a list of bob's tasks

Add kinto documentation

What is it ? How to run it ? How to configure it?

Using the _since querystring filter has no effect

I've tried using the _since querystring filter as explained in the tutorial, but it seems to have no effect.

GETing any of those urls returns the exact same list (the full list of records)

http GET http://0.0.0.0:8888/v1/buckets/default/collections/tasks/records?_since=1436094288171 -v  --auth 'user:password'
http GET http://0.0.0.0:8888/v1/buckets/default/collections/tasks/records?_since=foobar -v  --auth 'user:password'
http GET http://0.0.0.0:8888/v1/buckets/default/collections/tasks/records?_since=`date +%s` -v  --auth 'user:password'

The last one uses the current timestamp as the value, which means it should return an empty list.

Tag a new version !

sunion() takes at least 2 arguments (1 given)

https://app.getsentry.com/mozilla/kinto-dev/group/71399462/

"PUT   /v1/buckets/newyork" ? (? ms) sunion() takes at least 2 arguments (1 given) lang=None; exception=Traceback (most recent call last):
  File "/home/ubuntu/venvs/kinto/local/lib/python2.7/site-packages/pyramid/tweens.py", line 21, in excview_tween
    response = handler(request)
  File "/home/ubuntu/venvs/kinto/local/lib/python2.7/site-packages/pyramid/router.py", line 163, in handle_request
    response = view_callable(context, request)
  File "/home/ubuntu/venvs/kinto/local/lib/python2.7/site-packages/pyramid/config/views.py", line 596, in __call__
    return view(context, request)
  File "/home/ubuntu/venvs/kinto/local/lib/python2.7/site-packages/pyramid/config/views.py", line 329, in attr_view
    return view(context, request)
  File "/home/ubuntu/venvs/kinto/local/lib/python2.7/site-packages/pyramid/config/views.py", line 305, in predicate_wrapper
    return view(context, request)
  File "/home/ubuntu/venvs/kinto/local/lib/python2.7/site-packages/pyramid/config/views.py", line 243, in _secured_view
    result = _permitted(context, request)
  File "/home/ubuntu/venvs/kinto/local/lib/python2.7/site-packages/pyramid/config/views.py", line 241, in _permitted
    permission)
  File "/home/ubuntu/venvs/kinto/local/lib/python2.7/site-packages/cliquet/authorization.py", line 46, in permits
    get_bound_permissions=self.get_bound_permissions)
  File "/home/ubuntu/venvs/kinto/local/lib/python2.7/site-packages/cliquet/permission/__init__.py", line 112, in check_permission
    object_id, permission, get_bound_permissions)
  File "/home/ubuntu/venvs/kinto/local/lib/python2.7/site-packages/cliquet/storage/redis.py", line 18, in wrapped
    return func(*args, **kwargs)
  File "/home/ubuntu/venvs/kinto/local/lib/python2.7/site-packages/cliquet/permission/redis.py", line 90, in object_permission_authorized_principals
    return self._decode_set(self._client.sunion(*list(keys)))
TypeError: sunion() takes at least 2 arguments (1 given); uid=631c2d625ee5726172cf67c6750de10a3e1a04bcd603bc9ad6d6b196fa8257a6; errno=None; agent=HTTPie/0.9.2; authn_type=BasicAuth; collection_id=bucket; collection_timestamp=1436354267425

Allow POST on buckets using the id_generator or the id provided in the data.

Define how we want to handle principals

what they are and how to obtain them?

(This is a copy/paste from @Natim on #33)

Users and principals

The authentication process let us get some useful information about our user:

Unique ID
Email address
Scopes

All the authentication scheme doesn't let us have all the same information.

Our proposal is to build a list of principals for a user with all the information we've got including uid, email, scopes to build a user principals as well as a list of principals coming from the configuration, organisations setup and collection setup.

As a example, if I (uid=4567, email=[email protected]) get connected using FxA with the profile and payments scope and that I have access to the servicedesnuages bucket I will have the following principals:

["user:4567", "user:[email protected]",
 "bucket:servicedesnuages", "scope:payements", "scope:profile"]

Then some permissions can be set to any of this principals to let me have them.

For example, on a collection I can set:

{"read_all_records": ["bucket:servicedesnuages"]}

Using the server from the tutorial gives 403

Following the tutorial which uses the https://kinto.dev.mozaws.net instance gives 403 answers.

Solution: either make it accept any user, either change the tutorial to use a local instance http://0.0.0.0:8888 (explaining that it's available after a "pip install -r dev-requirements.txt && make serve").

Separate the README and the actual documentation

Currently, the documentation uses the README file and as such displays a link to the documentation, which isn't useful when we're already on the documentation page (because you already know where the documentation is).

Removing the links from the readme (as was attempted previously) also removes them from the github repository, where it's actually valuable. I believe the best way to proceed would be to actually have the readme as a different resource than the actual documentation.

GETing /v1/buckets raises a 500

2015-07-05 14:28:16,402 ERROR [venusian][Dummy-5] "GET   /v1/buckets" ? (? ms) 'None:read' lang=None; exception=Traceback (most recent call last):
  File "/Users/mathieu/.virtualenvs/kinto/lib/python2.7/site-packages/pyramid-1.6a1-py2.7.egg/pyramid/tweens.py", line 20, in excview_tween
    response = handler(request)
  File "/Users/mathieu/.virtualenvs/kinto/lib/python2.7/site-packages/pyramid-1.6a1-py2.7.egg/pyramid/router.py", line 145, in handle_request
    view_name
  File "/Users/mathieu/.virtualenvs/kinto/lib/python2.7/site-packages/pyramid-1.6a1-py2.7.egg/pyramid/view.py", line 527, in _call_view
    response = view_callable(context, request)
  File "/Users/mathieu/.virtualenvs/kinto/lib/python2.7/site-packages/pyramid-1.6a1-py2.7.egg/pyramid/config/views.py", line 601, in __call__
    return view(context, request)
  File "/Users/mathieu/.virtualenvs/kinto/lib/python2.7/site-packages/pyramid-1.6a1-py2.7.egg/pyramid/config/views.py", line 327, in attr_view
    return view(context, request)
  File "/Users/mathieu/.virtualenvs/kinto/lib/python2.7/site-packages/pyramid-1.6a1-py2.7.egg/pyramid/config/views.py", line 303, in predicate_wrapper
    return view(context, request)
  File "/Users/mathieu/.virtualenvs/kinto/lib/python2.7/site-packages/pyramid-1.6a1-py2.7.egg/pyramid/config/views.py", line 241, in _secured_view
    result = _permitted(context, request)
  File "/Users/mathieu/.virtualenvs/kinto/lib/python2.7/site-packages/pyramid-1.6a1-py2.7.egg/pyramid/config/views.py", line 239, in _permitted
    permission)
  File "/Users/mathieu/.virtualenvs/kinto/lib/python2.7/site-packages/cliquet/authorization.py", line 46, in permits
    get_bound_permissions=self.get_bound_permissions)
  File "/Users/mathieu/.virtualenvs/kinto/lib/python2.7/site-packages/cliquet/permission/__init__.py", line 112, in check_permission
    object_id, permission, get_bound_permissions)
  File "/Users/mathieu/.virtualenvs/kinto/lib/python2.7/site-packages/cliquet/permission/memory.py", line 78, in object_permission_authorized_principals
    keys = get_bound_permissions(object_id, permission)
  File "/Users/mathieu/kinto/kinto/authorization.py", line 146, in get_bound_permissions
    return build_permissions_set(*args, **kwargs)
  File "/Users/mathieu/kinto/kinto/authorization.py", line 129, in build_permissions_set
    for obj, permission_list in inheritance_tree[bound_permission].items():
KeyError: 'None:read'; uid=10ea4e5fbf849196a4fe8a9c250b737dd5ef17abbeb8f99692d62828465a9823; errno=None; agent=HTTPie/0.8.0; authn_type=BasicAuth

Add a kinto init command

We should have a kinto command which allow to:

Bootstrap a configuration file, with kinto init;
Run the migrations with kinto migrate;
Run the server with kinto start.

This is related to mozilla-services/cliquet#327, which aims to provide a start command for cliquet, but we shouldn't ask our clients to type "cliquet": that should be completely opaque to them.

List statsd counters in docs

Add setting to reveal internal error details

Would be useful for the cloud storage backend development...

How users find which server they need to talk to?

We need to find a way to have users chose which server they want to use.

There are a lot of exploration to do here around this idea, but thinking about that while it's not too late and that we don't have a completely centralized service would prevent us from redoing the same error some other projects did.

We should hmac the userid for the default userid bucket_id

At the moment the default bucket_id is the authenticated_userid, this can let people use this information to determine if a user is using the service or not (an leak user privacy)

I propose that we hmac the authenticated_userid to build the bucket_id in order to prevent malicious user to guess the user bucket_id.

Info link on error pages does not mention kinto

{
    "code": 500, 
    "errno": 999, 
    "error": "Internal Server Error", 
    "info": "https://github.com/mozilla-services/cliquet/issues/", 
    "message": "A programmatic error occured, developers have been informed."
}

In the info field, we should point the kinto issues repository instead

PostgreSQL by default ?

put cliquet[postgresql] in requirements
put storage_url in config (default postgres:postgres@localhost/postgres)

Give write permission to bucket creator automatically

We are using ``If-None-Match: *`` to create the default bucket if it doesn't exists but it fails if deleted.

Following mozilla-services/cliquet#365, we need to find a way to create the bucket if it doesn't already exists or if it has been deleted.

Attach record data to BATCH 412 responses.

In order for clients to properly notify from conflicting records on BATCH operations, it would be useful to have the record data attached to the JSON response body, eg.

{
  "errno": 999,
  "message": "Failed batch subrequest",
  "code": 412,
  "error": "Precondition Failed",
  "data": { "id": "…" }
}

Also, it would be nice to have a dedicated errno and message for races.

Define users, objects and permissions

Objects

I have identified three kinds of objects:

bucket: A set of collection
collection: A set of records with permission and schema
record: To share some record of a collection.

For bucket, we only need to have a list of owners that have admin rights on the bucket.

For collections we can have permissions on each level of a collection:

The collection existence: Today as soon as you access it it exists but you may want to clear it all
The collection records: CRUD Any of them, some of them
The collection permissions: Read and Update
Eventually the collection schema: Read and Update

Permissions

Here is a proposal to handle collection sharing and users permissions

The collection owner can change permissions and give specific access to some other users

At first we could only let the collection owner manage permissions on her collection and just delegate the following rights:

create_record: add new records to the collection
read_all_records: read any collection's records
update_all_records: update any collection's records
delete_all_records: delete any collection's records
read_shared_records: read records shared with you (your principal is set on the record)
update_shared_records: update and change records shared with you
delete_shared_records: delete records shared with you

IMHO this could be enough for our current use cases.

The collection owner can also add admin (Also works for the orga model)

A second enhancement could be to go further in this and let you define administrators on the collection, you can then share a collection with someone and let her do:

read_permissions: read the model permissions (who can do what)
update_permissions: change the permissions for the model
delete_collection: delete the collection and all its data

Everything is flexible and go through permissions

An alternative would be to have a list of these 10 permissions and let the owner define them for any users and then check if the permissions is present for the user.

This will let us have more flexibility.

Add hosted docker image for speedup / deployment

The docker image currently must be built from source and will require a re-run of dependency installation any time the source changes (due to how layering works). Also, it is harder to deploy images since the code must be shipped independently. Instead, if you hosted the image on docker hub we could pull the image which would be a little faster and allow for direct deployment. One nice thing about this is that docker hub will automatically rebuild the image each time you push a commit to master.

Ask us if you want examples on how to set this up. Here is one of our pre-built images: https://registry.hub.docker.com/u/mozillamarketplace/solitude/

We can link to it in our docker compose file like this:

solitude:
  image: mozillamarketplace/solitude
  ...

This lets us also deploy images directly to a server.