kinto / kinto-attachment Goto Github PK

View Code? Open in Web Editor NEW

43.0 43.0 10.0 726 KB

Attach files to Kinto records

Home Page: https://kinto.github.io/kinto-attachment/

License: Apache License 2.0

Python 95.31% Makefile 4.69%

kinto-attachment's Introduction

Kinto

Kinto is a minimalist JSON storage service with synchronisation and sharing abilities.

Requirements

Python: 3.8+
Backends: In-memory (development), PostgreSQL 9.5+ (production)

kinto-attachment's People

Stargazers

Watchers

Forkers

rlugojr pombredanne mozillazg gabisurita artines1 hugmatj python-repository-hub f20e63e4-3167-4b3f-b099-7b172a79df91 grahamalama

kinto-attachment's Issues

Add S3 connection options to pyramid_storage

Upstream: danjac/pyramid_storage#8

Switch to boto3 for Amazon S3 backend

Currently only python 2 is supported for Amazon S3

See https://github.com/boto/boto3

Requires danjac/pyramid_storage#13

What is the best way to store encrypted files with Kinto-attachment?

@jadjay is looking for a way to build a file sharing API that let user share encrypted files as part as the C.H.A.T.O.N.S framasoft project.

How will you do that with Kinto and Kinto-Attachment?

Simple fonctionnal test with curl + uwsgi on TravisCI

Add an option not to prepend the base_url in the record location.

We want to be able to have related file path to the base_url in records (It should be the default according to #24)
We want to have the base_url in the Hello view.

Multiple Attachments?

Why can't a record have multiple attachments?

Server crash when uploading a file on a another field than "attachment".

For instance:

$ http --auth token:my-secret --form POST \
  http://localhost:8888/v1/buckets/default/collections/coll/records/$(uuid)/attachment \
  "[email protected]"

HTTP/1.1 500 Internal Server Error !

Support Kinto.js

Any way to make this work with Kinto.JS Sync? It would be nice if I could attach the file offline in Kinto.JS, then have it extracted to S3 on the server during synch.

Should PUT be able to strip the attachment attribute?

Since it does not delete the underlying attachment, we should probably prevent erasing the attachment attribute.

We should add a test to make sure randomize=False cannot be used with keep_old_files.

In that case we will override old files at least for the File System storage.

Instead of using a random filename would it be possible to use the ``{record_id}.ext`` as the file name?

Remove files on flush

If we want to enable this plugin on the demo server, we need to remove files when the server is flushed.

Requires Kinto/kinto#309

Add logging to upload/download scripts

Support chunk uploads for big files

Configure public settings to expose the CDN base url in the hello page.

Obfuscate location filename (using uuid)

provide original filename in attachment metadata
remote location should be random
update README

What is the best way to store file thumbnails using kinto-attachment?

I would like to store both a picture and its thumbnail in kinto-attachment.
Shall I create two collections gallery and gallery_thumbnails?

How can I store both files in the same collection/record?

AttributeError: 'NoneType' object has no attribute 'file'

kinto-attachment 0.7.1, messing around with the API from js, managed to crash the server with some rotten request:

ERROR:kinto.core.views.errors:"POST  /v1/buckets/custom/collections/plop/records/2f3473f9-8259-4cb3-9ec7-121d151e1a4f/attachment" ? (? ms) 'NoneType' object has no attribute 'file' agent=node-fetch/1.0 (+https://github.com/bitinn/node-fetch) authn_type=basicauth collection_id=record collection_timestamp=1468484695624 errno=110 exception=Traceback (most recent call last):
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/tweens.py", line 22, in excview_tween
    response = handler(request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid_tm/__init__.py", line 101, in tm_tween
    reraise(*exc_info)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid_tm/compat.py", line 15, in reraise
    raise value
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid_tm/__init__.py", line 83, in tm_tween
    response = handler(request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/router.py", line 158, in handle_request
    view_name
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/view.py", line 547, in _call_view
    response = view_callable(context, request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/config/views.py", line 182, in __call__
    return view(context, request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/viewderivers.py", line 393, in attr_view
    return view(context, request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/viewderivers.py", line 371, in predicate_wrapper
    return view(context, request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/viewderivers.py", line 302, in _secured_view
    return view(context, request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/viewderivers.py", line 442, in rendered_view
    result = view(context, request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/viewderivers.py", line 147, in _requestonly_view
    response = view(request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/cornice/service.py", line 573, in wrapper
    response = view_(request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/kinto_attachment/views/services.py", line 19, in attachment_post
    return post_attachment_view(request, SINGLE_FILE_FIELD)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/kinto_attachment/views/__init__.py", line 34, in post_attachment_view
    gzipped=gzipped)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/kinto_attachment/utils.py", line 126, in save_file
    content.file.seek(0)
AttributeError: 'NoneType' object has no attribute 'file' lang=None uid=a9d81e1b56f2b77103e87601621a99b3d82a15abf7c72ce57a032d0964317c7a

I unfortunately can't provide a curl/httpie command for reproducing as fetch doesn't expose such a feature.

Add option to Gzip content in demo form

C.f. https://bugzilla.mozilla.org/show_bug.cgi?id=1212353

Replace existing attachment when one already exists

Add pattern in download path

Currently, every attachment of every collections are stored in the same folder.

It would be nice to be able to specify something like:

kinto.attachment.base_path = /tmp/{collection_id}

kinto.attachment.aws.bucket = {bucket-id}-{collection-id}

Add delete /attachment endpoint

Delete attachment
Remove metadata from record
Delete link record-file

500 when posting with empty filename

Spotted while hacking on file attachment in kinto-admin

Improvements on the tox upload/download script

Remy suggested to use localhost instead of kinto.dev in https://github.com/Kinto/kinto-attachment/blob/master/tox.ini#L26

Check record permissions on upload

Remove attachment attribute instead of setting it to null when deleting attachment

Currently when an attachment is removed, the attachment property is set to null instead of being completely removed.

Note: This is related to Kinto/kinto#623

Concurrency headers

Fail if provided ETag does not match existing record
Fail with If-None-Match if record already exists

Why CORS arguments for services are hard-coded?

If we import from cliquet import Service, we're supposed to get CORS setup, and no need to repeat cors_enabled=True and cors_origin='*'

Add a SeaFile backend.

While deploying to heroku @enguerran was struggleling to find a good option for serving files.

How do we deal with unknown mime types?

As seen on Fennec, some boxes just don't have mimetypes registered see:

pocmo/Fennec-DLC-Catalog@b6abf51

we should deal with this in kinto-attachement. https://github.com/Kinto/kinto-attachment/blob/master/scripts/upload.py#L61

maybe by raising an error ? another option is to guess it on the server-side in Kinto when it's not provided by the client maybe ?

option to gzip uploaded file on the fly

Once we have the proper hooking mechanism in place - see Kinto/kinto#677

We should offer the option to gzip files that are posted to Kinto, like how it's actually done for Fennec https://github.com/pocmo/Fennec-DLC-Catalog/blob/master/create_catalog.py#L136

That will let us use kinto-admin for managing fonts in Fennec

Add a requirements file for scripts

The kinto-attachment/setup.py is responsible of dependencies of the plugin. However scripts have one dependency which is kinto-client we should add a requirements file there to help executing the scripts.

Do not try to write a file in heartbeat if server set to readonly

Port kinto-attachment to python3

File hash as hex digest instead of base 64

500 when posting invalid binary string

While hacking on attachments in kinto-admin, we had a 500.

Probably posting a ByteString in the FormData instead of Blob.

Full URL is stored in records attributes

When the root_url setting changes, the full URL in records is not automatically updated.

Switch to after commit events

Currently when a delete request is rolledback, the attachment file is deleted.
The new AfterResourceChanged allows to register subscribers that are executed if and only if the commit was done and successful.

For example, in order to delete files from filestorage:

Use ResourceChanged events to mark file links as «deleted»
Use AfterResourceChanged events to delete files that have file links marked as deleted

Crash with Kinto 5.3

I'm curious to see how we didn't catch this in kinto-dist integration tests...

Support sending attachments without filename

Use case: from kinto-http.js, being able to attach a file coming from a dataURI (without filename).

def save_file(content, request, randomize=True, gzipped=False):
    folder_pattern = request.registry.settings.get('attachment.folder', '')
    folder = folder_pattern.format(**request.matchdict) or None

    # Read file to compute hash.
    if not isinstance(content, cgi.FieldStorage):
        error_msg = 'Filename is required.'
        raise_invalid(request, location='body', description=error_msg)

Crash if posted data is not JSON

posted field a=b and server crashed with json parse error

init() takes 2 positional arguments but 3 were given

Setting up kinto-attachment 0.7.0 locally against Kinto 3.2.1, Ubuntu 16.05, Python 3.5.1 (standard dist package).

kinto.ini:

[app:main]
use = egg:kinto

# Required by integration tests
kinto.flush_endpoint_enabled = true

# Required by basic auth
kinto.userid_hmac_secret = a-secret-string

# Allow browsing all buckets
kinto.bucket_read_principals = system.Authenticated

# Plugins registration
kinto.includes = kinto.plugins.default_bucket
                 kinto_attachment

# Kinto-attachment
kinto.attachment.base_url = http://cdn.service.org/files/
kinto.attachment.folder = {bucket_id}/{collection_id}
kinto.attachment.keep_old_files = true
kinto.attachment.base_path = /tmp

[server:main]
use = egg:waitress#main
host = 0.0.0.0
port = 8888

The server starts fine:

$ .venv/bin/kinto --ini test/kinto.ini start
Starting server in PID 7954.
serving on http://0.0.0.0:8888

But requesting GET http://0.0.0.0:8888/v1/buckets/custom/collections/plop/records/96a38b32-2bf2-47a2-9ddc-944965ebd6dd/attachment explodes with:

ERROR:kinto.core.views.errors:"GET   /v1/buckets/custom/collections/plop/records/96a38b32-2bf2-47a2-9ddc-944965ebd6dd/attachment" ? (? ms) __init__() takes 2 positional arguments but 3 were given agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36 authn_type=None errno=None exception=Traceback (most recent call last):
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/tweens.py", line 22, in excview_tween
    response = handler(request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid_tm/__init__.py", line 101, in tm_tween
    reraise(*exc_info)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid_tm/compat.py", line 15, in reraise
    raise value
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid_tm/__init__.py", line 83, in tm_tween
    response = handler(request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/pyramid/router.py", line 127, in handle_request
    root = root_factory(request)
  File "/home/niko/work/kinto-http.js/.venv/lib/python3.5/site-packages/kinto_attachment/utils.py", line 32, in __init__
    resource = Record(request, self)
TypeError: __init__() takes 2 positional arguments but 3 were given lang=fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4 uid=None

What am I doing wrong? Edit: I know this GET request is not supported, though I was not expecting it to raise.

kinto-attachment doesn't work with current kinto-master

ERROR    "POST  /v1/buckets/fennec/collections/fonts/records/9b0249f6-8bda-42ea-84fb-db4c11716b49/attachment" ? (? ms) 'Service' object has no attribute 'viewset'

Add a heartbeat for the remote file storage.

Add a files: true / files: false in the heartbeat endpoint when the plugin is activated.

Decide if the file hash should be hex digest or base64 digest.

The doc said the latter but the code implements the former.

Should we let it like it is implemented or change it to use a base64 digest?

The usage is that people are using hexdigest for shasum. In cryptography we are using base64. Matter of taste I supposed.

Error when using local storage

Here is an extract of my config file:

kinto.includes = kinto.plugins.default_bucket
                 kinto_attachment

kinto.attachment.base_path = /tmp
kinto.attachment.base_url = http://localhost/
kinto.attachment.folder = {bucket_id}/{collection_id}
kinto.attachment.keep_old_files = true

I posted an attachment after creating the corresponding collections:

http --auth bob:passwd --form POST http://localhost:8888/v1/buckets/default/collections/pictures/records/de5c7f0a-ea9a-466f-ba36-6e69f76155bd/attachment "attachment@~/Pictures/Firefox-Nebula-Background.jpg"

As a result I got a error 500:

HTTP/1.1 500 Internal Server Error
Access-Control-Expose-Headers: Backoff,Retry-After,Alert,Content-Length
Content-Length: 196
Content-Type: application/json; charset=UTF-8
Date: Tue, 08 Mar 2016 16:50:21 GMT
Server: waitress

{
    "code": 500,
    "errno": 999,
    "error": "Internal Server Error",
    "info": "https://github.com/mozilla-services/cliquet/issues/",
    "message": "A programmatic error occured, developers have been informed."
}

On the server, I got this error:

2016-03-08 17:50:21,353 ERROR [venusian][waitress] "POST  /v1/buckets/57465e3f-bbb8-58d0-7190-465e69faa967/collections/pictures/records/de5c7f0a-ea9a-466f-ba36-6e69f76155bd/attachment" ? (? ms) You can only set Request.body to bytes (not <class 'str'>) agent=HTTPie/0.9.3; collection_timestamp=1457455821327; authn_type=BasicAuth; collection_id=record; uid=d693e011c88aeb3c6e28ad0b95e73120e007687d81f7e3d7ec3204eaa2c3e8ab; lang=None; errno=110; exception=Traceback (most recent call last):
[stacktrace]
TypeError: You can only set Request.body to bytes (not <class 'str'>)
2016-03-08 17:50:21,354 INFO  [venusian][waitress] "POST  /v1/buckets/57465e3f-bbb8-58d0-7190-465e69faa967/collections/pictures/records/de5c7f0a-ea9a-466f-ba36-6e69f76155bd/attachment" 500 (31 ms) request.summary time=2016-03-08T17:50:21.354000; agent=HTTPie/0.9.3; collection_timestamp=1457455821327; authn_type=BasicAuth; collection_id=record; uid=d693e011c88aeb3c6e28ad0b95e73120e007687d81f7e3d7ec3204eaa2c3e8ab; lang=None; errno=999

But the attachment is stored in /tmp/57465e3f-bbb8-58d0-7190-465e69faa967/pictures/f5044403-6314-4b63-b52e-2f93a721fd8b.jpg.

Could you help me understand what is wrong?