Need a way to natively copy objects so that files don't need to be downloaded and re-uploaded in order to duplicate.

Some internal projects have a CLI interface to a storage backend that should get ported over and implemented in one location.

In tests_cli.py, 3 of the tests are failing due to messages returned as a repr not having quotes in the test literal.

In test_backends.py, 1 test is failing due to the Arrow.timestamp method being referenced rather than called.

To enable removing creds from application configuration files we should enable use of AWS configuration files and support profiles.

We've developed some helpers useful for hooking storage up to a keg view and organizing path locations a bit. With some small modifications, it would be useful to include that here in the library.

Add a backend to access a local filesystem under a given root directory.

Prevent accessing files outside of the given root, (ex. symlinks and ../ in paths)

Clients should be able to leverage the type information that's provided in the library.

This can be done using the approach described in PEP 561. See also the related mypy docs.

Sometimes we'll want unique filenames in storage, but preserve the original filename for download. I'd imagine this is possible on most if not all of the backends we support.

S3 note: https://medium.com/@leeprovoost/overriding-file-name-of-an-s3-object-using-pre-signed-url-and-aws-sdk-go-4256f30e7809

It would be nice if azure storage didn't have to take name / key as auth, but could also take a sas link

Our signed link token generation uses the itsdangerous JWS that got deprecated and removed. We will want to update using the same sort of method used in keg-auth.

See https://github.com/level12/evolve-airtonomy/commit/30d4a7bd3edab4fdf4576023b3b5ae2d1b1a3c32#diff-b80ca39e5ba09203c741fcc8e103bd16

This includes the entire Azure SDK which is 80+ packages. It should instead include only the packages that it directly needs.

Currently AzureStorage can use a container SAS URL. The idea here is to add support for using a blob SAS URL.

For S3, all objects uploaded via a signed link get the octet-stream content type. That's stored in S3 along with the object as metadata, and gets passed back when the object is requested.

In most cases, that works just fine. But: when an image from S3 is on a page, and the user opens the image in a new tab, the browser will look at that content type and automatically download the file (rather than displaying the image in a tab).

As of 0.5.11, we can inject a content type when getting a download link to workaround this issue. But, we really should allow an app to provide a correct content type at upload time.

copy does not resolve its arguments to the root path

This function is expecting the location param to have a value:

    @staticmethod
    def storage_prefix_path(location, filename):
        """Join the location path with the filename to get the full object path"""
        if filename.startswith('.'):
            filename = filename[1:]
        return '/'.join([location.value, filename])

But this usage in storage_delete_file could potentially pass a None value, as the parameter is optional and has None as the default value.

    @classmethod
    @storage_args()
    def storage_delete_file(cls, filename, storage_location=None, storage_profile=None):
        """Remove file data from storage."""
        storage_instance = cls.storage_get_profile(storage_profile)
        storage_instance.delete(path=cls.storage_prefix_path(storage_location, filename)) <------- here, storage_location could be None
        return True

Add a link_to method to the base storage backend for specific backends to implement.

• In the Azure backend this would return a SAS link
• In the S3 backend it would be a signed URL
• For SFTP and other backends that do not have this kind of feature, return a URL to a view in the application that will serve/accept the file via keg storage methods. The URL should include a verifiable signature that may expire.

As a keg specific addition, include a view mixin that implements file downloads/uploads for SFTP backends

• Linking to https://keg-storage.readthedocs.io in the readme would be helpful
• Typo in docs/source/index.rst: "Documenation"

If a file is written that is less than the writer's chunk_size the multipart upload is never created and close() returns without flushing or finalizing the upload

Why do we require keg in order to get the Storage plugin? keg_storage.__init__ has a _find_spec for keg yet I don't see anything related to keg in there.

Can we remove that, @bladams?

Master branch is failing CI for an azure import issue: https://app.circleci.com/pipelines/github/level12/keg-storage/32/workflows/2f3f2834-98a9-4c88-8e7a-40c83886e20b/jobs/141

Let's make everything prefixed with KEG_STORAGE

Fix:

• keg_storage.plugin.STORAGE_PROFILES
• Add a deprecation warning for STORAGE_PROFILES

To Document:

• Add a page for "configuration" to the docs with details on how to use the various configurations parameters for this library

Instead of having to implement an encrypt/decrypt step in every implementation, I wonder if it would be possible to implement it at the library level.

The steps of getting/putting a file are ones which require use of encrypt/decrypt, list/ delete don't need it.

Though we don't know which one we want in either case. For example, you might want to get an encrypted file and decrypt it in the same step or you might want to take a decrypted file and put it somewhere encrypted.

All that to say, we could wrap the implementation of the get and put such that they encrypt/decrypt automatically when asked to.

Situations we should support:

My guess is we should also support different keys. The situation I am thinking of is re-encrypting a file when getting or putting.

The AWS dependencies should be like the Azure stuff and moved behind an extras flag.

1.0 support will end on 2018-08-31