breakthrough-energy / fs-azureblob Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
As a user of fs-azureblob, I want to consult documentation about which aspects of pyfilesystem are supported by the library, as well as which aspects of blob storage are supported.
As a user of the fs-azureblob extension, I want to know what errors could be raised and prefer they are consistent with possible errors from other filesystems.
Pyfilesystem provides a bunch of built in errors that can represent underlying errors from the os, sdk, etc. We should handle any internal errors by converting to an fs error and re-raising if needed.
I want to see badges when I browse the github repo
Enable consumption of the extension by standard means
Steps to create an extension from the docs. The main points are:
entry_points
to setup.py
Opener
subclass to handle custom url protocoln/a
Technically, we could use the extension without doing this, but it make the package consistent with other extensions and possibly worth listing as an "official" 3rd party extension
I want progress bars for upload/download.
Write a custom download method instead of delegating to the base FS
implementation. The download
method allows for additional kwargs, one of which should be a callback, so that the exact progress bar implementation is left up to consumers.
Note: the BlobFile
implementation first downloads the blob to a temp file before any read operations occur. We should consider providing some context to the user since the progress bar function below is delayed until the initial download is complete. Alternatively, the behavior could be optimized, but this might be difficult to do while preserving the file-like-object api.
Not having progress bars.
Refer to the similar customization done for the ssh fs
Some of the tests require credentials to our storage account. It would be nice if an arbitrary developer could run these without needing access.
There is a storage emulator which could be run in a container. The emulator supports authentication, so it should suffice for testing error handling or any credential related functionality.
Update: the storage emulator is now deprecated and being replaced by azurite
The smart_open package implements a file api on top of various storage backends. It could potentially replace the BlobFile implementation which does not stream efficiently.
This should provide a description of the feature request, e.g.:
openbin
to simply call smart_open
When I get a chance, I will test this with the progress bar to see if the behavior is intuitive. Currently, a blob must be downloaded before the progress bar would start, which for large enough files is quite apparent.
Use blob storage as a pyfilesystem backend
Inherit from the base FS
class and implement the essential methods for the filesystem to work.
n/a
n/a
Make sure any relevant test cases pass.
Follow the testing guide from the docs. We should look into using the existing tests, but at the very least define our own test cases specific to blob storage. Not all features provided by other filesystems will apply here (e.g. symlinks, device files, other os stuff). One example is that blob storage doesn't have a concept of directories, but blobs can have paths. So [blob_container]/path/to/file.txt
would be a single blob named /path/to/file.txt
, and ls /path/to
doesn't work unless we fake it (which we do, at least tentatively).
n/a
n/a
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.