A Java (NIO2) FileSystem Provider for Amazon AWS S3.
Home Page: https://s3fs-nio.carlspring.org/
This is an implementation of an Amazon AWS S3 FileSystem provider using JSR-203 (a.k.a. NIO2) for Java 8.
Amazon Simple Storage Service provides a fully redundant data storage infrastructure for storing and retrieving any amount of data, at any time.
NIO2 is the new file management API, introduced in Java version 7.
This project provides a complete API implementation, for managing files and folders directly in Amazon S3.
We support JDK 8, 11 and 17.
You can check out our documentation here.
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "github-actions[bot]")
We need to add some documentation on what sort of caching we support and how it works.
The following tasks will need to be carried out:
We should set up issue templates for the project. This will help bring more consistency to the way that issues and feature requests are described.
The following tasks will need to be carried out:
At the moment, it's not possible to use multiple buckets, credentials and regions. In a more sophisticated application this would be quite a limitation.
The format of the amazon.properties currently looks like this:
s3fs.bucket.name=/bucket
s3fs.access.key=AKIADEMOACCESSKEY
s3fs.secret.key=sfsoj291aSFfasfsafs1fsa1
s3fs.region=eu-west-2
s3fs.protocol=https
Once #136 has been implemented, (and the underscores in the properties have been converted to dots), we should consider doing something like this:
s3fs.${configurationName}.bucket.name=/bucket
s3fs.${configurationName}.access.key=AKIADEMOACCESSKEY
s3fs.${configurationName}.secret.key=sfsoj291aSFfasfsafs1fsa1
s3fs.${configurationName}.region=eu-west-2
s3fs.${configurationName}.protocol=https
For example:
s3fs.filestore1.bucket.name=/bucket
s3fs.filestore1.access.key=AKIADEMOACCESSKEY
s3fs.filestore1.secret.key=sfsoj291aSFfasfsafs1fsa1
s3fs.filestore1.region=eu-west-2
s3fs.filestore1.protocol=https
This is just a proposal, which is open to better suggestions, if anyone has such.
The following tasks will need to be carried out:
This task:
We should define the project's licensing terms.
pom.xml file states it is (only) under an Apache 2.0. As @jarnaiz has not replied, and, since it isn't perfectly unclear under which of the two licenses the project is, our spin-off will be dual-licensed.pom.xml.The S3Factory class manages the build of a new Amazon S3 instance, which needs a region. This is provided by getting it from a properties file, and, in the case there's no property defined, the region is obtained by using a default region provider chain.
As specified in this Pull Request discussion, although it's highly unlikely, there could be a different region provider chain needed by a consumer of the API (this would have to be custom, as the SDK only has Default). Might want to provide a customization option for this, rather than hard coding new DefaultAwsRegionProviderChain.
The following tasks will need to be carried out:
org.carlspring.cloud.storage.s3fs.S3AwsRegionProviderChain.software.amazon.awssdk.regions.providers.AwsRegionProviderChain.software.amazon.awssdk.regions.providers.SystemSettingsRegionProvidersoftware.amazon.awssdk.regions.providers.AwsProfileRegionProvidergetRegion method should not be overridden, so method from parent will be used.org.carlspring.cloud.storage.s3fs.S3Factory class, getRegionmethod, by replacing the DefaultAwsProviderChain with the new one.This task:
We should set up pull request templates for the project. This will help bring more consistency to the way that pull request are described.
The following tasks will need to be carried out:
The AWS SDK for Java 2.x utilizes a new, non-blocking SDK architecture built on Netty to support true non-blocking I/O. Taking advantage of it, we can also switch the NIO implementation to use AsynchronousFileChannel. This should give a significant performance boost.
A plain FileChannel implements InterruptibleChannel so it, as well as anything that uses it such as the OutputStream returned by Files.newOutputStream(), has the unfortunate [1], [2] behaviour that performing any blocking operation on it (e.g. read() and write()) in a thread in interrupted state will cause the Channel itself to close with java.nio.channels.ClosedByInterruptException.
If this is a problem, using AsynchronousFileChannel instead is a possible alternative.
The following tasks will need to be carried out:
S3FileChannel extend AsynchronousFileChannel instead of FileChannel and override the respective methods associated to this new class.S3FileSystemProvider override the method newAsynchronousFileChannel instead of newFileChannel from FileSystemProvider.This task:
The project is still using JUnit 4.x. We should make the necessary code changes to migrate to JUnit 5 (Jupiter).
A lot of the plugins are significantly outdated, or don't have explicit versions and we should make sure we're using up-to-date versions.
The following tasks will need to be carried out:
We need to add some meaningful labels for the project.
The following labels will need to be added:
needs triagerequirements pendingin progreson holdhelp wantedhacktoberfestgood first issuefeatureThe S3Factory class manages the build of a new Amazon S3 instance, which right now it's using an Apache HTTP Client.
As specified in this Pull Request discussion, this is locking in customers to the ApacheHttpClient, which adds a dependency they may not want. It's needed to provide an option for other SdkHttpClient implementations.
The UrlConnectionHttpClient is fairly popular choice in Java-based Lambda functions as it has faster startup time, so less impact to cold starts.
The following tasks will need to be carried out:
SdkHttpClient implementationsThis task:
Hi,
I'm trying to run the integration tests before submitting a PR. But I'm having a hard time getting the amazon-test.properties file to load. EnvironmentBuilder.class.getResourceAsStream("/amazon-test.properties"); returns null. (I'd argue that should be checked for null before passing props.load because it took me more time than I'd like to admit to find the cause.)
On one hand, the path is /amazon-test.properties which would imply to me it has to be put in the root of whatever filesystem the source is on. If that is intended that could be problematic for people running on Windows.
However, even if I just change it to EnvironmentBuilder.class.getResourceAsStream("amazon-test.properties"); and put that file in either the root of the repo or in the target directory it still doesn't work.
Any advice?
Thanks,
Adam
A lot of the dependencies are significantly outdated and we should make sure we're using up-to-date versions.
The following tasks will need to be carried out:
As we'd like to make a somewhat "fresh" spin-off of the Upplication/Amazon-S3-FileSystem-NIO2 project, we'd like to make it clearer for people that this is a new project and for this we will need new artifact coordinates.
The new artifact coordinates will be:
<groupId>org.carlspring.cloud.aws</groupId>
<artifactId>s3fs-nio2</artifactId>
<packaging>jar</packaging>
pom.xml as required.The integration tests currently leave behind the resources they have created without cleaning up after the execution.
We should improve this so that when you test with a real S3 bucket, it cleans up and reduces any costs for hosting.
Currently, these tests are using constructs like this to generate random names for the files:
Path file = fileSystemAmazon.getPath(bucket, UUID.randomUUID().toString());
While this does do the job, there is no way to actually clean up, unless we can invoke a "delete all" operation against the S3 bucket via the AWS SDK. This is inefficient.
We can do something like what we've done in our strongbox project. In essence, thanks to @sbespalov, we have implemented annotations (and a mechanism around them) for creating test repositories and test artifacts. These are created for every test method and then disposed of, once it's completed (there is an option to preserve these, of course, if required for debugging purposes).
spring-test to the mix just for this, without having to turn this project into one that requires Spring as a runtime dependency.@S3TestPath annotation.S3TestPathContext. The path can still use UUID-s for this, but this way they'll be tracked and removed.S3TestPathExecutionListener.org.carlspring.cloud.storage.s3fs.path.ToURLITorg.carlspring.cloud.storage.s3fs.S3ClientITorg.carlspring.cloud.storage.s3fs.FilesITorg.carlspring.cloud.storage.s3fs.spike.EnvironmentITorg.carlspring.cloud.storage.s3fs.ExampleClassITorg.carlspring.cloud.storage.s3fs.fileSystemProvider.NewFileSystemITorg.carlspring.cloud.storage.s3fs.fileSystemProvider.GetFileSystemITorg.carlspring.cloud.storage.s3fs.fileSystemProvider.NewByteChannelITorg.carlspring.cloud.storage.s3fs.S3UtilsITorg.carlspring.cloud.storage.s3fs.FileSystemsITThe project uses SLF4J. However, the pom includes a dependency on a specific logging implementation, logback. Logback should be removed from the pom or moved to a different scope like test or provided if it is needed for testing.
We should add a section in the wiki the explains:
amazon-test.properties.Optimally, there should be some screenshots.
This task:
Apply the Carlspring code style rules.
We should set up Sonarcloud so that we can make sure that the current code is up to standard and so that future contriburions via pull requests are also properly analyzed for code issues.
The following tasks will need to be carried out:
We need to investigate what would be required in order to add support for MinIO.
The following tasks will need to be carried out:
MinIOITTestSuite based on the AmazonS3ITSuite one.testcontainers before all tests in the suite (and then stops it at the end). (check #60)This task:
Under JDK11, we are seeing the following error (which doesn't prevent the code from working, but which should be properly addressed):
[ERROR] WARNING: An illegal reflective access operation has occurred
[ERROR] WARNING: Illegal reflective access using Lookup on org.mockito.internal.util.reflection.InstrumentationMemberAccessor$Dispatcher$ByteBuddy$yCgmqpdD to class java.nio.channels.spi.AbstractInterruptibleChannel
[ERROR] WARNING: Please consider reporting this to the maintainers of org.mockito.internal.util.reflection.InstrumentationMemberAccessor$Dispatcher$ByteBuddy$yCgmqpdD
[ERROR] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
[ERROR] WARNING: All illegal access operations will be denied in a future release
Once org.mockito:mockito:3.5.6 has been released, we should try this out again and report back our findings.
The following tasks will need to be carried out:
org.mockito:mockito to >= 3.5.6.This task:
The multipart upload API is designed to improve the upload experience for larger objects. You can upload an object in parts. These object parts can be uploaded independently, in any order, and in parallel. You can use a multipart upload for objects from 5 MB to 5 TB in size.
The following tasks will need to be carried out:
CreateMultipartUploadRequest and get the upload id.UploadPartRequest and CompletedPart.completeMultipartUpload operation with CompletedMultipartUpload request to tell S3 to merge all uploaded parts and finish the multipart operation.This task:
When using S3FS as a file system for Mina SFTP, it's not possible to change the directory (cd command):
sftp> ls -l
d--------- 1 OWNER@ GROUP@ 0 Jan 25 10:51 bar
d--------- 1 OWNER@ GROUP@ 0 Jan 25 10:42 foo
sftp> cd foo
Can't change directory: Can't check target
POSIX permissions are not being returned for these directories, which is causing an exception in the SFTP client when attempting to cd to the directory.
The following tasks will need to be carried out:
S3Utils, if the path is a directory, obtain the bucket ACL, then obtain its ACL grant permissions, convert them to Posix file permissions and return them with the response.We need to investigate what will need to be done in order to upgrade to com.github.marschall:memoryfilesystem from 2.1.0 to 0.9.1 due to incompatibilities with JDK11. This causes a problem with timestamps like this:
[ERROR] Failures:
[ERROR] ReadAttributesTest.readAttributesAll:463 expected:<2020-08-23T10:19:52.0635Z> but was:<2020-08-23T21:19:52.635Z>
[ERROR] ReadAttributesTest.readAttributesFile:92 expected:<2020-08-23T21:19:52.669127Z> but was:<2020-08-23T21:19:52.669Z>
This issue seems to be related to the higher precision of timestamps introduced in JDK 9 (JDK-8068730)
This task:
We need to sort out the failing integration tests.
To run the build with the integration tests enabled, run:
mvn clean package -Pintegration-tests
The following tasks will need to be carried out:
master of Amazon-S3-FileSystem-NIO2.We should gather a list of badges that we'd like to display such as:
The following tasks will need to be carried out:
Some test methods aren't following the camel case naming convention. We should rename these in order to be more consistent.
We should set up a proper wiki with searchable/indexable examples.
The following tasks will need to be carried out:
mkdocs.Currently, we're not collecting any code coverage, as this has been commented out.
The following tasks will need to be carried out:
jacoco-maven-plugin to the latest.This task:
We should upgrade to AWS' Java SDK v2.
The following tasks will need to be carried out:
sdk2 branch seems to have implemented this to some part. We should evaluate this work.The version of aws-java-sdk-s3 currently at use is 1.11.232. We need to upgrade to the latest version, (currently, 1.11.830) and make sure things still work as expected.
The following tasks will need to be carried out:
aws-java-sdk-s3.At the moment, the integration tests are executed as separate tests using the maven-failsafe-plugin. We need to re-work this so that they are executed as a suite. This needs to be done so that we could be able to re-run the same integration tests against different environments. For example, the default profile for integration tests should be against a real Amazon S3. However, if we re-work these to be executed as a suite, we could then add another suite with the same tests, but for MinIO instead.
What we can do is:
@Tag("integration-test-s3") annotation.@IncludeTags("integration-test-s3") annotationAmazonS3ITTestSuite.MinIOITTestSuite.maven-failsafe-plugin only execute the *ITSuite classes.The following tasks will need to be carried out:
maven-failsafe-plugin only execute *ITSuite-s.This task:
We should set up GitGuardian so that we can protect ourselves from accientally leaking production credentials.
The following tasks will need to be carried out:
We should create a logo for the project.
The logo should resemble something between the logo for AWS S3 and a file system.
We should check under what license the S3 logo is and whether it could be used for OSS purposes. (We need links that quote the official Amazon view on this.
We should consider replacing log4j with slf4j.
The following tasks will need to be carried out:
slf4j.We need to set up a matrix build that tests the code:
The following tasks will need to be carried out:
~/.m2/repository cache to speed-up build.There are a lot of methods (both in production and test code) that either have generic throws Exception clauses, or exception clauses for exceptions that aren't being thrown at all in the respective methods.
The following tasks will need to be carried out:
The parent projects were aimed to suppor JDK7, but we should move to at least JDK8 as a base JDK.
pom.xmlThe current way that the configuration is implemented is quite confusing and error-prone to work with. We need to clarify how we'd like this to work, gather the requirements and refactor this.
Map<String, Object> env;
String accessKey = System.getenv(ACCESS_KEY);
String secretKey = System.getenv(SECRET_KEY);
String region = System.getenv(REGION);
String protocol = System.getenv(PROTOCOL);
env = ImmutableMap.<String, Object>builder().put(ACCESS_KEY, accessKey)
.put(SECRET_KEY, secretKey)
.put(REGION, region)
.put(PROTOCOL, protocol)
.build();If there is an amazon.properties on the classpath, this will be used to load the properties. The configuration is loaded using org.carlspring.cloud.storage.s3fs.S3FileSystemProvider.
If not, the generic way can be used.
If there is an amazon-test.properties on the classpath, this will be used to load the properties. The configuration is loaded using org.carlspring.cloud.storage.s3fs.util.EnvironmentBuilder.
If not, the generic way can be used.
The following tasks will need to be carried out:
This task:
AWS S3 Access Points are unique hostnames attached to an S3 bucket, each with dedicated access policies. This allows large scale access control to be delegated to multiple APs, each dedicated to providing access to one user, rather than combining all access control in one large bucket policy. Larger scale users are increasingly using APs to simplify bucket access control.
S3 Access Points ARNs may be used by the AWS CLI and SDK in place of a bucket name; once an S3 AP is defined to a bucket mybucket, the following are equivalent:
s3://mybucket/
s3://arn:aws:s3:<region>:<account-id>:accesspoint/<ap-name>/
This task:
As the original authors were Spanish native-speakers, there are a lot of comments, variables, etc in the code that are in Spanish. We should translate these in English, so that it's clearer for other contributors that don't speak Spanish.
The code base is not that large, so going through it (both sources and resource files) should not take too long.
The following tasks will need to be carried out:
This is the master task for creating a spin-off of Upplication/Amazon-S3-FileSystem-NIO2.
CONTRIBUTING.md.s3fs-nio on Stackoverflow ( 
)While Upplication/Amazon-S3-FileSystem-NIO2 seems abandoned, there appear to be quite a few open pull requests. We should see if there are any of them that we would like to apply to our project and reach out to the respective contributors to ask them, if they'd like to contribute them back to our project.
This is the master task. The actual work should be split into sub-tasks and be added to the list below.
The following tasks will need to be carried out:
S3OutputStream that uploads data to S3 directly ( #95 )URI encoding with special charactersS3FileSystemProvider.readAttributes() sees a newly created fileattrs.isDirectory, give it full permissions control, so we don't get a NullPointerException because we did not initialize the permissionsWe should consider setting up our tests to use testcontainers.
The following tasks will need to be carried out:
testcontainers and Amazon.testcontainers so it starts the container and tests when -Pit-minio is used.testcontainer and use it in @MinioIntegrationTest with @ExtendWith in tests that can run against MinIO. An exampe of how to do this can be found here.URI when creating the filesystem depending on the running.it property (i.e. for S3 s3://s3.amazonaws.com/ and s3://localhost:9000/ for MinIO). Perhaps it would be a good idea to add this into the BaseIntegrationTest as a method which generates the appropriate URI for you.testcontainers via @MinioIntegrationTest where this is possible. Some integration tests can be ran against S3 and MinIO since the APIs are compatible. If that is possible, then the test can be annotated with both @S3IntegrationTest and @MinioIntegrationTest.This task:
We need to do this to avoid confusing future users in regards to where to find the code and whether they should be looking at one of the earlier projects from which we're inheriting the code.
com.upplication.s3fs to org.carlspring.cloud.storage.s3fs.We should enable snyk.io scans for the project, as well as for pull requests.
The following tasks will need to be carried out:
We should re-write the README.md and cover:
The following tasks will need to be carried out:
README.md, so that it reflects the new reality.We should convert all the properties to use dots instead of underscores, as this is the real Java convention.
How it looks currently:
amazon.propertiesbucket_name=/bucket
s3fs_access_key=AKIAEXAMPLEKEY
s3fs_secret_key=fadf3rafsadf3r23wafsdf9i209ifs
s3fs_region=eu-west-2
s3fs_protocol=https
How it should look:
amazon.propertiess3fs.bucket.name=/bucket
s3fs.access.key=AKIAEXAMPLEKEY
s3fs.secret.key=fadf3rafsadf3r23wafsdf9i209ifs
s3fs.region=eu-west-2
s3fs.protocol=https
The following tasks will need to be carried out:
bucket_name property to s3fs.bucket.name.This task:
We need to improve the documentation to properly explain the supported configuration properties and how to use them and what for.
The documentation for this is under docs/content/reference/configuration-options.md.
This task:
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.