Comments (9)
Yes, I am going to do it right now, thanks!
from s3mock.
@campidelli not sure what goes wrong with your code.
I added an integration test that uses the CRT client, and it works just fine, see linked PR...
from s3mock.
there are two ways to add the CRT client to the S3AsyncClient - see S3TestBase.kt
. It's in Kotlin, but the syntax is not too different from Java.
Not sure why, but I can't get one of the CRT clients to work with an https endpoint.
The test works with both of them.
Only the "autoCrt" client (using your way to set it up) lands in this method though:
at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:93) ~[s3-2.20.78.jar:na]
from s3mock.
is this the service you're working on?
https://github.com/campidelli/file-storage-service/blob/main/pom.xml#L92
The CRT dependency is included in the regular V2 SDK - you should use the same version coming from the AWS BOM POM you imported:
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>aws-crt-client</artifactId>
</dependency>
from s3mock.
Hi @afranken, thanks for the prompt response.
I have followed this tutorial (https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/transfer-manager.html) where they ask for this specific version or higher, I didn't know the client was part of the SDK already, good catch!
The project I am working on is similar to this, but it is an internal one.
If I change the Client to use the HTTP version, I lose the multipart upload feature< I quote from https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/crt-based-s3-client.html
Although both components use libraries from the AWS Common Runtime, the AWS CRT-based S3 client uses the aws-c-s3 library and supports the S3 multipart upload API features. Since the AWS CRT-based HTTP client is meant for general purpose use, it does not support the S3 multipart upload API features.
from s3mock.
For this specific test I am using the putObject
API, as you can check on the following code (from my REST controller):
@PostMapping
public Mono<ResponseEntity<UploadResult>> uploadHandler(@RequestHeader HttpHeaders headers, @RequestBody Flux<ByteBuffer> body) {
long length = headers.getContentLength();
if (length < 0) {
throw new UploadFailedException(HttpStatus.BAD_REQUEST.value(), Optional.of("required header missing: Content-Length"));
}
String fileKey = UUID.randomUUID().toString();
Map<String, String> metadata = new HashMap<>();
metadata.put("file_name", "test");
MediaType mediaType = headers.getContentType();
if (mediaType == null) {
mediaType = MediaType.APPLICATION_OCTET_STREAM;
}
log.info("UploadHandler: mediaType {}, length={}, fileKey = {}", mediaType, length, fileKey);
Mono<PutObjectResponse> putObjectResponseMono = Mono.fromFuture(s3client.putObject(PutObjectRequest.builder()
.bucket(s3config.getBuckets().upload())
.contentLength(length)
.key(fileKey)
.contentType(mediaType.toString())
.metadata(metadata)
.build(),
AsyncRequestBody.fromPublisher(body)));
return putObjectResponseMono
.map((response) -> {
checkUploadResult(response);
return ResponseEntity
.status(HttpStatus.CREATED)
.body(new UploadResult(HttpStatus.CREATED, new String[]{fileKey}));
});
}
from s3mock.
Can you please include this in your test and see if it works?
Thanks!
from s3mock.
Just added a test, works without problems:
https://github.com/adobe/S3Mock/blob/main/integration-tests/src/test/kotlin/com/adobe/testing/s3mock/its/CrtAsyncV2IT.kt
must still be something with the way you set up the client, or the interaction with Spring reactive?
Maybe you could try to reproduce the problem in your example service...
from s3mock.
I have updated my project and the error doesn't happen anymore.
However, I am facing another issue, which is probably an implementation error, I would gladly accept a hand here.
#1182
Thanks!
from s3mock.
Related Issues (20)
- When upgrading to SpringBoot 3.1.1 I get NoClassDefFoundError: javax/servlet/Filter HOT 9
- accessKey and secretKey invalid HOT 9
- @JJRdec Could you please open a new ticket with a description on how to reproduce the issue?
- Upload Files read timeout (with Persist data) HOT 3
- SSLException: Unsupported or unrecognized SSL message HOT 6
- Copying object with new storage class does not appear to save storage class HOT 5
- initialBuckets is ignored if at least one bucket exists with a bucketMetadata.json
- Service: Amazon S3; Status Code: 404; Error Code: 404 HOT 2
- org.eclipse.jetty.server.Request : java.lang.IllegalStateException: Form is larger than max length 200000 HOT 1
- while reading large number of record from s3mock getting Exception( java.io.IOException: Attempted read on closed stream) HOT 2
- S3Mock is incompatible with Spring Boot 3.2.x and Spring Framework 6.x HOT 3
- createBucket fails using S3AsyncClient with NoSuchBucketException HOT 1
- Support S3AsyncClient from software.amazon.awssdk:s3 HOT 2
- Cannot put object acl without a policy HOT 5
- When upgrading to Spring 3.2.2: ClassNotFoundException: javax.xml.bind.annotation.XmlElement HOT 1
- When using async uploads, S3Mock incorrectly stores unsigned, chunked uploads without removing meta (chunk, checksum) information. HOT 14
- Some key names lead to request paths that do not comply to RFC3986, leading to failing requests in S3Mock HOT 7
- When starting S3Mock in Docker, root folder is not accessible from outside of the container HOT 4
- Synchronisation issues HOT 3
- missing namespace xml header. HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from s3mock.