Comments (3)
@JJRdec
I have an M1 MacBook here and the test case is running without problems:
$ cat docker-compose.yml
services:
s3mock:
image: adobe/s3mock:3.1.0
environment:
- initialBuckets=bucket1,bucket2
- root=/data
- debug=true
- retainFilesOnExit=true
ports:
- 9090:9090
volumes:
- ./data:/data
$ docker compose up -d
$ truncate -s 1G output.file
$ aws s3 cp ./output.file s3://bucket1/ --endpoint-url=http://localhost:9090
Completed 1.0 GiB/1.0 GiB (134.0 MiB/s) with 1 file(s) remaining
upload: ./output.file to s3://bucket1/output.file
This took 38s.
Without a mounted volume it takes about half that time.
I see that you're running Docker on Windows.
Unfortunately, volumes are known to be super slow in Windows.
Search the web, many people are complaining about this.
Can't do much here in the short term, I'm afraid.
My advice is to use smaller files for testing, or move to a different operating system.
Docker on Mac introduced VirtioFS which is way faster than the previously available formats.
Supposedly, Docker on Linux is also way faster than Docker on Windows.
At some point, I plan to refactor the way we handle multipart uploads (can't fix #1205 without a refactoring), but I can't say yet if that will speed up the process... Currently we persist every part separately and when the multipart upload is completed, we synchronously write all parts to a new file and delete all parts.
The speed of this operation is highly dependent on the speed of the disk.
from s3mock.
Thanks for the detail around this issue. I see now this is more a docker on windows related issue!
For others on Windows we see two workaround options:
- Set the read timeout to infinite (only for local develop of course)
- Use WSL2 and mount a volume on linux subsystem instead on windows.
@afranken I'll leave it up to you if you want to close this issue, thanks!
from s3mock.
@JJRdec sorry I can't do more right now.
I'll close this issue, you should open an issue in the Docker for win repo if you haven't already done so:
https://github.com/docker/for-win
maybe - if enough people ask about it - the Docker team will implement a speed improvement for Windows in the future :)
from s3mock.
Related Issues (20)
- SSLException: Unsupported or unrecognized SSL message HOT 6
- Copying object with new storage class does not appear to save storage class HOT 5
- initialBuckets is ignored if at least one bucket exists with a bucketMetadata.json
- Service: Amazon S3; Status Code: 404; Error Code: 404 HOT 2
- org.eclipse.jetty.server.Request : java.lang.IllegalStateException: Form is larger than max length 200000 HOT 1
- while reading large number of record from s3mock getting Exception( java.io.IOException: Attempted read on closed stream) HOT 2
- S3Mock is incompatible with Spring Boot 3.2.x and Spring Framework 6.x HOT 3
- createBucket fails using S3AsyncClient with NoSuchBucketException HOT 1
- Support S3AsyncClient from software.amazon.awssdk:s3 HOT 2
- Cannot put object acl without a policy HOT 5
- When upgrading to Spring 3.2.2: ClassNotFoundException: javax.xml.bind.annotation.XmlElement HOT 1
- When using async uploads, S3Mock incorrectly stores unsigned, chunked uploads without removing meta (chunk, checksum) information. HOT 14
- Some key names lead to request paths that do not comply to RFC3986, leading to failing requests in S3Mock HOT 7
- When starting S3Mock in Docker, root folder is not accessible from outside of the container HOT 4
- Synchronisation issues HOT 3
- missing namespace xml header. HOT 4
- When using async uploads, S3Mock incorrectly stores unsigned, chunked uploads without removing meta (chunk, checksum) information when the file is >16KB HOT 2
- Validate checksums for uploads and parts
- NumberFormatException when uploading files HOT 5
- Content-MD5 does not match object md5 HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from s3mock.