Git Product home page Git Product logo

Comments (5)

arno01 avatar arno01 commented on May 14, 2024 1

@arno01 As the ETAG Handling now is fixed, do you want to give this another try?
We also have a integration test that might be a good starter here:
https://github.com/storj/gateway/testsuite/integration/aws-cli/duplicity.sh

Confirming, the recent storj gateway (v1.1.3) does not have the ETag issue anymore.
Thank you!

from storj.

cpustejovsky avatar cpustejovsky commented on May 14, 2024

Will be looking into this issue this month 👍

from storj.

arno01 avatar arno01 commented on May 14, 2024

These instructions should include how to set up the Storj S3 gateway.

The Storj S3 gateway is not working in smaller environments with low RAM.
I've tried running Storj S3 gateway v0.30.5, and have used aws s3 cp for getting the file of 128MiB size from it. The gateway gets killed shortly:

[Sun Jan 26 12:46:21 2020] Out of memory: Kill process 6933 (gateway_linux_a) score 505 or sacrifice child
[Sun Jan 26 12:46:21 2020] Killed process 6933 (gateway_linux_a) total-vm:685200kB, anon-rss:524600kB, file-rss:0kB, shmem-rss:0kB
[Sun Jan 26 12:46:21 2020] oom_reaper: reaped process 6933 (gateway_linux_a), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

I have 600 MiB RAM free:

# free -mh
              total        used        free      shared  buff/cache   available
Mem:           985M        229M        588M        880K        166M        618M
Swap:            0B          0B          0B

Everything will work as soon as I add more swap, though with the reduced performance, obviously.

And then the gateway returns with the S3DataError: BotoClientError: ETag from S3 did not match computed MD5 error, because the gateway server does not return ETag. I've opened a separate issue for this #3761

An alternative would be to add Storj backend support directly to the duplicity tool.

Update

Here is a python-boto fix (tested v2.44.0 in Ubuntu Bionic) so it would skip ETag checking and let the duplicity finish the backup:

# diff -Nur /usr/lib/python2.7/dist-packages/boto/s3/key.py.orig /usr/lib/python2.7/dist-packages/boto/s3/key.py
--- /usr/lib/python2.7/dist-packages/boto/s3/key.py.orig	2018-12-14 04:44:57.000000000 +0000
+++ /usr/lib/python2.7/dist-packages/boto/s3/key.py	2020-01-26 21:11:19.243840849 +0000
@@ -976,7 +976,9 @@
             # object.
             server_side_encryption_customer_algorithm = response.getheader(
                 'x-amz-server-side-encryption-customer-algorithm', None)
-            if server_side_encryption_customer_algorithm is None:
+            server = response.getheader('Server', None)
+            storj_gateway = server == 'Minio/DEVELOPMENT.GOGET (linux; amd64)'
+            if (server_side_encryption_customer_algorithm is None and not storj_gateway):
                 if self.etag != '"%s"' % md5:
                     raise provider.storage_data_error(
                         'ETag from S3 did not match computed MD5. '

backup to storj using duplicity via storj gateway

$ gateway_linux_amd64 setup
$ gateway_linux_amd64 run

$ export AWS_ACCESS_KEY_ID="REDACTED"
$ export AWS_SECRET_ACCESS_KEY="REDACTED"
$ export PASSPHRASE="REDACTED"

$ duplicity \
  --s3-unencrypted-connection \
  --s3-multipart-chunk-size=50 \
  --volsize=200 \
  --exclude-if-present=do-not-backup \
  --full-if-older-than=1W \
  --sign-key REDACTED \
  --encrypt-key REDACTED \
  -- /srv s3://127.0.0.1:7777/backups/mysite123

from storj.

stefanbenten avatar stefanbenten commented on May 14, 2024

@arno01 As the ETAG Handling now is fixed, do you want to give this another try?
We also have a integration test that might be a good starter here:
https://github.com/storj/gateway/testsuite/integration/aws-cli/duplicity.sh

from storj.

RishmithaRaheem avatar RishmithaRaheem commented on May 14, 2024

If you have any questions about how to get started or want to discuss further, add a comment.

Can somebody send me a document or a youtube video that will get me started on working on this ticket (setup project in local etc.,) ? Im very new to open source contribution hence need some mentorship to start off!

from storj.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.