pottava / aws-s3-proxy Goto Github PK
View Code? Open in Web Editor NEWReverse proxy for AWS S3 with basic authentication.
Home Page: https://hub.docker.com/r/pottava/s3-proxy/
License: MIT License
Reverse proxy for AWS S3 with basic authentication.
Home Page: https://hub.docker.com/r/pottava/s3-proxy/
License: MIT License
When I try to add download a file using wget and the file is already present, the server returns 500. I think it makes more sense to return a 204, not an Internal server error
It's a feature enhancement to display folder content or index.html if exists. New variable DIRECTORY_LISTINGS_CHECK_INDEX is introduced to control the behavior.
Given s3 bucket content:
├── folder
├── ── subfolder1
│ └──── index.html
├── ── subfolder2
│ └──── index.html
└── index.html
Desired result if DIRECTORY_LISTINGS=true
and DIRECTORY_LISTINGS_CHECK_INDEX=true
:
GET / returns /index.html content
GET /folder/ returns list of folders - [subfolder1, subfolder2]
GET /folder/subfolder1 returns /folder/subfolder1/index.html content
GET /folder/subfolder2 returns /folder/subfolder2/index.html content
Please add support for arm64 for docker images
Hi,
it looks like aws-s3-proxy cannot proxy encrypted objets on a bucket with Server Side Encryption (SSE) configured. The browser shows the following error from S3
<Error>
<Code>InvalidArgument</Code>
<Message>
Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4.
</Message>
<ArgumentName>Authorization</ArgumentName>
<ArgumentValue>null</ArgumentValue>
<RequestId>3431EC777CCE0481</RequestId>
<HostId>
KB4eUwlajOnuFcHuxFmpGoCy3s+kftrtdlB3JrALmCg9qEw9LvkHmtSkvOk2xhIZmgSOfOzYgrM=
</HostId>
</Error>
Looking at the documentation, AWS Signature Version 4 should be managed by the golang SDK. Am I missing something ?
Hey @pottava ,
It seems this repo is inactive and there are a lot of awesome PRs lined up. I am happy to help out! Let me know what you think.
ben
When gzipping/deflating content, the body payload is properly encoded, but the Content-Length header is not updated.
The Content-Length header should be updated to reflect the length of the payload returned, while it currently contains the length of the payload before the encoding
The dockerfile sets CONTENT_ENCODING to true by default
Hi!
All works well except basic auth. It no ask the pass and I can access to any resource without any creds.
This is my docker-compose.yml
proxy:
image: pottava/s3-proxy
ports:
Logs:
$ docker-compose up --build
Creating proxy ... done
Attaching to proxy
proxy | 2020/01/14 09:45:43 [config] Proxy to mifprojects-english-study-video
proxy | 2020/01/14 09:45:43 [config] AWS Region: eu-central-1
proxy | 2020/01/14 09:45:43 [config] Basic authentication: admin
proxy | 2020/01/14 09:45:43 [service] listening on :80
proxy | 2020/01/14 09:45:50 [myip:33520] 0.114 200 GET /
The fix in #31 doesn't seem to work if you attempt to list a directory (prefix) full of only subdirectories (longer prefixes) as page.Contents
is 0 and it stops after the first page.
Various AWS SDK documentation and examples (e.g. here on GitHub just pass in a function that always returns true
, apparently the SDK will stop calling it after the last page.
A better fix is something like this:
--- a/internal/service/amazon-s3.go
+++ b/internal/service/amazon-s3.go
@@ -37,7 +37,7 @@ func (c client) S3listObjects(bucket, prefix string) (*s3.ListObjectsOutput, err
func(page *s3.ListObjectsOutput, lastPage bool) bool {
result.CommonPrefixes = append(result.CommonPrefixes, page.CommonPrefixes...)
result.Contents = append(result.Contents, page.Contents...)
- return len(page.Contents) == 1000
+ return true
})
return result, err
}
Hello
Very glad I found this project, thanks.
There is not enough LDAP support for base_auth authentication.
Hi there, thanks for creating the v2 release with the fix from PR 31, however I get an error when I try to use it with go get while building a custom Docker image:
$ go get -d -u github.com/pottava/[email protected]
go: finding github.com v2.0.1
go: finding github.com/pottava/aws-s3-proxy v2.0.1
go: finding github.com/pottava v2.0.1
go: finding github.com/pottava/aws-s3-proxy v2.0.1
go get github.com/pottava/[email protected]: github.com/pottava/[email protected]: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v2
The previous version works fine with go get (v1.4.1
). It seems to me based on some Googling that to fix this, Go wants you to modify the module name in go.mod like so:
- module github.com/pottava/aws-s3-proxy
+ module github.com/pottava/aws-s3-proxy/v2
What do you think? An alternative might be to create a v1.5.0
release, since there's not really enough code change to warrant a major version release.
PS, I originally asked this question in Issue 32, but the author closed it (rightfully, it's a new issue).
Getting a nebulous error:
SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
I am sure that I have IAM credentials for the bucket in question. Anyone else run into this issue?
I've added BASIC_AUTH_USER
and BASIC_AUTH_PASS
but when I go to http://localhost:8000
it doesn't ask for credentials - just show the directory listing.
My docker-compose
proxy:
image: pottava/s3-proxy
restart: always
ports:
- 8000:80
environment:
- AWS_REGION=us-east-1
- AWS_ACCESS_KEY_ID=
- AWS_SECRET_ACCESS_KEY=
- AWS_S3_BUCKET=
- ACCESS_LOG=true
- DIRECTORY_LISTINGS=true
- DIRECTORY_LISTINGS_FORMAT=html
- BASIC_AUTH_USER=admin
- BASIC_AUTH_PASS=pass
container_name: proxy
``
There's a good use case for straight directory listings. Prefixes without an immediate index.html key would just list the prefix.
Hey,
are they any plans to provide a helm chart for using this in a Kubernetes cluster?
Best regards
Matthias
I'm testing out the proxy as a simple authentication proxy to get files from S3 regardless av tool used to download.
I do not know if it matters for my specific usecase, but I'd think several users would benefit from better http status code handling.
It now seems like every "error" is returned with 500.
Instead, it would be an improvement if for example "NoSuchKey" errors were returned as 404 back to the client.
NoSuchKey: The specified key does not exist. status code: 404, request id: XXXXXXXXX, host id: YyYYYYYYYYYYYYYYYYYYYYYYYYY
I have no experience with Go, but from the Go SDK documentation it's seems like something similar to this is the way to go:
result, err := svc.AbortMultipartUpload(params)
if err != nil {
// Cast err to awserr.Error to handle specific error codes.
aerr, ok := err.(awserr.Error)
if ok && aerr.Code() == <error code to check for> {
// Specific error code handling
}
return err
}
fmt.Println("AbortMultipartUpload result:")
fmt.Println(result)
Any chance to see new release with this fix?
#31
Are there any gzip options?
Is adding the gzip option possible?
Great work @pottava on the tool, was a life saver when I need to proxy some s3 request.
Please is it possible to list objects more than 1000 in a path?
Thanks
Hello,
I am running the following
docker run -d -p 8080:80 -e AWS_REGION -e AWS_S3_BUCKET -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY pottava/s3-proxy
docker logs 2fffaa2150e0 2023/02/13 04:00:27 [config] Proxy to bni-software-test-reports 2023/02/13 04:00:27 [config] AWS Region: us-east-1 2023/02/13 04:00:27 [service] listening on :80
http://<host-ip>:8080/<bucket-name>/<folder>/<index.html>
Could you take a look and let me know if this url and port is correct? Thank you!
We are trying to deploy this in an EKS cluster, using IAM roles for serviceaccounts, but it does not seem to be able to authenticate to s3 using that role and we get access denied errors for our requests.
Hi!
I'm trying to build aws-s3-proxy from original Dockerfile. I've got eroor on build process:
`Step 1/15 : FROM golang:1.13.7-alpine3.11 AS builder
1.13.7-alpine3.11: Pulling from library/golang
c9b1b535fdd9: Pull complete
cbb0d8da1b30: Pull complete
d909eff28200: Pull complete
f9e7c84be0a1: Pull complete
8a0c202c45df: Pull complete
Digest: sha256:7e8c9c559ca6cf6535f501ca5eec46cae69489b5b8ef9cdec5abcaa5b09e707b
Status: Downloaded newer image for golang:1.13.7-alpine3.11
---> 87eefb76f0a8
Step 2/15 : RUN apk --no-cache add gcc musl-dev git
---> Running in 72db009345b5
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
(1/17) Upgrading musl (1.1.24-r0 -> 1.1.24-r3)
(2/17) Installing libgcc (9.3.0-r0)
(3/17) Installing libstdc++ (9.3.0-r0)
(4/17) Installing binutils (2.33.1-r1)
(5/17) Installing gmp (6.1.2-r1)
(6/17) Installing isl (0.18-r0)
(7/17) Installing libgomp (9.3.0-r0)
(8/17) Installing libatomic (9.3.0-r0)
(9/17) Installing mpfr4 (4.0.2-r1)
(10/17) Installing mpc1 (1.1.0-r1)
(11/17) Installing gcc (9.3.0-r0)
(12/17) Installing nghttp2-libs (1.40.0-r1)
(13/17) Installing libcurl (7.79.1-r0)
(14/17) Installing expat (2.2.9-r1)
(15/17) Installing pcre2 (10.34-r1)
(16/17) Installing git (2.24.4-r0)
(17/17) Installing musl-dev (1.1.24-r3)
Executing busybox-1.31.1-r9.trigger
OK: 128 MiB in 31 packages
Removing intermediate container 72db009345b5
---> 10308d64b5e2
Step 3/15 : RUN go get -u github.com/pottava/aws-s3-proxy
---> Running in 4cc26d32a974
src/github.com/go-openapi/swag/loading.go:44:28: undefined: os.ReadFile
src/github.com/go-openapi/swag/loading.go:50:28: undefined: os.ReadFile
src/github.com/go-openapi/swag/loading.go:119:10: undefined: io.ReadAll
The command '/bin/sh -c go get -u github.com/pottava/aws-s3-proxy' returned a non-zero code: 2
`
Is there opportunity to fix it?
Hi, is it possible to add features for response handling?
We use this proxy to provide a static angular website to the internal vpc only. We need to route back requests with 404 or 403 code to index.html and code 200.
Thanks in advance.
suppose that we have this structure in a the s3 bucket proxied by s3-proxy :
├── folder1
│ ├── f11
│ └── f12
└── folder2
├── f21
└── f22
when DIRECTORY_LISTINGS=true
I expect that http://proxy-endpoint/
list folder1 and folder2, but it returns empty list.
however http://proxy-endpoint/folder1/
list files inside folder1
Thanks
Given s3 bucket content:
├── folder
│ └── index.html
└── index.html
Desired result:
GET /
returns /index.html
content
GET /folder/
returns /folder/index.html
content
GET /folder
returns /folder/index.html
content
Actual result:
GET /
returns /index.html
content
GET /folder/
returns /folder/index.html
content
GET /folder
returns 404
Problem: AWS SDK HTTP client is customly constructed without Proxy
, hence whole thing is not working behind proxy.
Solution:
Add proxy here
Like:
transport := &http.Transport{
Proxy: http.ProxyFromEnvironment,
MaxIdleConns: config.Config.MaxIdleConns,
IdleConnTimeout: config.Config.IdleConnTimeout,
DisableCompression: config.Config.DisableCompression,
TLSClientConfig: tlsCfg,
}
Regardless how ACCESS_LOG is set, aws-s3-proxy logs to stderr only. Documentation appears to show setting ACCESS_LOG=true would log to to stdout.
Can add object caching from S3 to reduce traffic with Amazon?
I've set up the s3 proxy as mentioned in the documentation. When I try to access, I get the following error:
AuthenticationFailed: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:92bba39e-d01e-0128-7334-131fb6000000
Time:2020-04-15T14:44:45.2488458Z
status code: 403, request id: , host id:
Is this a known issue ?
From some of the search on google, it seems so.
https://visdup.blogspot.com/2019/03/azurestorage-blob-server-failed-to.html
Whilst this seems to be reliable when accessed directly, when behind my Kubernetes cluster's Nginx ingress I get a lot of "upstream prematurely closed connection while sending to client" errors and Chrome fails to load a high percentage of the resources. If I request a single file using curl it seems to work. This is using HTTP/1.1.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.