fsouza / s3-upload-proxy Goto Github PK

Tool for proxying HTTP uploads to S3 and Elemental MediaStore, designed for video live streaming.

License: BSD 2-Clause "Simplified" License

Go 100.00%

golang go s3 livestream mediastore proxy aws-s3 hacktoberfest

s3-upload-proxy's Introduction

s3-upload-proxy

Tool for proxying HTTP uploads to S3 buckets and Elemental MediaStore Containers (added later, bad naming :D). Useful for private network protected environments.

Running locally

Make sure you have latest Go, then make sure you have AWS credentials properly configured (s3-upload-proxy uses the default credential provider chain, so you can use environment variables or file-based configuration).

Having Go and AWS credentials, just set the environment variable BUCKET_NAME, build and start the process:

% export BUCKET_NAME=some-bucket
% go build -o s3-upload-proxy
% ./s3-upload-proxy

If you want to use MediaStore, the environment variable UPLOAD_DRIVER should be set to "mediastore" and BUCKET_NAME must be set to the container name on mediastore:

% export UPLOAD_DRIVER=mediastore BUCKET_NAME=mediastoretest
% go build -o s3-upload-proxy
% ./s3-upload-proxy

Environment variables

s3-upload-proxy configuration's is defined using the following environment variables:

Variable	Default value	Required	Description
UPLOAD_DRIVER	s3	No	Upload driver to use (options are "mediastore" or "s3")
BUCKET_NAME		Yes	Name of the S3 bucket or the mediastore container (depends on the upload driver)
HEALTHCHECK_PATH	/healthcheck	No	Path for healthcheck
HTTP_PORT	80	No	Port to bind (unsigned int)
LOG_LEVEL	debug	No	Logging level
CACHE_CONTROL_RULES		No	JSON array with cache control rules (see below)
MEDIASTORE_CHUNKED_TRANSFER	false	No	Whether to enable chunked transfer with MediaStore for lower latency

Defining cache-control rules

The tool also allow configuration for cache-control rules. The value of the environment variable CACHE_CONTROL_RULES is a JSON array with the rules. An example:

% export CACHE_CONTROL_RULES='[{"regexp":".mp4$","value":"public, max-age=3600"},{"regexp":".ts$","value":"public, max-age=2, s-maxage=999999"},{"regexp":".m3u8$","value":"private"}]'

Notice that the extension must include the dot.

Also available on Docker Hub: https://hub.docker.com/r/fsouza/s3-upload-proxy/.

s3-upload-proxy's People

Contributors

Stargazers

Watchers

Forkers

sherlockholmes mitch-warrenburg alexcheema kevleyski diablowu flavioribeiro andrewulrich

s3-upload-proxy's Issues

Invoke goreleaser from drone

Currently, I manually invoke goreleaser after tagging, but we can do this in the CI.

ValidationException: Invalid request.\n\tstatus code: 400,

I just have to say again, thank you SO MUCH for fixing DASH and for adding in HTTP DELETE.

I'm testing it out and it's just what I needed. However, the most recent build seems to crash after a period of time.

Here is an example crash below. Interestingly it seems to happen after segment 176... to 179... every time.

Does this work with mediastore?

Hey. When I try using this for s3, it works fine. When I try mediastore, the objects get created in the bucket, but they are zero bytes in size.

Is it working for others?

api error InvalidSignatureException: Credential should be scoped to a valid region, not ''. "

Hi,

When I run s3-upload-proxy with MediaStore from the main branch or binary release, I receive the error:

api error InvalidSignatureException: Credential should be scoped to a valid region, not ''. "

If I revert to commit 793d1164921d6e42b4bec26686e76001995f218b, I can properly upload to my container.

I have tried setting environment variables from the command line and setting my ~.aws/config file, but nothing has helped. Do you have any suggestions?

My environment is:

amd64 Ubuntu 18.04 VM
go version go1.16.7 linux/amd64
~/.aws/config
AWS_REGION=us-west-2
AWS_DEFAULT_REGION=us-west-2

I am really excited to use the MEDIASTORE_CHUNKED_TRANSFER feature, which is unavailable on the working commit. I hope we can get this working! :)

Support request cancellation

We need to propagate the request context to be able to support request cancellation.

HTTP DELETE

Hey! Thanks again for swiftly fixing the mediastore bug!

My http PUTs are working great to mediastore :). However, http DELETE seems to be ineffective. Is that something that I should expect to work, or, is that outside of the scope of what this shim does?

Thank you again! This is really helpful!

Dependabot can't parse your go.mod

Dependabot couldn't parse the go.mod found at /go.mod.

The error Dependabot encountered was:

go: github.com/aws/[email protected] requires
	github.com/stretchr/[email protected] requires
	gopkg.in/[email protected]: unrecognized import path "gopkg.in/yaml.v2" (parse https://gopkg.in/yaml.v2?go-get=1: no go-import meta tags ())

View the update logs.