monder / goofys-docker Goto Github PK

View Code? Open in Web Editor NEW

20.0 3.0 9.0 10 KB

S3 volume for docker based on goofys

License: MIT License

Go 95.63% Shell 4.37%

volume docker volume-plugin s3 aws

goofys-docker's Introduction

goofys-docker is a docker volume plugin wrapper for S3

Overview

The inital idea behind mounting s3 buckets as docker volumes is to provide store for configs and secrets. The volume as per goofys does not have features like randow-write support, unix permissions, caching.

Getting started

Requirements

The docker host should have FUSE support with fusermount cli utility in $PATH

Building

There are prebuilt binaries availble here. If you need to build it yourself there is a helper file build.sh that will run a container that builds the application using go 1.5. Version 1.5 is used to workaround moby/moby#20865

Configuration

Currently there is no support for configuration options, but the defaults are reasonable for most of the cases. The most simple way to configure aws credentials is to use IAM roles to access the bucket for the machine, aws configuration file or ENV variables. The credentials will be used for all buckets mounted by goofys-docker.

Running

./goofys-docker

The socket /run/docker/plugins/goofys.sock will be created to interact with docker. Ownership of the file is root:wheel

Using with docker

Create a new volume by issueing a docker volume command:

docker volume create --name=test-docker-goofys --driver=goofys

That will create a volume connected to test-docker-goofys bucket. The region of the bucket will be autodetected.

Nothing is mounted yet.

Launch the container with test-docker-goofys volume mounted in /home inside the container

docker run -v test-docker-goofys:/home:ro -it busybox sh
/ # cat /home/test
test file content
/ # ^D

It is also possible to mount a subfolder:

docker volume create --name=test-docker-goofys/folder --driver=goofys
docker run docker run -v test-docker-goofys/folder:/home:ro -it busybox sh
/ # cat /home/test
test file content from folder
/ # ^D

If multiple folders are mounted for the single bucket on the same machine, only 1 fuse mount will be created. The mount will be shared by docker containers. It will be unmouned when there be no containers to use it.

License

MIT

goofys-docker's People

Contributors

Stargazers

Watchers

Forkers

anrim djmaze amesianx scottf-tvw cybernagle shubh-k04

goofys-docker's Issues

s3.ERROR code=NoCredentialProviders

Hello, I just downloaded the plugin and running the socket

wget https://github.com/monder/goofys-docker/releases/download/v0.0.1/goofys-docker
sudo ./goofys-docker &

I created a docker volume and attempt to run in busybox and I have this error ("s3-bucket" is the name of my bucket)

docker volume create --name=s3-bucket --driver=goofys
docker run -v s3-bucket:/home -it busybox sh

2024/05/31 17:19:22 Mounting volume s3-bucket on /var/lib/docker-volumes/goofys/s3-bucket
2024/05/31 17:19:22.464853 s3.ERROR code=NoCredentialProviders msg=no valid providers in chain. Deprecated.
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors, err=<nil>

2024/05/31 17:19:22.489969 s3.INFO Switching from region 'eu-west-1' to 'us-east-1'
2024/05/31 17:19:22.491939 s3.ERROR code=NoCredentialProviders msg=no valid providers in chain. Deprecated.
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors, err=<nil>

2024/05/31 17:19:22.491966 main.ERROR Unable to access 's3-bucket': NoCredentialProviders: no valid providers in chain. Deprecated.
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors
docker: Error response from daemon: failed to populate volume: error while mounting volume '': VolumeDriver.Mount: Goofys: initialization failed.

I have my credentials under ~/.aws/credentials in both my user directory and root directory. I also have the env variables available as well. I am able to use the aws cli just fine. Do you know what might be causing this?

Cannot find package "github.com/monder/goofys-docker/internal"

./build.sh
[INFO] Downloading dependencies. Please wait...
[INFO] --> Fetching github.com/aws/aws-sdk-go
[INFO] --> Fetching github.com/coreos/go-systemd
[INFO] --> Fetching github.com/docker/docker
[INFO] --> Fetching github.com/docker/go-connections
[INFO] --> Fetching github.com/docker/go-plugins-helpers
[INFO] --> Fetching github.com/codegangsta/cli
[INFO] --> Fetching github.com/go-ini/ini
[INFO] --> Fetching github.com/jacobsa/fuse
[INFO] --> Fetching github.com/jmespath/go-jmespath
[INFO] --> Fetching github.com/kahing/goofys
[INFO] --> Fetching github.com/kardianos/osext
[INFO] --> Fetching github.com/Microsoft/go-winio
[INFO] --> Fetching github.com/opencontainers/runc
[INFO] --> Fetching github.com/sevlyar/go-daemon
[INFO] --> Fetching github.com/shirou/gopsutil
[INFO] --> Fetching github.com/Sirupsen/logrus
[INFO] --> Fetching golang.org/x/net
[INFO] --> Fetching golang.org/x/sys
[INFO] Setting references.
[INFO] --> Setting version for github.com/codegangsta/cli to 0eb4e0be6c214f8904ef6989b11072c7b897c657.
[INFO] --> Setting version for github.com/coreos/go-systemd to 8aa4c5cfbebc9ae1c3ce10df0dbf75658ae574ef.
[INFO] --> Setting version for github.com/aws/aws-sdk-go to 7bad22e9cff8295edef29844b4daa55ecfb057b9.
[INFO] --> Setting version for github.com/docker/docker to 20f81dde9bd97c86b2d0e33bbbf1388018611929.
[INFO] --> Setting version for github.com/docker/go-connections to 5b7154ba2efe13ff86ae8830a9e7cb120b080d6e.
[INFO] --> Setting version for github.com/go-ini/ini to 12f418cc7edc5a618a51407b7ac1f1f512139df3.
[INFO] --> Setting version for github.com/docker/go-plugins-helpers to b6b774486d89219eba2d2b72ed50d32880167671.
[INFO] --> Setting version for github.com/jmespath/go-jmespath to 0b12d6b521d83fc7f755e7cfc1b1fbdd35a01a74.
[INFO] --> Setting version for github.com/jacobsa/fuse to 0b9db0a706bc0248f8a37cba1dd5372b4c9a8b72.
[INFO] --> Setting version for github.com/shirou/gopsutil to bae75faa5ad1212d3e80f11f5e9cd147c6be9198.
[INFO] --> Setting version for github.com/opencontainers/runc to fdc9fb841ee16714a3fa46bb85d6c42660588e3d.
[INFO] --> Setting version for github.com/kahing/goofys to bda62d906b8d6dc6499be87705466cb6e2575bf2.
[INFO] --> Setting version for github.com/kardianos/osext to 29ae4ffbc9a6fe9fb2bc5029050ce6996ea1d3bc.
[INFO] --> Setting version for github.com/Sirupsen/logrus to cd7d1bbe41066b6c1f19780f895901052150a575.
[INFO] --> Setting version for github.com/sevlyar/go-daemon to 3bf5e993af87194518e81cf9a81c1c53190a8911.
[INFO] --> Setting version for github.com/Microsoft/go-winio to 2255376ea2c82956af54f1eb5a56340a0af29111.
[INFO] --> Setting version for golang.org/x/net to ef00b378c73f107bf44d5c9b69875255ce89b79a.
[INFO] --> Setting version for golang.org/x/sys to d4feaf1a7e61e1d9e79e6c4e76c6349e9cab0a03.
[INFO] Exporting resolved dependencies...
[INFO] --> Exporting github.com/codegangsta/cli
[INFO] --> Exporting github.com/docker/docker
[INFO] --> Exporting github.com/docker/go-connections
[INFO] --> Exporting github.com/docker/go-plugins-helpers
[INFO] --> Exporting github.com/go-ini/ini
[INFO] --> Exporting github.com/coreos/go-systemd
[INFO] --> Exporting github.com/Microsoft/go-winio
[INFO] --> Exporting github.com/kahing/goofys
[INFO] --> Exporting github.com/opencontainers/runc
[INFO] --> Exporting github.com/jacobsa/fuse
[INFO] --> Exporting github.com/jmespath/go-jmespath
[INFO] --> Exporting github.com/aws/aws-sdk-go
[INFO] --> Exporting github.com/sevlyar/go-daemon
[INFO] --> Exporting github.com/kardianos/osext
[INFO] --> Exporting github.com/shirou/gopsutil
[INFO] --> Exporting github.com/Sirupsen/logrus
[INFO] --> Exporting golang.org/x/sys
[INFO] --> Exporting golang.org/x/net
[INFO] Replacing existing vendor dependencies
driver.go:15:2: cannot find package "github.com/monder/goofys-docker/internal" in any of:
/go/src/github.com/monder/goofys-docker/vendor/github.com/monder/goofys-docker/internal (vendor tree)
/usr/local/go/src/github.com/monder/goofys-docker/internal (from $GOROOT)
/go/src/github.com/monder/goofys-docker/internal (from $GOPATH)

fusermount: if you are sure this is safe, use the 'nonempty' mount option.

2017/01/10 02:18:33 Mounting volume binlog-bglcorp-com-au on /var/lib/docker-volumes/goofys/binlog-test-com-au
2017/01/10 02:18:34.434004 s3.ERROR code=incorrect region, the bucket is not in 'eu-west-1' region msg=301 request=

2017/01/10 02:18:34.467775 s3.INFO Switching from region 'eu-west-1' to 'ap-southeast-2'
docker: Error response from daemon: VolumeDriver.Mount: Mount: mount: running fusermount: exit status 1

stderr:
fusermount: mountpoint is not empty
fusermount: if you are sure this is safe, use the 'nonempty' mount option.
See 'docker run --help'.

Documentation?

Hi,

I'm interested in playing with this project, but would need some basic documentation to get oriented started. Is that on your roadmap anytime soon? Or is this repo just for your personal use?