Is it possible to implement snapshot create as new volume (cloning)?
My use case is to provide a base mount point with an immutable installation of the software and creates multiples copies of that base image as snapshot for new mount point.
Now I doing this:

$ docker run --shm-size=1g --name apex --hostname apex -p 1521:1521 -p 8080:8080 --volume-driver=btrfs -v apex-5.1.4:/u01/app/oracle/oradata oracle/database:11.2.0.2-xe
$ docker stop apex
$ docker exec -ti plugin_btrfs buttervolume snapshot apex-5.1.4
$ docker exec -ti plugin_btrfs buttervolume restore "apex-5.1.4@2017-09-14T20:18:29.012797" apex-5.1.4-app1
$ docker exec -ti plugin_btrfs buttervolume restore "apex-5.1.4@2017-09-14T20:18:29.012797" apex-5.1.4-app2
$ docker exec -ti plugin_btrfs buttervolume restore "apex-5.1.4@2017-09-14T20:18:29.012797" apex-5.1.4-app3
$ docker run --shm-size=1g --name apex-app1 --hostname apex -p 15121:1521 -p 8081:8080 --volume-driver=btrfs -v apex-5.1.4-app1:/u01/app/oracle/oradata oracle/database:11.2.0.2-xe
$ docker run --shm-size=1g --name apex-app2 --hostname apex -p 15221:1521 -p 8082:8080 --volume-driver=btrfs -v apex-5.1.4-app2:/u01/app/oracle/oradata oracle/database:11.2.0.2-xe
$ docker run --shm-size=1g --name apex-app3 --hostname apex -p 15321:1521 -p 8083:8080 --volume-driver=btrfs -v apex-5.1.4-app3:/u01/app/oracle/oradata oracle/database:11.2.0.2-xe

I would like to do something like this:

$ docker run --shm-size=1g --name apex --hostname apex -p 1521:1521 -p 8080:8080 --volume-driver=btrfs -v apex-5.1.4:/u01/app/oracle/oradata oracle/database:11.2.0.2-xe
$ docker stop apex
$ docker exec -ti plugin_btrfs buttervolume clone apex-5.1.4 apex-5.1.4-app1
$ docker exec -ti plugin_btrfs buttervolume clone apex-5.1.4 apex-5.1.4-app2
$ docker exec -ti plugin_btrfs buttervolume clone apex-5.1.4 apex-5.1.4-app3
$ docker run --shm-size=1g --name apex-app1 --hostname apex -p 15121:1521 -p 8081:8080 --volume-driver=btrfs -v apex-5.1.4-app1:/u01/app/oracle/oradata oracle/database:11.2.0.2-xe
$ docker run --shm-size=1g --name apex-app2 --hostname apex -p 15221:1521 -p 8082:8080 --volume-driver=btrfs -v apex-5.1.4-app2:/u01/app/oracle/oradata oracle/database:11.2.0.2-xe
$ docker run --shm-size=1g --name apex-app3 --hostname apex -p 15321:1521 -p 8083:8080 --volume-driver=btrfs -v apex-5.1.4-app3:/u01/app/oracle/oradata oracle/database:11.2.0.2-xe

this enhancement will be great to prepare environment for training purpose.
Many thanks, in advance. Marcelo.

One thing that can prevent people from using Buttervolume is that it gives the impression that one needs to have a dedicated BTRFS partition. However it is possible to format a single file and mount it as /var/lib/buttervolume. To ease the initial setup, either add something like a buttervolume init command, or explain how to do in the README file.

We have to make this plugin complaint with v1.12

https://docs.docker.com/engine/extend/plugins_volume/#command-line-changes

Seen in docker daemon logs:
Mar 23 03:53:38 nepri env[959]: time="2018-03-23T03:53:38.297181357Z" level=warning msg="Volume driver btrfs returned an error while trying to query its capabilities, using default capabilities: VolumeDriver.Cap
Mar 23 03:53:38 nepri env[959]:
Mar 23 03:53:38 nepri env[959]:
Mar 23 03:53:38 nepri env[959]:
Mar 23 03:53:38 nepri env[959]: <title>Error: 404 Not Found</title>
Mar 23 03:53:38 nepri env[959]: <style type="text/css">
Mar 23 03:53:38 nepri env[959]: html {background-color: #eee; font-family: sans;}
Mar 23 03:53:38 nepri env[959]: body {background-color: #fff; border: 1px solid #ddd;
Mar 23 03:53:38 nepri env[959]: padding: 15px; margin: 15px;}
Mar 23 03:53:38 nepri env[959]: pre {background-color: #eee; border: 1px solid #ddd; padding: 5px;}
Mar 23 03:53:38 nepri env[959]: </style>
Mar 23 03:53:38 nepri env[959]:
Mar 23 03:53:38 nepri env[959]:
Mar 23 03:53:38 nepri env[959]:

Error: 404 Not Found

Sorry, the requested URL 'http://unix:/run/docker/plugins/btrfs.sock/VolumeDriver.Capabilities&#039;
Mar 23 03:53:38 nepri env[959]: caused an error:

Not found: '/VolumeDriver.Capabilities'

$ docker info
Containers: 17
Running: 13
Paused: 0
Stopped: 4
Images: 428
Server Version: 17.09.1-ce
Storage Driver: overlay
Backing Filesystem: extfs
Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: btrfs local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: v0.13.2 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
seccomp
Profile: default
selinux
Kernel Version: 4.14.19-coreos
Operating System: Container Linux by CoreOS 1632.3.0 (Ladybug)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 62.71GiB
Name: nepri
ID: WHLI:2RVN:SOD5:TUOM:PWUA:WRVT:KHYZ:OQPB:V7SM:GEDG:FN2Q:4YRK
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

hi,
in my use case, i'd like to setup db volume container for QA testing pursue, use the feature of btrfs that each container use the snapshot the volume so that any testing data will be discard once QA finish the test (clean the containers), also the testing data in each container won't pollute the base data in the volume.
so the workflow is such as below:

1. create base data volume
2. create container use that base volume
3. create snapshot of the volume before run the container
4. [perform routine test as QA team do...]
5. remove the container
6. remove the snapshot that container binds.

for now, i was hoping buttervolume can create the snapshot when Mount and remove it when Unmount in automatic manner, however looks like buttervolume does not support this.

i'm thinking to suppose this by introducing a env or conf property, (e.g. btrfs_driver_snapshot_on_mount=true/false), if it's true, then buttervolume will create a snapshot and track it somewhere so that the correct snapshot will be removed when unmount.
what do you think?

see also: https://docs.docker.com/engine/extend/plugins_volume/#volumedrivermount

Ubuntu 16.04
Docker 17.06.2-ce
btrfs on /var/lib/docker

docker run -d --privileged -v /var/lib/docker/volumes:/var/lib/docker/volumes -v /run/docker/plugins:/run/docker/plugins anybox/buttervolume docker volume create -d btrfs testbtrfs Error response from daemon: create testbtrfs: VolumeDriver.Create: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html> <head> <title>Error: 500 Internal Server Error</title> <style type="text/css"> html {background-color: #eee; font-family: sans;} body {background-color: #fff; border: 1px solid #ddd; padding: 15px; margin: 15px;} pre {background-color: #eee; border: 1px solid #ddd; padding: 5px;} </style> </head> <body> <h1>Error: 500 Internal Server Error</h1> <p>Sorry, the requested URL <tt>&#039;http://unix:/run/docker/plugins/btrfs.sock/VolumeDriver.Create&#039;</tt> caused an error:</p> <pre>Internal Server Error</pre> </body> </html>

service log show
`WARNING:root:No config file /etc/buttervolume/schedule.csv
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/buttervolume-1.3.1-py3.5.egg/buttervolume/plugin.py", line 42, in volume_create
btrfs.Subvolume(volpath).create()
File "/usr/local/lib/python3.5/dist-packages/buttervolume-1.3.1-py3.5.egg/buttervolume/btrfs.py", line 40, in create
out = run('btrfs subvolume create "{}"'.format(self.path))
File "/usr/local/lib/python3.5/dist-packages/buttervolume-1.3.1-py3.5.egg/buttervolume/btrfs.py", line 7, in run
stderr=stderr).stdout.decode()
File "/usr/lib/python3.5/subprocess.py", line 398, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'btrfs subvolume create "/var/lib/docker/volumes/testbtrfs"' returned non-zero exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/bottle-0.12.13-py3.5.egg/EGG-INFO/scripts/bottle.py", line 862, in _handle
return route.call(**args)
File "/usr/local/lib/python3.5/dist-packages/bottle-0.12.13-py3.5.egg/EGG-INFO/scripts/bottle.py", line 1740, in wrapper
rv = callback(*a, **ka)
File "/usr/local/lib/python3.5/dist-packages/buttervolume-1.3.1-py3.5.egg/buttervolume/plugin.py", line 44, in volume_create
return json.dumps({'Err': e.strerror})
AttributeError: 'CalledProcessError' object has no attribute 'strerror'
INFO:root:New scheduler job at 2018-01-07 15:54:42.349692
WARNING:root:No config file /etc/buttervolume/schedule.csv
INFO:root:New scheduler job at 2018-01-07 15:55:42.351171
WARNING:root:No config file /etc/buttervolume/schedule.csv
`

The main thread reacts to the SIGTERM signal, and propagates this to the Timer by canceling it. It happened once that the buttervolume docker plugin restart endlessly (for any external reason), and relaunch the schedule endlessly, creating too many snapshots, leading to a btrfs Unallocated space too low.
More investigation need to be done, but the plugin is probably killed instead of terminated, so the Timer thread continues to work, blocking the kill, creating other snapshots, etc. It's probaby worth considering reacting to other signals such as SIGKILL.

while restoring a volume (which happens when moving volume from nodes in mlfmonde/cluster) volumes labels are lost.

To reproduce:

$ docker volume create -d anybox/buttervolume:latest --label test=True test
gle@rbx-6-any-3 ~ $ docker volume ls
DRIVER                       VOLUME NAME
anybox/buttervolume:latest   test
$ docker volume inspect test
[
    {
        "CreatedAt": "0001-01-01T00:00:00Z",
        "Driver": "anybox/buttervolume:latest",
        "Labels": {
            "test": "True"
        },
        "Mountpoint": "/var/lib/buttervolume/volumes/test",
        "Name": "test",
        "Options": {},
        "Scope": "local"
    }
]
$ buttervolume snapshot test
$ docker volume rm test
$ docker volume inspect test
[]
Error: No such volume: test
buttervolume restore test
$ docker volume inspect test
[
    {
        "CreatedAt": "0001-01-01T00:00:00Z",
        "Driver": "anybox/buttervolume:latest",
        "Labels": null,
        "Mountpoint": "/var/lib/buttervolume/volumes/test",
        "Name": "test",
        "Options": null,
        "Scope": "local"
    }
]

Found an error in the display of an exception.
To reproduce it : start buttervolume with a wrong mount of /var/lib/buttervolume so that it is not a BTRFS filesystem or subvolume. Then try to create a volume with docker volume create -d btrfs truuc :

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/buttervolume-2.0.0-py3.5.egg/buttervolume/plugin.py", line 64, in volume_create
    btrfs.Subvolume(volpath).create()
  File "/usr/local/lib/python3.5/dist-packages/buttervolume-2.0.0-py3.5.egg/buttervolume/btrfs.py", line 40, in create
    out = run('btrfs subvolume create "{}"'.format(self.path))
  File "/usr/local/lib/python3.5/dist-packages/buttervolume-2.0.0-py3.5.egg/buttervolume/btrfs.py", line 7, in run
    stderr=stderr).stdout.decode()
  File "/usr/lib/python3.5/subprocess.py", line 398, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'btrfs subvolume create "/var/lib/buttervolume/volumes/truuc"' returned non-zero exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/bottle-0.12.13-py3.5.egg/EGG-INFO/scripts/bottle.py", line 862, in _handle
    return route.call(**args)
  File "/usr/local/lib/python3.5/dist-packages/bottle-0.12.13-py3.5.egg/EGG-INFO/scripts/bottle.py", line 1740, in wrapper
    rv = callback(*a, **ka)
  File "/usr/local/lib/python3.5/dist-packages/buttervolume-2.0.0-py3.5.egg/buttervolume/plugin.py", line 66, in volume_create
    return json.dumps({'Err': e.stderr})
  File "/usr/lib/python3.5/json/__init__.py", line 230, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python3.5/json/encoder.py", line 198, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.5/json/encoder.py", line 256, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.5/json/encoder.py", line 179, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: b'ERROR: not a btrfs filesystem: /var/lib/buttervolume/volumes\n' is not JSON serializable

Hello, what is the relationship between buttervolume and the built-in BTRFS storage driver ?
I'm a bit confused because in the Running the plugin section --volume-driver=anybox/buttervolume:latest is used while in Creating and deleting volumes it is volume-driver=btrfs.
Thanks in advance for the clarification :)

The docs says the configuration and schedules reside in /etc/buttervolume/schedule.csv and /etc/buttervolume/config.ini. As far as I find out this is just the path inside the plugin container, not on the host machine. Is this correct?
I think its better to tell the user to put the files inside /var/lib/buttervolume/config.

In the readme it says, I can put configuration settings into

/etc/buttervolume/config.ini file inside the container or /var/lib/buttervolume/config/config.ini on the host

However, I tried changing my snapshots and volumes path in /var/lib/buttervolume/config/config.ini but they still end up in the default place.

Quickly searching through the code also shows no references to that file, it might be an error in the readme.

However, configuring through a file on the host would be useful to me. I could not figure out the other option of setting environment variables for configuration.

I'm curious. Does this volume driver creates block devices on the fly and all?

Can be usefull during some incident-related activities.
Something like buttervolume schedule disable buttervolume schedule enable

At least while purging snapshots we are reading date that were generated by datetime.isoformat if we are unlucky we get the time with 0 microsecond, in that case the format is a bit different:

(Pdb) datetime(2017,4,3,21,24,0,101).isoformat()
'2017-04-03T21:24:00.000101'
(Pdb) datetime(2017,4,3,21,24,0,0).isoformat()
'2017-04-03T21:24:00'

So while parsing that string we can get a ValueError exception:

(Pdb) datetime.strptime("buttervolume-test-eaf7bb4c65ed4f2ea9c30ef6c33f0f00@2017-04-03T21:24:00@123".split('@')[1], "%Y-%m-%dT%H:%M:%S.%f")
*** ValueError: time data '2017-04-03T21:24:00' does not match format '%Y-%m-%dT%H:%M:%S.%f'
(Pdb) datetime.strptime("buttervolume-test-eaf7bb4c65ed4f2ea9c30ef6c33f0f00@2017-04-03T21:24:00.12@123".split('@')[1], "%Y-%m-%dT%H:%M:%S.%f")
datetime.datetime(2017, 4, 3, 21, 24, 0, 120000)

I am currently using an older version of Buttervolume inside a CI build agent, using docker exec buttervolume buttervolume. That works perfectly.

Is there any way to use the newer, managed plugin based version this way? docker-runc doesn't seem to work, as it doesn't detect that the plugin is running. /var/run/docker.sock and /run/docker/plugins/runtime-root/plugins.moby are mapped from the host.

On the host:

# drunc list
ID                                                                 PID         STATUS      BUNDLE                                                                                                                                       CREATED                          OWNER
abf6245ea65ee121ff48c30f99c283dac49d225221579ee4a140b7d8a843f200   19607       running     /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/plugins.moby/abf6245ea65ee121ff48c30f99c283dac49d225221579ee4a140b7d8a843f200   2018-11-01T16:26:26.605625462Z   root

Inside the CI container:

# drunc list
ID                                                                 PID         STATUS      BUNDLE                                                                                                                                       CREATED                          OWNER
abf6245ea65ee121ff48c30f99c283dac49d225221579ee4a140b7d8a843f200   0           stopped     /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/plugins.moby/abf6245ea65ee121ff48c30f99c283dac49d225221579ee4a140b7d8a843f200   2018-11-01T16:26:26.605625462Z   root

In the purge pattern, 'm' is for minutes, not months.
However there is no error when launching a purge or schedule command with the following pattern: 1w:1m:1y

It should fail after checking that 1m is smaller than 1w. Each pattern item should be smaller than the preceding

Got a exception while creating a volume through docker-compose

cow = opts.get(\"copyonwrite\", \"true\").lower()" plugin=d6ead4dd089e1fd02dfbe65eb6b68377350528d8ffd967459e222f686ae075eb
AttributeError: 'NoneType' object has no attribute 'get'" plugin=d6ead4dd089e1fd02dfbe65eb6b68377350528d8ffd967459e222f686ae075eb

Hi,
I'm currently trying to install buttervolume on a dedicated server.
I followed the install guide, I can now create volume using the anybox/buttervolume:latest driver but I have some problem with the buttervolume command.
I made aliases using

alias drunc="sudo runc --root /run/docker/plugins/runtime-root/plugins.moby/"
alias buttervolume="drunc exec -t $(drunc list|tail -n+2|awk '{print $1}') buttervolume"

But when I execute, for example :

buttervolume scheduled

That return

ERRO[0000] exec failed: container "1ed2837d4ffbddc9994111b57cbfd72a972a64c77a5211099349b8f986af1bcc" does not exist

I noticed that 1ed2837d4ffbddc9994111b57cbfd72a972a64c77a5211099349b8f986af1bcc is the plugin id but I don't understand why this error is talking about container.

Thank for you help!
Have a nice day

Debian : 10.7
Kernel : 4.19.0-12-amd64
Docker : 20.10.0, build 7287ab3
Buttervolume config : default

I'm not able to get buttervolume to work properly when the docker daemon is in "userns-remap" mode. I've mapped "root" in my containers to userid 10,000. Volume creation happens but then when it tries to map it into the /var/lib/docker/10000.10000/btrfs/snapshots it seems to lose the mapping. Instead, the volume ends up with nobody:nogroup as the owner.

Here is the output:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/docker/10000.10000/plugins/a611740455740ff69f9721be7f755ebf100baf337248815c37faf52405f12c02/propagated-mount/volumes/test2\\\" to rootfs \\\"/var/lib/docker/10000.10000/btrfs/subvolumes/a1a2b78e7e08eeb182408aceaa5791a419b8c491e80de0969d93e85eff0c0515\\\" at \\\"/home\\\" caused \\\"stat /var/lib/docker/10000.10000/plugins/a611740455740ff69f9721be7f755ebf100baf337248815c37faf52405f12c02/propagated-mount/volumes/test2: permission denied\\\"\"": unknown. ERRO[0001] error waiting for container: context canceled

On each new connection a process is spawned by sshd even the user is not connected.

I guess the zombie process is hold by the docker entrypoint itself ?!

• We should offer a way to let custom sshd config by adminsys binding /etc/ssh/sshd_config to allow listening sshd only on secure network interface
• We should bind sshd logs somewhere to let adminsys to configure some fail2ban like apps
• We must close zombie processus properly
• We should build this image automatically in order to upgrade software inside

We may want to split those points in different tickets !

hi,
I ran buttervolume from the pre-baked docker image:

root@ca4c855d6f0d:/# buttervolume snapshots
DEBUG:requests.packages.urllib3.connectionpool:http://localhost:None "POST /VolumeDriver.Snapshot.List HTTP/1.1" 200 28

1. http://localhost:None, the port "None" looks confusing, though the result is good
2. can we disable the DEBUG log by default?
   thx

docker compose up results in the following when creating a new volume in 3.9:

Oct 11 08:08:06 servilix dockerd[8809]: time="2022-10-11T08:08:06Z" level=error msg="Traceback (most recent call last):" plugin=37f2845ef60d5735e4dba3401fcf9e603960f694ca3f2917aec557e66d72524e
Oct 11 08:08:06 servilix dockerd[8809]: time="2022-10-11T08:08:06Z" level=error msg="  File \"/usr/local/lib/python3.9/dist-packages/bottle-0.12.23-py3.9.egg/EGG-INFO/scripts/bottle.py\", line 876, in _handle" plugin=37f2845ef60d5735e4dba3401fcf9e603960f694ca3f2917aec557e66d72524e
Oct 11 08:08:06 servilix dockerd[8809]: time="2022-10-11T08:08:06Z" level=error msg="    return route.call(**args)" plugin=37f2845ef60d5735e4dba3401fcf9e603960f694ca3f2917aec557e66d72524e
Oct 11 08:08:06 servilix dockerd[8809]: time="2022-10-11T08:08:06Z" level=error msg="  File \"/usr/local/lib/python3.9/dist-packages/bottle-0.12.23-py3.9.egg/EGG-INFO/scripts/bottle.py\", line 1756, in wrapper" plugin=37f2845ef60d5735e4dba3401fcf9e603960f694ca3f2917aec557e66d72524e
Oct 11 08:08:06 servilix dockerd[8809]: time="2022-10-11T08:08:06Z" level=error msg="    rv = callback(*a, **ka)" plugin=37f2845ef60d5735e4dba3401fcf9e603960f694ca3f2917aec557e66d72524e
Oct 11 08:08:06 servilix dockerd[8809]: time="2022-10-11T08:08:06Z" level=error msg="  File \"/usr/local/lib/python3.9/dist-packages/buttervolume-3.8-py3.9.egg/buttervolume/plugin.py\", line 62, in new_handler" plugin=37f2845ef60d5735e4dba3401fcf9e603960f694ca3f2917aec557e66d72524e
Oct 11 08:08:06 servilix dockerd[8809]: time="2022-10-11T08:08:06Z" level=error msg="    resp = json.dumps(handler(req))" plugin=37f2845ef60d5735e4dba3401fcf9e603960f694ca3f2917aec557e66d72524e
Oct 11 08:08:06 servilix dockerd[8809]: time="2022-10-11T08:08:06Z" level=error msg="  File \"/usr/local/lib/python3.9/dist-packages/buttervolume-3.8-py3.9.egg/buttervolume/plugin.py\", line 89, in volume_create" plugin=37f2845ef60d5735e4dba3401fcf9e603960f694ca3f2917aec557e66d72524e
Oct 11 08:08:06 servilix dockerd[8809]: time="2022-10-11T08:08:06Z" level=error msg="    option_copyonwrite = opts[\"copyonwrite\"].lower()" plugin=37f2845ef60d5735e4dba3401fcf9e603960f694ca3f2917aec557e66d72524e
Oct 11 08:08:06 servilix dockerd[8809]: time="2022-10-11T08:08:06Z" level=error msg="TypeError: 'NoneType' object is not subscriptable" plugin=37f2845ef60d5735e4dba3401fcf9e603960f694ca3f2917aec557e66d72524e
Oct 11 08:08:06 servilix dockerd[8809]: time="2022-10-11T08:08:06.813106506Z" level=error msg="Handler for POST /v1.41/volumes/create returned error: create homeassistant_haconfig: VolumeDriver.Create: \n    <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n    <html>\n        <head>\n            <title>Error: 500 Internal Server Error</title>\n            <style type=\"text/css\">\n              html {background-color: #eee; font-family: sans;}\n              body {background-color: #fff; border: 1px solid #ddd;\n                    padding: 15px; margin: 15px;}\n              pre {background-color: #eee; border: 1px solid #ddd; padding: 5px;}\n            </style>\n        </head>\n        <body>\n            <h1>Error: 500 Internal Server Error</h1>\n            <p>Sorry, the requested URL <tt>&#039;http://waitress.invalid:/run/docker/plugins/btrfs.sock/VolumeDriver.Create&#039;</tt>\n               caused an error:</p>\n            <pre>Internal Server Error</pre>\n        </body>\n    </html>\n"

Adding driver_opts in the compose file solves this, but shouldn't be necessary.

volumes:
  haconfig:
    driver: anybox/buttervolume:latest
    driver_opts:
      option_copyonwrite: "true"

I've tested when /usr/lib/buttervolume is mounted with compress (also tested with compress-force) options. Everything written to plugin-backed volume is not compressed (according to btrfs usage stats and compsize utility)

In the same time, when I create a native btrfs volume at the same device, mount it locally completely unrelated to docker, and write large files — compression works as expected (according to btrfs and compsize).

After having a slave node disconnected during one day, the following snapshots appear when the slave is reconnected. All these snapshots are not necessary. Even if the purge process will erase most, we have to check how they appear.
Maybe avoid to reschedule a replication if another one for the same volume is not finished?

cluster_caddy_ssl@2018-06-14T11:10:25.549293
cluster_caddy_ssl@2018-06-14T11:35:07.725268
cluster_caddy_ssl@2018-06-14T11:39:03.489281
cluster_caddy_ssl@2018-06-14T11:39:25.813912
cluster_caddy_ssl@2018-06-14T11:54:27.434850
cluster_caddy_ssl@2018-06-14T12:04:00.185161
cluster_caddy_ssl@2018-06-14T12:04:37.264123
cluster_caddy_ssl@2018-06-14T12:19:38.738478
cluster_caddy_ssl@2018-06-14T12:34:40.191789
cluster_caddy_ssl@2018-06-14T12:49:41.919209
cluster_consul_docker_cfg@2018-06-14T11:10:24.695190
cluster_consul_docker_cfg@2018-06-14T11:35:10.019119
cluster_consul_docker_cfg@2018-06-14T11:39:05.787531
cluster_consul_docker_cfg@2018-06-14T11:39:25.127532
cluster_consul_docker_cfg@2018-06-14T11:54:26.636645
cluster_consul_docker_cfg@2018-06-14T12:03:59.193377
cluster_consul_docker_cfg@2018-06-14T12:04:36.565095
cluster_consul_docker_cfg@2018-06-14T12:19:38.045933
cluster_consul_docker_cfg@2018-06-14T12:34:39.497284
cluster_consul_docker_cfg@2018-06-14T12:49:41.143325

During a reboot or a restart of the Docker daemon, the buttervolume container isn't always the first to be started.

That means that Docker will create local storage volumes for containers that are started before buttervolume is up and running.

Is there any way to avoid this?

Version 18.9.0 of Docker CE no longer includes the docker-runc command. In stead, the containerd.io package is installed as a dependency, which contains the runc command.

This means that the Bash aliases in the readme no longer work. They also have another problem: The command inside the bacticks is run immediately when the alias is defined, so if it's put into .bash_aliases, you get a password prompt when starting a new shell.

I have had success putting the following in .bash_aliases:

function drunc () {
  sudo runc --root /run/docker/plugins/runtime-root/plugins.moby/ $@
}

function buttervolume () {
  drunc exec -t $(drunc list|tail -n+2|awk '{print $1}') buttervolume $@
}

is there a better way than rsync, cp or mv after creating a new volume?

The documentation has the following example:

docker run --privileged -v /var/lib/docker/volumes:/var/lib/docker/volumes -v /run/docker/plugins:/run/docker/plugins anybox/buttervolume

However, running docker exec buttervolume buttervolume snapshot some_volume exits with error code 1, as /var/lib/docker/snapshots isnt a btrfs volume.

The following works:

docker run --privileged \
-v /var/lib/docker/volumes:/var/lib/docker/volumes \
-v /var/lib/docker/snapshots:/var/lib/docker/snapshots \
-v /run/docker/plugins:/run/docker/plugins \
anybox/buttervolume

Should /var/lib/docker/received also be added the same way?

Is it possible to change the port used for SSH form the default 1122?

This file seems to indicate setting the environment variable SSH_PORT should work, but I tried:

docker plugin set anybox/buttervolume SSH_PORT=3181

which gives me:

Error response from daemon: setting "SSH_PORT" not found in the plugin configuration

After running the plugin in front mode with buttervolume run, I end up with some kind of freeze when doing docker volume ls. It's due to the /run/docker/plugins/btrfs.sock being created. After deleting it docker works again, but it should not be a problem.

Hi, many thanks for the great idea so far. Iam loving btrfs and doing scheduled stuff like snapshotting, syncing, etc. directly with docker is a really great addition.

After installing the plugin like described there is no socket inside /var/run/docker/plugins/. The plugin created a socket at some subpath, e.x. /var/run/docker/plugins/4537bd7d3e863809942031a9f87c54c2f5c38922d4286c1d9ebbcea3d6df4f22/btrfs.sock.
I could bring the plugin to work by creating a symbolic link ln -s /4537bd7d3e863809942031a9f87c54c2f5c38922d4286c1d9ebbcea3d6df4f22/btrfs.sock /var/run/docker/plugins/btrfs.sock and manually starting the daemon buttervolume run.

Running buttervolume run printed the following:

Starting scheduler job every 60s
Listening to requests...
Serving on http://unix:/run/docker/plugins/btrfs.sock
INFO:root:New scheduler job at 2018-06-06 12:22:20.475458

The buttervolume egg v. 1.3 (https://pypi.python.org/pypi/buttervolume/1.3) seems to broken.

# pip3 install buttervolume 
Collecting buttervolume
  Downloading buttervolume-1.3.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-c_alp5nk/buttervolume/setup.py", line 17, in <module>
        + open('CHANGES.rst').read(),
    FileNotFoundError: [Errno 2] No such file or directory: 'CHANGES.rst'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-c_alp5nk/buttervolume/

Guess the CHANGES.rst file is missing in the package.

as an alternative to sending the incremental snapshots to another host, they could be sent to an object storage

I'm getting an error when installing or enabling the buttervolume plugin. I've tried to install the plugin through the repository:

> docker plugin install anybox/buttervolume
...
Error response from daemon: dial unix /run/docker/plugins/<uuid>/btrfs.sock: connect: no such file or directory

The plugin was installed but cannot be enabled.

> docker plugin ls
ID          NAME                         DESCRIPTION                      ENABLED
<ID>        anybox/buttervolume:latest   BTRFS Volume Plugin for Docker   false

Trying to enable it returns the same error:

> docker plugin enable anybox/buttervolume:latest 
Error response from daemon: dial unix /run/docker/plugins/<uuid>/btrfs.sock: connect: no such file or directory

Apparently the btrfs.sock file isn't created during the installation.

amazing work!

I think could be very interesting to add a clustered storage system with buttervolume

to sync volumes every X minutes between nodes of a cluster.
you can control source/destination of the sync checking which host have the vol mounted and control the replication factor(x2, x3)
currently there are solutions for clustered storage that have same features, but with buttervolume you can have async replication that is the only way when you want to replicate in a high latency scenario.

It happened a few times that we find a subvolume under a volume directory, instead of the volume itself being the suvolume.
We suspect this comes from either a restore on an existing volume or something else we have to reproduce.