Git Product home page Git Product logo

rest-server's Introduction

Rest Server

Status badge for CI tests Go Report Card GoDoc License Powered by

Rest Server is a high performance HTTP server that implements restic's REST backend API. It provides secure and efficient way to backup data remotely, using restic backup client via the rest: URL.

Requirements

Rest Server requires Go 1.18 or higher to build. The only tested compiler is the official Go compiler. Building server with gccgo may work, but is not supported.

The required version of restic backup client to use with rest-server is v0.7.1 or higher.

Build

For building the rest-server binary run CGO_ENABLED=0 go build -o rest-server ./cmd/rest-server

Usage

To learn how to use restic backup client with REST backend, please consult restic manual.

$ rest-server --help

Run a REST server for use with restic

Usage:
  rest-server [flags]

Flags:
      --append-only            enable append only mode
      --cpu-profile string     write CPU profile to file
      --debug                  output debug messages
  -h, --help                   help for rest-server
      --htpasswd-file string   location of .htpasswd file (default: "<data directory>/.htpasswd")
      --listen string          listen address (default ":8000")
      --log filename           write HTTP requests in the combined log format to the specified filename
      --max-size int           the maximum size of the repository in bytes
      --no-auth                disable .htpasswd authentication
      --no-verify-upload       do not verify the integrity of uploaded data. DO NOT enable unless the rest-server runs on a very low-power device
      --path string            data directory (default "/tmp/restic")
      --private-repos          users can only access their private repo
      --prometheus             enable Prometheus metrics
      --prometheus-no-auth     disable auth for Prometheus /metrics endpoint
      --tls                    turn on TLS support
      --tls-cert string        TLS certificate path
      --tls-key string         TLS key path
  -v, --version                version for rest-server

By default the server persists backup data in the OS temporary directory (/tmp/restic on Linux/BSD and others, in %TEMP%\\restic in Windows, etc). If rest-server is launched using the default path, all backups will be lost. To start the server with a custom persistence directory and with authentication disabled:

rest-server --path /user/home/backup --no-auth

To authenticate users (for access to the rest-server), the server supports using a .htpasswd file to specify users. By default, the server looks for this file at the root of the persistence directory, but this can be changed using the --htpasswd-file option. You can create such a file by executing the following command (note that you need the htpasswd program from Apache's http-tools). In order to append new user to the file, just omit the -c argument. Only bcrypt and SHA encryption methods are supported, so use -B (very secure) or -s (insecure by today's standards) when adding/changing passwords.

htpasswd -B -c .htpasswd username

If you want to disable authentication, you must add the --no-auth flag. If this flag is not specified and the .htpasswd cannot be opened, rest-server will refuse to start.

NOTE: In older versions of rest-server (up to 0.9.7), this flag does not exist and the server disables authentication if .htpasswd is missing or cannot be opened.

By default the server uses HTTP protocol. This is not very secure since with Basic Authentication, user name and passwords will be sent in clear text in every request. In order to enable TLS support just add the --tls argument and add a private and public key at the root of your persistence directory. You may also specify private and public keys by --tls-cert and --tls-key.

Signed certificate is normally required by the restic backend, but if you just want to test the feature you can generate password-less unsigned keys with the following command:

openssl req -newkey rsa:2048 -nodes -x509 -keyout private_key -out public_key -days 365 -addext "subjectAltName = IP:127.0.0.1,DNS:yourdomain.com"

Omit the IP:127.0.0.1 if you don't need your server be accessed via SSH Tunnels. No need to change default values in the openssl dialog, hitting enter every time is sufficient. To access this server via restic use --cacert public_key, meaning with a self-signed certificate you have to distribute your public_key file to every restic client.

The --append-only mode allows creation of new backups but prevents deletion and modification of existing backups. This can be useful when backing up systems that have a potential of being hacked.

To prevent your users from accessing each others' repositories, you may use the --private-repos flag which grants access only when a subdirectory with the same name as the user is specified in the repository URL. For example, user "foo" using the repository URLs rest:https://foo:pass@host:8000/foo or rest:https://foo:pass@host:8000/foo/ would be granted access, but the same user using repository URLs rest:https://foo:pass@host:8000/ or rest:https://foo:pass@host:8000/foobar/ would be denied access. Users can also create their own subrepositories, like /foo/bar/.

Rest Server uses exactly the same directory structure as local backend, so you should be able to access it both locally and via HTTP, even simultaneously.

Systemd

There's an example systemd service file included with the source, so you can get Rest Server up & running as a proper Systemd service in no time. Before installing, adapt paths and options to your environment.

Docker

Rest Server works well inside a container, images are published to Docker Hub.

Start server

You can run the server with any container runtime, like Docker:

    docker pull restic/rest-server:latest
    docker run -p 8000:8000 -v /my/data:/data --name rest_server restic/rest-server

Note that:

  • contrary to the defaults of rest-server, the persistent data volume is located to /data.
  • By default, the image uses authentication. To turn it off, set environment variable DISABLE_AUTHENTICATION to any value.
  • By default, the image loads the .htpasswd file from the persistent data volume (i.e. from /data/.htpasswd). To change the location of this file, set the environment variable PASSWORD_FILE to the path of the .htpasswd file. Please note that this path must be accessible from inside the container and should be persisted. This is normally done by bind-mounting a path into the container or with another docker volume.
  • It's suggested to set a container name to more easily manage users (--name parameter to docker run).
  • You can set environment variable OPTIONS to any extra flags you'd like to pass to rest-server.

Customize the image

The published image is built from the Dockerfile available on this repository, which you may use as a basis for building your own customized images.

    git clone https://github.com/restic/rest-server.git 
    cd rest-server
    docker build -t restic/rest-server:latest .

Manage users

Add user
docker exec -it rest_server create_user myuser

or

docker exec -it rest_server create_user myuser mypassword
Delete user
docker exec -it rest_server delete_user myuser

Prometheus support and Grafana dashboard

The server can be started with --prometheus to expose Prometheus metrics at /metrics. If authentication is enabled, this endpoint requires authentication for the 'metrics' user, but this can be overridden with the --prometheus-no-auth flag.

This repository contains an example full stack Docker Compose setup with a Grafana dashboard in examples/compose-with-grafana/.

Why use Rest Server?

Compared to the SFTP backend, the REST backend has better performance, especially so if you can skip additional crypto overhead by using plain HTTP transport (restic already properly encrypts all data it sends, so using HTTPS is mostly about authentication).

But, even if you use HTTPS transport, the REST protocol should be faster and more scalable, due to some inefficiencies of the SFTP protocol (everything needs to be transferred in chunks of 32 KiB at most, each packet needs to be acknowledged by the server).

One important safety feature that Rest Server adds is the optional ability to run in append-only mode. This prevents an attacker from wiping your server backups when access is gained to the server being backed up.

Finally, the Rest Server implementation is really simple and as such could be used on the low-end devices, no problem. Also, in some cases, for example behind corporate firewalls, HTTP/S might be the only protocol allowed. Here too REST backend might be the perfect option for your backup needs.

Contributors

Contributors are welcome, just open a new issue / pull request.

rest-server's People

Contributors

ae-govau avatar andreaso avatar buschjost avatar cgonzalez avatar deajan avatar dependabot[bot] avatar dwmunster avatar enrico204 avatar fd0 avatar jinnko avatar jsbergbau avatar juergenhoetzel avatar lgommans avatar lwis avatar mebus avatar mholt avatar michaeleischer avatar networkexception avatar pagdot avatar ph818 avatar qbit avatar r3dey3 avatar rafacouto avatar rawtaz avatar telenieko avatar tim-seoss avatar ubitux avatar wojas avatar wscott avatar zcalusic avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

rest-server's Issues

Integrate with syslog

Hi,

The title is pretty self-explanatory: would it be possible to integrate with syslog?

Our logs are growing pretty fast so we need rotation, among other nice features of syslog :)

TLS hot reload

Hello,
thank you for rest-server! I'm using it with letsencrypt-issued TLS certificates and I'm interested to know if hot-reload of certificates is planned or something that rest-server could get? For example upon SIGHUP (#80) or automatically e.g. with https://github.com/dyson/certman.

Make importable

Hi, me again. 😄 I'm this 👌 close to having this implemented as a Caddy plugin, which will allow you to add the restic directive to your Caddyfile, so you can use HTTPS to transfer files instead of HTTP, without having to worry about certificates.

My motivation is to avoid relying on SFTP which gets slower and slower forever during the transfer, until the transfer takes weeks instead of hours. My hope and expectation is that HTTPS won't do this because it doesn't have the crazy flow control that SSH + SFTP layers on top of TCP's. Or something.

Anyway, one benefit of this being a Caddy plugin is that Caddy can just run indefinitely and keep the
server going (instead of having to restart it to reload renewed certs), and also serve other websites on the same port too. (One alternative you could do is use autocert into your program, to get this feature without integrating with Caddy. Autocert requires only a couple lines of code.)

However, in order for me to use this as a library, I need to be able to import it. Specifically, I think I need access to the setupMux() function. And if the mux and handlers and stuff could be moved into a non-main package, then I could import it.

So, in summary, I'm requesting:

  1. Move as much as possible out of the main package so it can be imported
  2. Export setupMux() so it can be used by other packages in their own handlers

I think I could submit a PR if you're OK with this.

rest-server with non-local (Google Cloud, B2, S3..) backend

Just starting with restic (which actually seems to be "too good to be true"! :-) so maybe I missed a feature. Otherwise: Feature request!?

Having been to all kinds of bad situations, I consider the threat model in restic/restic#784 very real:
"Servers-to-be-backuped" (STBB) are taken over by an adversary trying to delete backups for extortion purposes.
I'd extend it to "admin fat fingers" (accidentally deleting backups from the STBB). So I like the idea of --append-only with a dedicated rest-server (RS) that doesn't share admin credentials with STBBs.

I would like to let RS store its data on the cloud backends (remote storage) as implemented by restic's client itself: Google Cloud Storage, Backbaze B2, AWS S3, SFTP... (personally interested in GCS and B2)
Why?

  • I don't have an RS with adequate storage.
  • I don't want to maintain libraries and credentials for remote storage on all STBB.
  • I would like to switch remote storage without reconfiguring a dozen STBBs but only one RS.

Is this possible with rest-server?

Philipp

Make configuration not global

I'm using this as a library and would like to start REST servers for different data directories. Currently, the Config is global and cannot be customized per-mux.

I will submit a PR for this. Do you have any design suggestions for making this change?

File open issue

Hi,

while trying to backup via restic I get the following error on a synology NAS (arm device)

POST /data/84da53f26249f5774d99ccafecf244e770a6b37da22faa7ffbb817d8ff641224
SaveBlob()
open /volume1/Public/data/84/84da53f26249f5774d99ccafecf244e770a6b37da22faa7ffbb817d8ff641224: no such file or directory

I traced it back to handlers.go:239

tf, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_EXCL, 0600)

and reason could either be that the folder is not created (it isn't there) or that the permissions can't be set.

Did anyone have a similar issue?

Koen

server response unexpected: 500 Internal Server Error (500)

Versions

restic version : 0.7.0
rest-server version : 0.9.3

Steps to reproduce:

Step1. Create directory rest-test

root@backup-server-restic:/backup# mkdir rest-test

Step2. Initialize it as local repo

root@backup-server-restic:/backup# /usr/local/bin/restic -r /backup/rest-test init                             
enter password for new backend:                      
enter password again:                                
created restic backend f8ff0dbe7b at /backup/rest-test                                                    

Please note that knowledge of your password is required to access                                         
the repository. Losing your password means that your data is                                              
irrecoverably lost. 

Step3. Run rest-server

root@backup-server-restic:/backup# /usr/local/bin/rest-server --path /backup/rest-test/ --listen ":8002" --debug
rest-server 0.9.3 compiled with go1.8.1 on linux/amd64
Data directory: /backup/rest-test/
Authentication disabled
Starting server on :8002

Step4. Try to create snapshot of /tmp directory using rest-server api, get 500 internal server error

Output from client:

root@backup-server-restic:/backup/rest-local/tmp# /usr/local/bin/restic -r rest:http://127.0.0.1:8002/ backup /tmp/
enter password for repository: 
scan [/tmp]
scanned 9 directories, 0 files in 0:00
[0:00]   0B/s  0B / 0B  9 / 9 items  0 errors  ETA 0:00 
duration: 0:00, 0.00MiB/s
server response unexpected: 500 Internal Server Error (500)
restic/backend/rest.(*restBackend).Save
        src/restic/backend/rest/rest.go:134
restic/repository.(*Repository).savePacker
        src/restic/repository/packer_manager.go:122
restic/repository.(*Repository).Flush
        src/restic/repository/repository.go:248
restic/archiver.(*Archiver).Snapshot
        src/restic/archiver/archiver.go:733
main.runBackup
        src/cmds/restic/cmd_backup.go:493
main.glob..func2
        src/cmds/restic/cmd_backup.go:39
github.com/spf13/cobra.(*Command).execute
        src/github.com/spf13/cobra/command.go:631
github.com/spf13/cobra.(*Command).ExecuteC
        src/github.com/spf13/cobra/command.go:710
github.com/spf13/cobra.(*Command).Execute
        src/github.com/spf13/cobra/command.go:669
main.main
        src/cmds/restic/main.go:63
runtime.main
        /usr/local/go/src/runtime/proc.go:185
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:2197

Output from server:

root@backup-server-restic:/backup# /usr/local/bin/rest-server --path /backup/rest-test/ --listen ":8002" --debug                                                                                                          
rest-server 0.9.3 compiled with go1.8.1 on linux/amd64                                                    
Data directory: /backup/rest-test/                   
Authentication disabled                              
Starting server on :8002                             
HEAD /config                                         
CheckConfig()                                        
GET /keys/                                           
ListBlobs()                                          
GET /keys/5751c09eaf4fdfa0c96c627e8dce0ba3871ae9c50fc532f0a13572d4fca58b76                                
GetBlob()                                            
GET /config                                          
GetConfig()                                          
GET /locks/                                          
ListBlobs()                                          
POST /locks/b7fa70e3bfaf7423e44342e77a6cf48d1012678b7e8cecaaf53a1d5d697eb445                              
SaveBlob()                                           
GET /locks/                                          
ListBlobs()                                          
GET /locks/b7fa70e3bfaf7423e44342e77a6cf48d1012678b7e8cecaaf53a1d5d697eb445                               
GetBlob()                                            
GET /index/                                          
ListBlobs()                                          
GET /snapshots/                                      
ListBlobs()                                          
POST /data/b11f8ad215c01849238d2e462b4ace65fe7023a947335d1dfd313dd66e5d2a13                               
SaveBlob()                                           
open /backup/rest-test/data/b1/b11f8ad215c01849238d2e462b4ace65fe7023a947335d1dfd313dd66e5d2a13: no such file or directory                                                                                           
DELETE /locks/b7fa70e3bfaf7423e44342e77a6cf48d1012678b7e8cecaaf53a1d5d697eb445                            
DeleteBlob()  

Comments

When i initialize rest repository on step2 via rest-server api, everything works good. I only see problem when initialize it locally and then try work with it using rest-server api.

Further prometheus metrics

Hi!

Great work. This is exactly how backups should be.

What I'd wish: Expose not only rather technical data in the '/metrics' endpoint but also information about the backup status. I could then easily setup alarms (e.g. "latest backup is older than 3 days") and be assured everything is fine.

Best wishes
Adrian Schneider

go mod vendor Problems with goji

make failed:

go: inconsistent vendoring in /opt/rest-server:
        github.com/beorn7/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/goji/[email protected]+incompatible: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/golang/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/gorilla/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/inconshreveable/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/matttproud/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/miolini/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/prometheus/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/prometheus/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/prometheus/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/prometheus/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/spf13/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/spf13/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        [email protected]+incompatible: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        golang.org/x/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        [email protected]+incompatible: is replaced in go.mod, but not marked as replaced in vendor/modules.txt
        github.com/gorilla/[email protected]: is replaced in go.mod, but not marked as replaced in vendor/modules.txt

run 'go mod vendor' to sync, or use -mod=mod or -mod=readonly to ignore the vendor directory
Makefile:15: recipe for target 'rest-server' failed
make: *** [rest-server] Error 1

go mod vendor failed also
go: github.com/goji/[email protected]+incompatible used for two different module paths (github.com/goji/goji and goji.io)

Can't access metrics when private repos is enabled

When having --private-repos and --prometheus, the /metrics can't be accessed.

The fix would be to add a isMetricsPath() like this:

diff --git a/handlers.go b/handlers.go
index de3f328..04eb5c2 100644
--- a/handlers.go
+++ b/handlers.go
@@ -132,6 +132,14 @@ func isUserPath(username, path string) bool {
        return len(path) == len(prefix) || path[len(prefix)] == '/'
 }
 
+func isMetricsPath(path string) bool {
+       prefix := "/metrics"
+       if !strings.HasPrefix(path, prefix) {
+               return false
+       }
+       return len(path) == len(prefix) || path[len(prefix)] == '/'
+}
+
 // AuthHandler wraps h with a http.HandlerFunc that performs basic authentication against the user/passwords pairs
 // stored in f and returns the http.HandlerFunc.
 func AuthHandler(f *HtpasswdFile, h http.Handler) http.HandlerFunc {
@@ -141,7 +149,7 @@ func AuthHandler(f *HtpasswdFile, h http.Handler) http.HandlerFunc {
                        http.Error(w, http.StatusText(http.StatusUnauthorized), http.StatusUnauthorized)
                        return
                }
-               if Config.PrivateRepos && !isUserPath(username, r.URL.Path) {
+               if Config.PrivateRepos && !isUserPath(username, r.URL.Path) && !isMetricsPath(r.URL.Path) {
                        http.Error(w, http.StatusText(http.StatusUnauthorized), http.StatusUnauthorized)
                        return
                }

I will submit a PR with that patch.

Release new version

It's been a long time since the last release. There have been some improvements regarding metrics and a security fix for append-only mode.

It would be nice it these are available in a released version.

Support subdirectories with --private-repos

What should rest-server do differently?

Currently only one level of username is supported with the --private-repos flag, e.g. http://server/username is supported but not http://server/username/foo, http://server/username/foo/bar, etc.

Rest-server should allow at least one additional level, e.g. http://server/username/foo/bar.

We should at the same time consider if there's a point in, and without too much complexity, possible to, make it accept any number of subdirectories, e.g. http://server/username/foo/bar/star/har.

What are you trying to do? What is your use case?

Example use case: #76 (comment)

Add docker port mapping

In readme guide please add: -p 80:80

Start server

docker run --name myserver -p 80:80 -v /my/data:/data restic/rest-server

If you don't forward the container port then restic cannot communicate with the container on the rest server.

Request path sanitization issues

There seems to be a small request path sanitization issue:

bitsie:~ weingart$ curl -v --path-as-is http://toby:toby@localhost:8000/data/../ && echo ''
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8000 (#0)
* Server auth using Basic with user 'toby'
> GET /data/../ HTTP/1.1
> Host: localhost:8000
> Authorization: Basic dG9ieTp0b2J5
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Thu, 27 Jul 2017 05:10:41 GMT
< Content-Length: 64
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host localhost left intact
[".htpasswd","config","data","index","keys","locks","snapshots"]
bitsie:~ weingart$

I've not tried the other endpoints, nor tried other methods of sending bogus paths.

Reports and statistics about usage

Hello,

it would be nice to have at least a CLI utility to see some metrics, for example I want to have a list of all repos with the date of the last backup/snapshot and the total size it takes on disk repo by repo. Basically a bit like what we can do client by client on client side, but do it on server side with a more global view.

This is crucial info to supervise that we could add to our monitoring station, as backups need to be done frequently and we need to know when it is not the case for whatever reason.

It could be non-specific to rest-server backend but since that's what I use that's what I'm interested in :)

Rename --cpuprofile to --cpu-profile

The --cpuprofile flag is inconsistently named, it should be --cpu-profile as it's actually two words (like --append-mode and --private-repos).

I realize people aren't always happy with renames, but we're still at a pre-1.0.0 version and it's better to do it now than later if it's going to be done. Also, this is a flag that probably almost noone ever uses anyway, so it should be fine.

Anyone against?

unexpected HTTP response code 500

Hey,

i ve Problem.

I installed the restic-server and started it on port 8000.
An init and backup doesnt be work.

debug-output:

CheckConfig()
stat /tmp/restic/ovm4154/config: no such file or directory
POST /ovm4154/keys/93bcd6be459943629ee0e7f21795c9f00d9d2cab7813c19e6a8284e2be977b18
SaveBlob()
open /tmp/restic/ovm4154/keys/93bcd6be459943629ee0e7f21795c9f00d9d2cab7813c19e6a8284e2be977b18: no such file or directory
GET /ovm4154/keys/
ListBlobs()
GET /ovm4154/keys/9212e47add82a13c56944d9c627e06f1f329bc0f40ffa72bda9bd94c20ac8aee
GetBlob()
GET /ovm4154/config
GetConfig()
GET /ovm4154/locks/
ListBlobs()
POST /ovm4154/locks/b2324dbd1842a8ebdef807865be7d69adc44f5a5b16218488387802d0ad5c664
SaveBlob()
GET /ovm4154/locks/
ListBlobs()
GET /ovm4154/locks/b2324dbd1842a8ebdef807865be7d69adc44f5a5b16218488387802d0ad5c664
GetBlob()
GET /ovm4154/index/
ListBlobs()
GET /ovm4154/snapshots/
ListBlobs()
POST /ovm4154/data/8b721003b31308f0bc931d242cb747e11ac1667454e57ab190416c777f7424c2
SaveBlob()
open /tmp/restic/ovm4154/data/8b/8b721003b31308f0bc931d242cb747e11ac1667454e57ab190416c777f7424c2: no such file or directory
POST /ovm4154/data/d9eb46fdbf9083fca170e23c73516b333b225bbd940ffc89aa4a16057d3bfe72
SaveBlob()
open /tmp/restic/ovm4154/data/d9/d9eb46fdbf9083fca170e23c73516b333b225bbd940ffc89aa4a16057d3bfe72: no such file or directory
POST /ovm4154/data/0e6ff305af31380f304b46d1ce97b6f4e106521147f3390f8308a077fab81d29
SaveBlob()
open /tmp/restic/ovm4154/data/0e/0e6ff305af31380f304b46d1ce97b6f4e106521147f3390f8308a077fab81d29: no such file or directory

Do you have any idea?

greetings
Sam

Don't do mkdirAll, if a directory is missing

Been trying to figure out why something keeps recreating my restic repo...and blocking the system from mounting my restic repo.

Was adding all kinds of extra tests to ensure that services wait for the volume to appear, which can take a while (sometimes hours) after bouncing raid box. Turns out the problem is on the other side of the bounce, where volume has disappeared, but rest-server refuses to accept that, and makes the mountpoint non-empty and blocks it from ever coming back.

The mkdirAll was the fix for issue #40

WIsh I had snapped out that new NAS back when I thought I could afford one....

Scoop for rest-server

Installing and managing restic on Windows is easier with Scoop. However, cannot find it for rest-server. Any interest?

unable to list pack f8d9161a: unexpected HTTP response code 404

Sometimes the rest-server returns a 404 response code instead of the pack.

The file in question definitely exists:

sh-4.2# ls data/main/data/f8 | grep f8d9
f8d9161a32b18bf4dc2a86d31b20881d820c8373fcc4d8031e0a1001664506a6

It seems that the server fails to return the file properly at times.

The server is started within systemd, could this be an issue with some limit?

Auto-generate subfolders

In the README it's explained that in order to use previously local repos, moving them to your rest-server, you have to run this one-liner: for i in {0..255}; do mkdir -p $(printf "data/%02x" $i); done.

Would it not make sense to have rest-server simply create those directories automatically when needed? I imagine there's a way to check it being needed or not just once during a backup run or similar.

Why?

Was hoping to get some more information (here or in the README) as to why I or anybody would want to use this over the SFTP backend. Does it have better performance characteristics? If so, why? What about compared to OpenSSH-HPN? What does this give you that SFTP doesn't?

Set --append-only and --max-size per user

It would be nice if I could set the --append-only option per user, to be able to backup multiple hosts, where some are trusted and should be allowed to change data afterwards (e.g. forget old snapshots) and some are not, using a single rest-server instance with a shared --path and a shared .htpasswd file.

My current solution consists of two different instances - one with the --append-only flag and one without - using two different remotes(/hostnames, using lighttpd as a proxy), two different paths and two different htpasswd files, which isn't that great.

Fork and Die

Hi, there way to let this daemon fork and die.
I see you use systemd, that make it for you.

Use case:
Some NAS not use systemd, then fork and die is required for correct working.
It could it be implemented as command line parameter.
I found a go lib that allow without big effort Fork and Die.

Feature request: limit repo size

I'd like to host a REST server for multiple users and I don't want them to use up all of my server's disk space.
So it would be nice to be able limit the overall disk space the repos take, i.e. the recursive size of --path.
Furthermore, I'd optimally be able to set different repo sizes for different users.

rest-server log POST request size is zero

POST requests are logged with a size of "0"
GET requests are logged with the correct size

Is it possible to log the request size for POST requests too?
Want to do some stats based on access logs.

Tested with rest-server 0.9.6

BR Joerg

Repository URLs for --private-repos in README

The README currently states rest:https://foo:pass@host:8000/foo/bar as a valid URL for user foo when using --private-repos. But when using such a URL rest-server answers with a 500 Internal Server Error (ignored by restic) first and with a 404 Not Found second on restic init.

We could just fix this error in the documentation or enable such multiple repos for one user. There are some corner cases where this may be interesting to have.

Implementing it for some other separator (e.g. '+') would be pretty easy. I thought I need this, so I already implemented it in https://github.com/moho1/rest-server/tree/multiple_private_repos , but then I found out I don't need it, so I am not opening a pull request here but leave it as discussion. I can happily open a pull request if this is wanted this way.

Implementing it with '/' as separator would be much nicer, but more complex.

dns issue?

restic won't connect to the rest-server via domain name, only via ip.
I get this error:

restic -r rest:http://mini.local:8000 backup Documents/
unable to open config file: client.Head: Head http://mini.local:8000/config: dial tcp: lookup mini.local on 192.168.20.1:53: no such host
Is there a repository at the following location?
rest:http://mini.local:8000
./rest-server --path /Users/phil/backups/restic
rest-server 0.9.4 (v0.9.4-0-g0a0ed9c) compiled with go1.8.3 on darwin/amd64
Data directory: /Users/phil/backups/restic
Authentication disabled
Starting server on :8000

ping mini.local works fine.
This is on a local network - the server and clients are macs (10.12.6).
192.168.20.1 is my router.
Any ideas?

Compare with minio

Following #9 i'd like to know how it compare to minio ?
I see that it's dead simple, which is a killer feature, but maybe it's also more efficient ?

Support LDAP Authentication

The .htpasswd auth is difficult to scale or manage beyond a few users. Supporting LDAP would allow for large number of users in both *nix and Windows environments.

Handle SIGHUP

This is related to https://forum.restic.net/t/rest-server-crashes-or-exits-unexpectedly/1073/4

version: 0.9.7 or dfe9755 (master)

Steps to reproduce

  • Start rest-server
  • kill -HUP <pid>

What happens

rest-server exits.

What I expect

At least not to die, (ignoring the signal?). Reloading log files would be nice, and that seems to be the usual behaviour for this signal.

Some background, I discovered this because rest-server was unexpectedly dying, and logfiles would be empty. It appears that synology, the box we're running it on, generated a logrotate configuration for it and was killing it every so often. That wasn't fun.

I deactivated it for now, but handling SIGHUP would be nice. Rotating log files can be useful :)

rest: Server User / Password

Hello @ALL,

from my Point, it's not a good solution to send the Passwort / User with the URL.
Everyone can see the data in the Process list.

is there any other option ?
like export as variable ... or entering after starting restic ?

Thanks a lot :-),
René

Show a webpage when visited from a browser.

Docker image ID

da93e5693693

What should rest-server do differently?

Health checks! Some way to somehow query the server is not dead nor a zombie. Active gophers only.

What are you trying to do? What is your use case?

Discovering problems early.

Did rest-server help you today? Did it make you happy in any way?

I'll opt for a wonderful image instead, by Tadeusz Lakota:
image

Require a htpasswd by default

Currently rest-server fails open (does not require auth) if a .htpasswd file does not exist or cannot be opened. This is a pretty bad default from a security point of view.

I suggest that we require the .htpasswd file to be present, unless a -no-auth flag is set by the user to explicitly tell rest-server it should be wide open.

The downside of adding this is that it would break deployments for existing users that rely on their server being open. I'm not sure if this is a good idea, because it might break unattended backups for users and they might not be monitoring them.

PR #59 contains some comments on this.

Please put compiled binary files for rest-server in Releases

Hi!

Thank you for restic, it is awesome. I'm now writing deploy with ansible for integrating restic backup in our infrastructure and it will be great to just download already compiled binary file of rest-server instead of compiling it.

Ivan Luckyanov

SELinux Issues

After upgrading to latest Redhat 7.5 release I cannot start my restic/rest-server docker container. Seems to be some SE Linux issues. Maybe I'm missing something in my docker setup.

docker-compose up
Starting rest_srv ... done
Attaching to rest_srv
rest_srv | Error relocating /lib/ld-musl-x86_64.so.1: RELRO protection failed: Permission denied
rest_srv | Error relocating /entrypoint.sh: RELRO protection failed: Permission denied
rest_srv exited with code 127

tail audit.log
type=PROCTITLE msg=audit(1527685116.086:3277): proctitle=2F7573722F7362696E2F69707461626C6573002D2D77616974002D74006E6174002D4900444F434B45525F504F5354524F5554494E47002D73003132372E302E302E3131002D7000746370002D2D73706F7274003333303231002D6A00534E4154002D2D746F2D736F75726365003A3533
type=AVC msg=audit(1527685116.187:3278): avc: denied { read } for pid=16615 comm="entrypoint.sh" path="/lib/ld-musl-x86_64.so.1" dev="dm-2" ino=669806 **scontext=system_u:system_r:container_t:**s0:c462,c865 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
type=SYSCALL msg=audit(1527685116.187:3278): arch=c000003e syscall=10 success=no exit=-13 a0=7f5f76db6000 a1=1000 a2=1 a3=7f5f76b401bb items=0 ppid=16596 pid=16615 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="entrypoint.sh" exe="/bin/busybox" subj=system_u:system_r:container_t:s0:c462,c865 key=(null)
type=PROCTITLE msg=audit(1527685116.187:3278): proctitle=2F62696E2F7368002F656E747279706F696E742E7368
type=AVC msg=audit(1527685116.187:3279): avc: denied { read } for pid=16615 comm="entrypoint.sh" path="/bin/busybox" dev="dm-2" ino=266240 scontext=system_u:system_r:container_t:s0:c462,c865 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
type=SYSCALL msg=audit(1527685116.187:3279): arch=c000003e syscall=10 success=no exit=-13 a0=56308595f000 a1=4000 a2=1 a3=5630856a22d4 items=0 ppid=16596 pid=16615 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="entrypoint.sh" exe="/bin/busybox" subj=system_u:system_r:container_t:s0:c462,c865 key=(null)
type=PROCTITLE msg=audit(1527685116.187:3279): proctitle=2F62696E2F7368002F656E747279706F696E742E7368

Looks like the restic docker container files need to have container_t context?

Audit2allow gives me this but Not sure how to change context on the docker image filesystem.

module test_rule 1.0;

require {
type unlabeled_t;
type container_t;
class file { execute_no_trans open read };
}

#============= container_t ==============

#!!!! The file '/usr/bin/rest-server' is mislabeled on your system.
#!!!! Fix with $ restorecon -R -v /usr/bin/rest-server
allow container_t unlabeled_t:file execute_no_trans;

#!!!! This avc is allowed in the current policy
allow container_t unlabeled_t:file { open read };

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.