Is there a blog demo tutorial? I still get various errors when I get started for the first time. Thank you.

Hi thanks a lot for this library, very useful (especially the option to backup on demand via command).

Any chance of generating a tag for version 15 (release ~1 month ago)? It's the latest PG available in Docker but backup fails because of pg_dump version mismatch.

pg_dump: error: server version: 15.1 (Debian 15.1-1.pgdg110+1); pg_dump version: 14.5
pg_dump: error: aborting because of server version mismatch

When not set the SCHEDULE env defaults an empty string, failing the check at

if [ -z "$SCHEDULE" ]; then
    # TODO: how to make CTRL-C work?
    echo "WARNING: $SCHEDULE is null. Going to sleep."
    tail -f /dev/null # do nothing forever
else
  exec go-cron "$SCHEDULE" /bin/sh backup.sh
fi

Resulting in the else clause to never run

Hi! I'm trying to run this project but I'm getting an error I can't decipher:

04/02/2023 6:47:19 PM new cron: @daily
04/02/2023 6:47:19 PM (0x51b998,0x59acf8)
04/02/2023 6:47:19 PM Stopping
04/02/2023 6:47:19 PM Waiting
04/02/2023 6:47:19 PM Exiting
04/02/2023 6:47:20 PM new cron: @daily

My docker compose looks like this:

version: '3'

services:
  pg:
    container_name: pg
    image: postgres:15
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      PGDATA: /data/postgres
      PGDATABASE: ${POSTGRES_DB}
    volumes:
       - pg:/data/postgres
    ports:
      - "5432:5432"

  pg_backup:
    image: eeshugerman/postgres-backup-s3:15
    container_name: pg_backup
    environment:
      SCHEDULE: '@daily'    
      BACKUP_KEEP_DAYS: 7    
      PASSPHRASE: ${PG_BACKUP_PASSPHRASE}
      S3_REGION: "" # Empty since is not required in Cloudflare R2
      S3_ACCESS_KEY_ID: ${PG_BACKUP_S3_ACCESS_KEY_ID}
      S3_SECRET_ACCESS_KEY: ${PG_BACKUP_S3_SECRET_ACCESS_KEY}
      S3_BUCKET: ${PG_BACKUP_S3_BUCKET}
      S3_PREFIX: ${PG_BACKUP_S3_PREFIX}
      S3_ENDPOINT: ${PG_BACKUP_S3_ENDPOINT}
      POSTGRES_HOST: pg
      POSTGRES_DATABASE: ${POSTGRES_DB}
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}

I don't know what's failing or why. Can you point me in the right direction?

It makes me a little uneasy that the cron we're using is just a binary curled from a hardcoded GitHub release URL of a little-known project.

Instead, we should do something like this: https://stackoverflow.com/a/47960145

This would probably be a breaking change (go-cron offers some special-sauce scheduling syntax, IIRC), so it would be a good opportunity to switch to a new image tagging convention which incorporates a version for postgres-backup-s3 itself, in addition to the PostgreSQL version.

Hi there!

I'm triying to use you docker image to backup my postgres data, which sits on a container in a rails app. After building the images, the backup script doesn't work due to a version mismatch between the server and image versions of pg_dump

The error issued in the console is as follows:

pg_dump: server version: 12.1 (Debian 12.1-1.pgdg100+1); pg_dump version: 11.7
pg_dump: aborting because of server version mismatch

The problem is that we cannot change the version of our server at least until the weekend, since the application can't stop running. How can I change the image's postgres version to 12.1 so that we could actually use it? Instructions would be highly appreciated!

Thanks 🤙🏾

I get the following error: docker stderr: pg_dump: error: could not translate host name "appname-db" to address: Name does not resolve

appname-db is the name of the postgres db container on the internal docker network I have setup for containers to interface with each other (it's a single server setup for all services such as app, cron, db etc). If I enter into the backup container and ping appname-db it resolves correctly and connects successfully. I have the following in my config too:

options:
      network: "private"

This works as a DNS reference for other containers such as my app or cron container to connect to the db. Is there something extra I need to do to make the internal DNS work with this service?

I'm using Kamal... My setup for backups is pretty much exactly the same as what is specified on this blog post I came across: https://guillaumebriday.fr/how-to-deploy-rails-with-kamal-postgresql-sidekiq-and-backups-on-a-single-host

First of all, thank you very much for the great project!

please is it possible to configure multiple schedulers/times?

Containers running this image throw errors when trying to remove them through docker compose down:

Delete "http://docker.example.com/v1.41/containers/00faf040af3d961331a939748b80e9cc151f18034b89a0cb194daee42990abd6?force=1&v=1": command [ssh -l ubuntu -- 3.234.133.243 docker system dial-stdio] has exited with exit status 1, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=

Maybe it lacks a proper SIGTERM handling?

I used self-signatures. minio which is selfhosted.
Let's Encrypt

raise the exception: SSL validation failed for https://minio.xxx.cn:3000/rvm/backup/rvm_2024-01-19T07%3A56%3A48.dump [SSL: TLSV1_UNRECOGNIZED_NAME] tlsv1 unrecognized name (_ssl.c:1006)

how to disable the ssl verify

AWS deprecated the permission of ListObjects in favor of ListObjectsV2, thus new policies can not have this permission.
My "delete backups after some days" is failing due to this permission. Could you please update it?
For reference: https://docs.aws.amazon.com/cli/latest/reference/s3api/list-objects.html

Hello, I noticed that my backups are not being deleted from S3 - I am using DigitalOcean as it has the same API as Amazon S3. Wondering what may be the issue, my config is as follow:

      BACKUP_KEEP_DAYS: '1'
      POSTGRES_DATABASE: postgres
      POSTGRES_HOST: vpn.myhost.com
      POSTGRES_PASSWORD: SECRET
      POSTGRES_PORT: '5433'
      POSTGRES_USER: backups_user
      S3_ACCESS_KEY_ID: SECRET_ID
      S3_BUCKET: AWESOME
      S3_ENDPOINT: https://nyc3.digitaloceanspaces.com
      S3_PREFIX: backup-pg-production
      S3_REGION: nyc3
      S3_SECRET_ACCESS_KEY: /ACCESS
      SCHEDULE: '@every 0h15m0s'

at line "exec go-cron "$SCHEDULE" /bin/sh backup.sh"

go-cron: not found. how to resolve?

Hello,

thank you for providing this cool repo!
I am using this cool repo in my docker-compose.yml. Unfortunately i get the following error:

2022/02/11 12:16:39 112: upload failed: ./db.dump to s3://MARIOS_S3_BUCKET/backup/DB_NAME_2022-02-11T12:16:17.dump An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.

I tested my credentials with my local aws cli and it was able to sync assets into the same bucket.

Do you have any idea why i get this error?
Thank you a lot!

I successfully restored a backup using docker.io/eeshugerman/postgres-backup-s3:15 on a container running docker.io/library/postgres:16-alpine, so I think it just works? Having official confirmation with a properly tagged image would be great!

Hi,

I'm getting an error saying,

what can i do to fix this?

Would it make sense to add support and configuration options for keeping backups for Days, Weeks, Months?
This is a repo based on current - https://github.com/prodrigestivill/docker-postgres-backup-local which provides these capabilities, so it should be straightforward to implement.

I've also asked this question in the other tool's discussion - prodrigestivill/docker-postgres-backup-local#93 (comment)

[Feature Request]

Hi eeshugerman,

thanks for this great tool! :-)

It would be great to use an AWS EC2 instance profile where you set the permissions in the IAM role for the instance. This way you don´t need to set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The AWS CLI will look for the instance profile when these parameters are not set automatically.

Could you make S3_ACCESS_KEY_ID and S3_SECRET_ACCESS_KEY optional for that?

Best regards,
Michael

The schickling image supports using pg_dumpall to backup all databases within a Postgres instance/cluster.

https://github.com/schickling/dockerfiles/blob/master/postgres-backup-s3/backup.sh#L65-L88

Could this also be supported in this image? This is the only missing feature holding me back from using this image over schickling's, as I'd like to have the backup be a "set it and forget it" cronjob even as more databases are added.

Would be open to try implementing it myself (will be testing against CloudNative-PG) and PR'ing the changes assuming all goes well.

webhook response:

{"error": "Automated Build only available on a Pro or Team plan"}

need to look into alternatives

Looks like it's available in the Alpine 3.15 release.

This postgres backup image is also based on schickling but stores the backup locally. Nonetheless, a few interesting options are added to improve backup management. They are:

• BACKUP_KEEP_DAYS Number of daily backups to keep before removal. Defaults to 7.
• BACKUP_KEEP_WEEKS Number of weekly backups to keep before removal. Defaults to 4.
• BACKUP_KEEP_MONTHS Number of monthly backups to keep before removal. Defaults to 6.
• BACKUP_KEEP_MINS Number of minutes for last folder backups to keep before removal. Defaults to 1440.

Is this something you could replicate?

Hello @eeshugerman

do you think that we can add notifications (email, webhook, slack...) to the project in case the backup fails, etc..? just an idea

Is it possible to add ARM support?

Hi Elliott, thanks for creating a useful tool! What are your thoughts on adding an option (can be even the default setting) to gzip the database backup dump to reduce the file size?

This feature is beneficial because it reduces the required storage and thus saves money. Thanks again @eeshugerman for your hard work on this!

Hello,
I'm trying to build the image(windows and wsl2 Ubuntu20.04) and I receive this error:

-----
 > [3/7] RUN sh install.sh && rm install.sh:
: not foundstall.sh: line 2:
#0 0.320 install.sh: set: line 3: illegal option -
------
Dockerfile:6
--------------------
   4 |
   5 |     ADD src/install.sh install.sh
   6 | >>> RUN sh install.sh && rm install.sh
   7 |
   8 |     ENV POSTGRES_DATABASE ''
--------------------
ERROR: failed to solve: process "/bin/sh -c sh install.sh && rm install.sh" did not complete successfully: exit code: 2

I have the following configuration:

backups:
  image: eeshugerman/postgres-backup-s3:16
  host: accessories
  env:
    clear:
      SCHEDULE: "@daily"
      BACKUP_KEEP_DAYS: 7
      S3_ENDPOINT: https://xxx.eu.r2.cloudflarestorage.com/sumiu-files
      S3_PREFIX: backup
      S3_REGION: auto
      S3_BUCKET: pg_backups
      POSTGRES_HOST: 10.0.0.3
      POSTGRES_DATABASE: sumiu_production
      POSTGRES_USER: postgres
    secret:
      - POSTGRES_PASSWORD
      - S3_ACCESS_KEY_ID
      - S3_SECRET_ACCESS_KEY

This is deployed with Kamal, btw

When I run the backup command like this: kamal accessory exec backups "sh backup.sh"

I get this error

docker stdout: Creating backup of sumiu_production database...
Uploading backup to pg_backups...
upload: ./db.dump to s3://pg_backups/backup/sumiu_production_2024-01-29T17:09:38.dump
Backup complete.
Removing old backups from pg_backups...
docker stderr: An error occurred (NoSuchKey) when calling the ListObjects operation: The specified key does not exist.

I was using S3 but I'm trying to change to cloudlfare's R2. My first suspicion was that it had some kind of persistence and tried to delete some "known" backup that exists on S3 but not on R2 but checking the script doesn't look like it; it looks more like an inconsistency on the API level, where S3 returns nothing, and R2 returns an error.

Do you think something can be done on the removal part of the script? If the query is empty, instead of pipe it aws $aws_args, just skip it.

It is worth noting that even though the command failed, the backup is there on R2, so I guess this will stop failing in 7 days

btw, thanks for this gem of a project, really nice!

The original image could be run without the SCHEDULE variable. When doing so it executes one-time backup creation.

When trying to run your version it outputs this instead:

WARNING:  is null. Going to sleep.

docker/hub-feedback#508 (comment)
https://docs.docker.com/docker-hub/builds/advanced/#build-hook-examples

Thank you for amazing project.

Want to mount a large local mount point where backup is create. My root was 20GB, I've another mount point which is around 100GB, And my backup is definitely ~16GB. So I can utilize larger space.