Comments (13)
Better resource:
https://docs.docker.com/config/containers/logging/configure/
Alternatively to changing the command you can also disable logging driver when starting volume backup by adding --log-driver none
option or local
driver (as explained in above link):
Tip: use the “local” logging driver to prevent disk-exhaustion
By default, no log-rotation is performed. As a result, log-files stored by the default json-file logging driver logging driver can cause a significant amount of disk space to be used for containers that generate much output, which can lead to disk space exhaustion.
from volume-backup.
Hm, I don't know why it needs more space, maybe it has something to do I with Docker, will try to investigate it further later.
However, one thing that drew my attention is that you are adding a timestamp to volume backup file name via DATENOW
environment variable. Does it mean that the old backup file is preserved? Maybe this can explain the fact that you need more disk space? Could you share your entire script source?
from volume-backup.
Old backup archives are deleted by cron after 2d. There is no problem. Problem is when i run volume-backup, probably need something like temp ?
When i run
docker logs i see stdout there.
#!/bin/bash
# https://github.com/getsentry/onpremise/issues/364
STAV_ZALOH="/var/tmp/docker_volume_backup.state"
DATENOW=$(date +\%F-\%H-\%M)
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
if [ -f $STAV_ZALOH ]; then
echo " EXISTUJE SOUBOR $STAV_ZALOH - KONCIM - KOUKNI CO OBSAHUJE ZA CHYBU"
exit 1
fi
#TURN OFF DOCKERs BEFORE BACKUP - BECAUSE DATA WILL CORRUPT / INCONSISTENT
cd /root/onpremise-20.10.1/; docker-compose -f docker-compose.override.yml down --timeout 30
if [ $? -ne 0 ]; then
echo "Nepodarilo se mi vypnout docker kontejnery pro zalohu. Bez toho nemuzu pokracovat - hrozi nekonzistence dat - po vyreseni smaz $STAV_ZALOH" > $STAV_ZALOH
exit 1
fi
mkdir /data/docker-backup/$DATENOW
BACKUP_DIR="/data/docker-backup/$DATENOW"
for i in sentry-data sentry-postgres sentry-redis sentry-zookeeper sentry-kafka sentry-clickhouse sentry-symbolicator; do
# echo $i
docker run -v $i:/volume --rm loomchild/volume-backup backup -c zstd - > $BACKUP_DIR/volume-$i-$DATENOW.tar.zst
if [ $? -ne 0 ]; then
echo "Neco se pokazilo u zalohy $i - koncim - Zkus spustit zalohu znovu - po vyreseni smaz $STAV_ZALOH" > $STAV_ZALOH
exit 1
else
echo "Zalohovano $BACKUP_DIR/volume-$i-$DATENOW.tar.zst"
fi
done
#Vsechno ok poustim docker zpet
cd /root/onpremise-20.10.1/; docker-compose -f docker-compose.override.yml up -d
if [ $? -ne 0 ]; then
echo "Neco se pokazilo u startu dockeru - nahod me rucne - po vyreseni smaz $STAV_ZALOH" > $STAV_ZALOH
exit 1
fi
# https://develop.sentry.dev/self-hosted/backup/
cd /root/onpremise-20.10.1/; docker-compose -f docker-compose.override.yml run --rm -T -e SENTRY_LOG_LEVEL=CRITICAL web export > $BACKUP_DIR/backup-$DATENOW.json
from volume-backup.
Hey, sorry for delay, unfortunately I won't be able to help you much with this as there's nothing specific in the script that uses temporary storage and tar and compression commands shouldn't use it either.
You can try to:
-
use the alternative version of volume-backup command without piping, e.g.:
docker run -v $i:/volume -v $BACKUP_DIR:/backup --rm loomchild/volume-backup backup -c zstd volume-$i-$DATENOW
-
add --tmpfs mount, as maybe docker stores some data on disk when piping
Could you also clarify how do you know that additional space is used during backup?
Is the double space being used only during each individual volume backup or for all volumes (in other words is space used n * v + v or 2 * n * v, where v is size of each volume)?
from volume-backup.
This is graph where you can see that free space is doubled when backup.
Ill check your advices soon and we will see.
from volume-backup.
It looks normal that the backup file is taking more and more space while archiving, what is weird that the usage suddenly decreases after backup is complete (at 00:35 - 00:36). Is the /data/docker-backup
directory is on a different disk / storage? Is it moved anywhere after backup is complete?
from volume-backup.
Before i start i want to notice when i see logs for docker container running i see probably stdout. Final disk usage drop is when backup job is finished. ??? SOME TEMPS/LOGS are deleted ???
root@sentry:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
799402a6f168 loomchild/volume-backup "/usr/bin/dumb-init …" About a minute ago Up About a minute beautiful_montalcini
root@sentry:~# docker logs 799402a6f168 | head
(�/�XTQZ �(:ih���e���c{$�Z�f�v{�x�#
��8��
��.R��?�WhB����A�`<H��L@d
�%��1hj0D�O�#��a�P,γ&?�-��y"Ok0,��"Yc
dI�3�$�@Y�by�3I�`@ 5iZ6=
laI
,� S��� ��{��� X� Oϲg��L&�@j(�@�F�E"�M�`Oc�*s����lZ�M,1I0sH�@6Y�%YW$
I���"�,�&s(@c�VTa��4���
)�4� ��G�5p�ɴ*,��Q�*(����* �f�Bi"��R�5�,�ؑh7M���Pl��
from volume-backup.
Interesting find. Yes, what you see in docker logs is the content of the stdout, so the zipped file.
It's possible that Docker stores it on disk somewhere while the container is running (we should research that).
My suggestion for next step: Please try executing volume-backup without using stdout/redirection with the command I mentioned above and see if the same storage usage spike occurs.
from volume-backup.
From a very fast research (first hit on Google, I am on mobile now): https://sematext.com/blog/docker-logs-location/
Could you monitor whether the file /var/lib/docker/containers/<container_id>/<container_id>-json.log
is growing during the backup and is deleted afterwards? (docker ps
should give you the container_id)
from volume-backup.
Better resource:
https://docs.docker.com/config/containers/logging/configure/Alternatively to changing the command you can also disable logging driver when starting volume backup by adding
--log-driver none
option orlocal
driver (as explained in above link):
Seems promising. Ill check that in next night run
from volume-backup.
Hi @pschonmann, did you see an improvement after making the change?
from volume-backup.
Logging option to none fixed my problem. Thanks.
from volume-backup.
That's great! Thank you for taking the time to investigate this, I will update the documentation for the others.
from volume-backup.
Related Issues (20)
- add verbose mode to backup HOT 3
- Signals, such as Ctrl-C are not handled HOT 1
- restore from stdin expects bzip2 input HOT 3
- add option to include arbitrary tar options HOT 2
- include options for no-clobber HOT 4
- One-minute sleep should no longer be necessary HOT 2
- Cannot restore backuped volume: Archive file *.tar.bz2 does not exist HOT 3
- Fails to Run on Raspberry Pi 4 (aarch64) HOT 2
- Labels are not backuped / restored HOT 9
- Multi-arch builds for arm64 HOT 3
- Add image to ghcr.io repository HOT 1
- Stop containers before backup HOT 1
- pbzip2 support? HOT 6
- Consider using parallel versions of algorithms by default HOT 2
- Automated monthly build (GitHub Actions) HOT 1
- Question: Why use `cat file |` instead of `< file`? HOT 3
- Consider an adaptive algorithm by default HOT 5
- Slow zstd compression HOT 2
- Uncompress error with "-c pigz" algorithm
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from volume-backup.