Implement automatic backups & Dropbox cloud storage about dsmr-reader HOT 19 CLOSED

I have seen OneDrive and dropbox integrations ,I think Dropbox published a
new api quite recently. Might be a good one :)

Groeten,

Jeroen Peters

2016-02-09 20:19 GMT+01:00 Dennis Siemensma [email protected]:

Backups should be made easy. We should check for any free online storage
we can use and integrate.

For example, it would be nice when users can create or link their Dropbox,
Onedrive, etc to the applicatie, which will ensure backups.

Note sure if possible, but worth investigating.

—
Reply to this email directly or view it on GitHub
#44.

from dsmr-reader.

Thanks for the info. Would be cool if it works, since most end users tend not to backup at all anyway. :]

from dsmr-reader.

I think I'll start this week with Dropbox. They advise using ChunkedUploader, which will divide data in 4MB chucks and automatically resume uploading when failed.

from dsmr-reader.

Nice, Dropbox allows apps to either have full account access or only access restricted to a dedicated folder for the app. The last one is nice.

from dsmr-reader.

One of the largest challenges is finding a way to use oAuth without publishing the secret key. If someone wants to abuse the application's key and get it blocked, they only have to look for it in the source code.

I would just like to get an access token without secret. Somehow.

from dsmr-reader.

I can't find a solution for it an lots of people complain about it regarding open source & oauth. I'll check it some other time.

from dsmr-reader.

Another suggestion is to just let end users create an app. Might work.

from dsmr-reader.

Cool, I already have it working. App owners can easily generate their own access token, without the oauth process. And the token is the only thing required.

import os

from dropbox.client import DropboxClient
from dropbox import rest


def backup():
    TOKEN = "*********************************"
    FILE = "/tmp/data.txt"
    client = DropboxClient(TOKEN)

    size = os.stat(FILE).st_size
    bigFile = open(FILE, 'rb')

    uploader = client.get_chunked_uploader(bigFile, size)
    print("uploading: ", size)
    while uploader.offset < size:
        try:
            upload = uploader.upload_chunked(chunk_size=1)
            print('upload', upload)
        except rest.ErrorResponse as e:
            print(e)
            # perform error handling and retry logic
    uploader.finish('endpoint file.txt')

from dsmr-reader.

I found https://github.com/django-dbbackup/django-dbbackup, which uses the builtin backup commands (for postgres/mysql) and also supports Dropbox. Still have to check it out though.

from dsmr-reader.

DBbackup is nice, although it required dropbox secret key in stead of token. Might not be a problem at all.

Compression is somewhat slow, but it certainly pressed file size. Benchmarked on laptop:

(dsmr-reader)dennis@laptop:~/workspace/dsmr-reader$ time ./manage.py dbbackup
Backing Up Database: dsmrreader
Running: pg_dump --username=****** --host=localhost ******
Backup tempfile created: 65.0 MiB
Writing file to FileSystemStorage: <bound method Storage.backup_dir of <dbbackup.storage.filesystem_storage.Storage object at 0x7f09e23bcda0>>, filename: dsmrreader-laptop.siemensma.net-2016-02-19-200925.psql
Writing file dsmrreader-laptop.siemensma.net-2016-02-19-200925.psql
Writing file dsmrreader-laptop.siemensma.net-2016-02-19-200925.psql

real    0m6.337s
user    0m1.834s
sys 0m0.397s

(dsmr-reader)dennis@laptop:~/workspace/dsmr-reader$ time ./manage.py dbbackup -z
Backing Up Database: dsmrreader
Running: pg_dump --username=****** --host=localhost ******
Backup tempfile created: 5.1 MiB
Writing file to FileSystemStorage: <bound method Storage.backup_dir of <dbbackup.storage.filesystem_storage.Storage object at 0x7f1ba6bd9da0>>, filename: dsmrreader-laptop.siemensma.net-2016-02-19-200936.psql.gz
Writing file dsmrreader-laptop.siemensma.net-2016-02-19-200936.psql.gz
Writing file dsmrreader-laptop.siemensma.net-2016-02-19-200936.psql.gz

real    0m18.984s
user    0m15.370s
sys 0m0.349s

from dsmr-reader.

I think DBbackup is nice, but lacks some dynamic settings. Also, dropbox authentication is made a little more complex than needed.
And last but not least, for restoring a backup, you'll have to stop your runserver/gunicorn anyway. So I'm thinking about have only a backup export feature and writing up an how-to for restores.

from dsmr-reader.

Starting backup module. I think it would be nice to just sync the entire backup folder, containing at most 7 backups (daily rotate). A 'simple' folder sync script is already created by dropbox:
https://github.com/dropbox/dropbox-sdk-python/blob/master/example/updown.py

from dsmr-reader.

Cool, I almost managed it. Added support for automatic backups for mysql and postgresl. Optional compression and optional syncs to dropbox.

Todo's are:

• Tests
• Allowing Dropbox to override rotating backups (it now adds (1) to the file for some reason).

from dsmr-reader.

Doh: def finish(self, path, overwrite=False, parent_rev=None):

from dsmr-reader.

Finaly completed. I'll check it out here first.

from dsmr-reader.

Merged, but first deployment on mysql test here, fails.

from dsmr-reader.

Seems like it the same issue I had this week with another instance. Caused by:

@transaction.atomic
def compact(dsmr_reading):

Removing the @transaction.atomic during one run of dsmr_backend 'fixes' it. It's possible that one of the signals called by the service crashes of starts another transaction.

CommandError: Exception @ <bound method AppConfig._on_backend_called_signal of <AppConfig: dsmr_consumption>>: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.

from dsmr-reader.

Deployed on production successfully! Backup was created within minute and dropbox synced as well, after generating and entering my token. :]

from dsmr-reader.

Marked stable.

from dsmr-reader.