Git Product home page Git Product logo

couchdb2s3's Introduction

CouchDB-2-s3

Export a CouchDB database as a line oriented JSON file, then upload that to S3. Then later on s32couchdb that same file to get it into CouchDB instance.

Install

npm install -g couchdb2s3

Usage

couchdb2s3 \
  --bucket my-bucket \
  --database http://localhost:5984/my-database
  [--gzip]                                      # Optionally gzip the export

and

s32couchdb \
  --bucket my-bucket \
  --database http://localhost:5984/my-database
  [--prefix `db/my-database`]                   # Optionally specify a s3 prefix,
                                                #   defaults to `db/[name of database]`.
  [--marker `db/my-database-2010-12-31`]        # Optionally specify a s3 marker,
                                                #   defaults to `db/[name of database]-[yyyy]-[mm]-[dd]`
                                                #   for yesterday's date.

Configuration

AWS credentials for uploading and retrieving exports can be provided by;

  1. IAM role assigned to an EC2
  2. Environment variables; AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY
  3. A dotenv configuration file

Caveats

  • s32couchdb will not create a new database for you, you'll have to do that on your own.
  • Attached documents are not currently supported.
  • As written these scripts expect to be used as part of a export system where only the most recent backups are meaningful.

couchdb2s3's People

Contributors

ianshward avatar miccolis avatar tmcw avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

couchdb2s3's Issues

Consider compression

A quick test confirms that we could expect gzip compressed backups to be 1/4 the size they are now.

Enabled GZIP, saving to a big drive, error: no space left.

Exporting a big couchdb database, hundreds of gigs in size, maybe a tb. I get this error when the exported file reaches 5gb in size,

Saving the file to /mnt/ which has plenty of space.

-rw-r--r-- 1 root root 5.1G Jul 25 07:56 couchdb2s3-mydb-1500962694752
Error: ENOSPC: no space left on device, write at Error (native)

df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  7.8G     0 100% /
devtmpfs         15G   60K   15G   1% /dev
tmpfs            15G     0   15G   0% /dev/shm
/dev/nvme0n1    871G  127G  701G  16% /mnt/data

df -iT

Filesystem     Type       Inodes IUsed    IFree IUse% Mounted on
/dev/xvda1     ext4       524288 91951   432337   18% /
devtmpfs       devtmpfs  3923128   465  3922663    1% /dev
tmpfs          tmpfs     3925382     1  3925381    1% /dev/shm
/dev/nvme0n1   ext4     57991168  2932 57988236    1% /mnt/data

I tried:
npm config set tmp /mnt/tmp
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

Update: I was saving to a wrong folder, it wasn't using the big mounted drive.
The above commands might help someone else.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.