Git Product home page Git Product logo

watchdb's Introduction

watchdb

A tool for easily replicating a SQLite database across a network.

Release

Overview

watchdb is a tool that enables quick setup of master-slave synchronization for SQLite databases across a network.

Synchronization is one-way only (changes made on the master will overwrite changes made on slaves). Slave databases are kept read-only to prevent accidental writes from application code.

watchdb replication is eventually consistent by design, so it's AP in CAP. If you need strong consistency (at the expense of performance and required changes to application code), take a look at rqlite.

Features

  • Works on Linux, OS X, Windows
  • No dependencies, just one binary
  • Replicates sqlite DB changes quickly across any number of nodes
  • No changes to application code required
  • Optional authentication and encryption supported

In the future:

  • Incremental (partial) DB syncing

Installation

Precompiled binaries for supported operating systems are available.

Alternatively, run go get github.com/dkulchenko/watchdb. You'll need a working sqlite3 binary in $PATH if you go this route. The precompiled binaries embed a copy of sqlite.

Usage

Watch a SQLite database:

watchdb watch mydb.sqlite

On a slave server, sync:

watchdb sync 127.0.0.1:8144 mydbcopy.sqlite

Easy as that. Any changes made to mydb.sqlite will quickly show up in mydbcopy.sqlite. Try it out!

Options

You can specify any option on the command line, or provide a configuration file (an example config is available at conf/example.yml):

watchdb watch --config-file=config.yml mydb.sqlite

Set a custom bind address/port:

watchdb watch --bind-addr=127.0.0.1 --bind-port=1234 mydb.sqlite

Authentication

Require an auth key to be sent before syncing is allowed (similar to Redis AUTH):

watchdb watch --auth-key 927bc430fc2195fa2f0caaf35d115c mydb.sqlite

and on the slave:

watchdb sync --auth-key 927bc430fc2195fa2f0caaf35d115c 127.0.0.1:8144 mydbcopy.sqlite

Or specify it in the configuration file using the auth_key parameter.

Encryption

watchdb supports SSL for encrypted syncing between nodes.

watchdb watch --ssl --ssl-key-file=key.pem --ssl-cert-file=key.crt mydb.sqlite

and on the client:

watchdb watch --ssl 127.0.0.1:8144 mydbcopy.sqlite

You may omit the ssl key file/cert file and a self-signed one will be generated for you at startup. Note that you'll need to provide the --ssl-skip-verify option on the client for this to work.

Why?

sqlite3 is an excellent database, and by far the easiest way to embed SQL into an app where using a larger DB like MySQL or PostgreSQL might not be possible. The downside, of course, is the single-node nature of the DB, which means that if the primary goes down, the data goes down with it.

I wanted to build a simple way to replicate a sqlite database across a network without requiring any special configuration or changes to application code. Hence, watchdb.

Contribute

  • Fork repository
  • Create a feature or bugfix branch
  • Open a new pull request

License

The MIT License (MIT)

Copyright (c) 2015 Daniil Kulchenko [email protected]

watchdb's People

Contributors

dkulchenko avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

watchdb's Issues

More than 100 Mb SQLite File not working

Hi,
I am trying to implement sync functionality with binary and I test all the below scenarios with cmd with admin privileges.

I have to implement sync functionality with the SQLite file and test it with a master and slave database on different computers with the same network path. But it seems, it's creating an issue when the Sqlite file size is bigger, and here are the different test cases which I have tried.

  • Small Sqilite file size of 40 MB - Via CMD Verified sync functionality and it is working.
  • Test the data with 138 MB Sqilite file size but it is creating an issue in data sync.
    What's the issue? In the Synced database, more than 50,000 records could not match with the master database
  • I have tried with 80 MB Sqlite file with the same above scenario -- the issue of file size but no issue of data sync.
  • I have tried with 104 MB with same above scenario -- issue of file size(On slave server after sync, the file size is 98 MB)
    The Steps I have taken is
    1) At Master and slave both the side manually I have copied 104 Mb Sqlite file.
    2) In the master database and slave DB, started to watch and sync.
    3) Modified data directly on master db[Deleted And updated Record via Sqlite studio]
    4) In the Synced database file size has been reduced to 98 Mb But the strange things I have found, there was not any data difference but still, the file size has been reduced.
  • With all of the above test cases, Another master Sqilite file with 121 MB file size, In this test scenario, I have found that there was a mismatch in the data and it was 50,000 data that has not been matched at sync database.

So in summary, I think when the Sqilite master database is file size going beyond 100 MB then it's creating a problem so please guide me on how should I implement sync functionality?

Thanks,
Shailesh

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.