ashayh / twine Goto Github PK

========================================
Twine

"a bundle of redis threads"

Utilities maintaining and querying a replicated
 partitioned (shard) Redis cluster

Formerly named 'Roma'
========================================


========================================
Contents
========================================

* Quick Start
* Poetic License
* Introduction
* Note on Partitions (Sharding)
* Configuration
   o twine.conf
   o redis.conf
   o monitrc
* Core utilities
   o twine-install
   o twine-setup
   o twine-start
   o twine-stop
* Monit control
* Append only file
* Twine Name


========================================
Quick Start
========================================

$ git clone git://github.com/alexgenaud/twine.git
$ cd twine
$ bash twine-install.sh
$ bash twine-setup.sh localhost

$ bash twine-start.sh
$ monit -c partitions/localhost/monit/monitrc

$ monit -c partitions/localhost/monit/monitrc stop all
$ bash twine-stop.sh
$ monit -c partitions/localhost/monit/monitrc quit


========================================
Poetic License
========================================

(c) 2010 Alexander E Genaud

This work ‘as-is’ we provide.
No warranty express or implied.
We’ve done our best,
to debug and test.
Liability for damages denied.

Permission is granted hereby,
to copy, share, and modify.
Use as is fit,
free or for profit.
On this notice, these rights rely.


========================================
Introduction
========================================

Twine is a set of utilities that should help you get a partitioned
redis cluster up and running quickly. It handles failover using monit.

Twine does not yet promote slaves to masters. However, with enough
partitions, it is often faster and infinitely easier to bring failed
masters back online than to reconfigure slaves, masters, and clients.

Monit can be configured to monitor the cluster every second. Again,
with enough partitions, each node can be very small and thus come back
up to speed in seconds or less.

A backup is simply a matter of copying (or zipping) the partition
directory on each host. And restoration is the reverse.


========================================
Note on Partitions (Sharding)
========================================

Querying a partitioned cluster is the responsibility of the client
(perhaps I can share some Java code shortly). Basically, a client
should use the same twine.conf with the client and redis cluster.
After all, it defines the various redis nodes and their partition
number. For every key-value query, the client must first hash the
key and apply a modulo to obtain the partition to query.

For example, we could set and get from 64 partitions (shards), each
in its own redis data node on ports 4200 to 4263 on localhost:

   redis-cli -p (hash(key) % 64) set key value
   redis-cli -p (hash(key) % 64) get key


========================================
Configuration
========================================

Twine consists of three configuration file types. There is a single
twine.conf file which specifies the redis nodes, partitions, and
network location. Each redis node has its own redis.conf, though these
are quite similar to each other. Finally, each host will have its
own monitrc file.

twine.conf
  a simple config file which consists of redis node names, types
  (data or status), partition number, host, port, and database
  number. For example, the last line of a 64 part cluster might
  look like this:

    b4263=data 63 127.0.0.1 4263 0

redis.conf
  This is standard redis configuration file minus all the
  comments. By default, the redis data nodes of a twine cluster
  will write data asynchronously at least once per minute. All
  of this can be changed either in the various redis.conf or in
  twine-setup that generates them.

monitrc
  monitrc is used by monit - a unix service monitor. One monitrc
  is generated by twine-setup for each host. The first line tells
  monit how often it should check the cluster. By default, it is
  set to five seconds. Httpd is required for some basic queries
  like status and summary.


========================================
Core utilities
========================================

twine (not yet available)
  wraps redis-cli, but hashes the key
  and executes the command on the
  appropriate partition (shard)

twined (not yet available)
  daemon (of which there may be many)
  which monitors the cluster. It
  competes with any other twined for
  alpha position. The alpha sets the
  cluster status and handles fail-over.
  It can be configured to send an email
  if the status is HALT or any Redis
  node is down for more than x seconds.

twine-setup [host]
  creates the entire cluster plumbing by default
  including all partitions and redis.conf
  according to the twine.conf. Optionally takes
  a host name as arguement. This is convenient for
  setting up each machine using the same twine.conf
  file. The host name must be listed in the twine.conf
  file.

twine-start [host]
  starts the entire cluster according
  to the twine.conf configuration file
  (must be run separately per machine)

twine-stop [host]
  stops the entire cluster according
  to the twine.conf configuration file

twine-status (not yet available)
  returns the cluster status in various
  formats: summary, list, xml, html

twine-hash (not yet available)
  A plug-able hash function wrapper
  which accepts two arguments:
  a single 7-bit ASCII string (key)
  and a modulus (number of partitions).
  It returns the partition number.

twine-backup (not yet available)
  backs up the entire cluster to
  directory(ies) according to twine.conf

twine-restore (not yet available)
  overwrites all cluster data from a
  previous backup.


========================================
Monit control
========================================

Twine-setup configures a monitrc control file for
each host, found in partitions/HOSTNAME/monit.

It is worth editing the monitrc file (or twine-setup)
to configure how often (in seconds) monit checks the status
of the cluster (first line set daemon x). Also, mail
alerts are worth configuring, or removing so as not to
clutter the log files.

To start monit:

   monit -c partitions/HOSTNAME/monit/monitrc

Try to kill some or all of the redis nodes (not in production, of
course) with the following commands and see how quickly monit
brings them back to life:

   partitions/HOSTNAME/NODENAME/stop.sh
      or
   kill -9 <PID>
      or
   sh twine-stop.sh

If find the following alias helpful, which displays monit and
redis processes:

   alias psef='ps -ef | grep -v volume-monitor | grep "[rm][eo][dn]i[st]"'

When the monitrc file is to your liking, you will probably want to
move it to '/etc/monit/monitrc' and/or run it automatically on startup,
from init.d.

   http://mmonit.com/wiki/Monit/FAQ#init


========================================
Append only file
========================================

Twine-setup contains a variable at the top APPENDONLY=yes|no
which sets each redis not in the cluster to save using
either the original asynchronous database dump (no) or the
new append only journalling (yes). 

Setting APPENDONLY to 'yes' will create a script in
partitions/HOSTNAME/appendonly/appendonly.sh which can be
invoked by crontab. This script will compact the journal
file asynchronously.

http://code.google.com/p/redis/wiki/BgrewriteaofCommand


========================================
Twine Name
========================================

Why Twine? Twine is a strong chord consisting of multiple threads.
Twine is used to bind things together. Likewise Twine, the utility,
helps bind a cluster of Redis threads.

What about Roma? Well, within two hours of creating the public github
repo, I was informed by two people that there was a distributed key-
value store named Roma and the names might confuse people. I agreed.