Easily run a persistent CitusDB cluster on Mesos with Marathon!

• Docker
• Mesos / Marathon Cluster
• Consul (+working registrator)

If you don't currently have a working Mesos / Marathon / Consul cluster, you should check out eBayClassifiedsGroup's PanteraS solution!

This Docker image has two functions: a CitusDB master that stores the shard information and metadata, and a CitusDB slave that stores the actual data on the shard. You'll need to run exactly one master, and at least one slave.

This image is based off the official Postgres 9.5 docker image, so we support any environment variable for configuration that the official image does.

We'll run the master outside of Marathon, directly in Docker since Marathon isn't exactly meant for persistent services. First, let's start some slaves.

Create a new application group in Marathon (I'm using /internal/citusdb) and add a new application.

From here, we have two options for how to tell Marathon to run this slave.

• Fixed-node
  Marathon will only run a CitusDB slave on the Mesos slaves specified.
  Each CitusDB slave will show in Marathon as a separate application, allowing you to take certain nodes out of service without worry that Marathon will reschedule the stopped service elsewhere.
  Pros: More certainty about where your CitusDB slaves are running. Good for fixed-size clusters.
  Cons: You must have a static set of Mesos slave hostnames, such as node1, node2 (or sun, moon, ...). Mesos slaves with autogenerated names such as mesos-slave-i-32bcde7b will not be able to run a CitusDB slave if the host is renamed or replaced, such as in an autoscaling group.

• Per-node
  Marathon will run a CitusDB slave on every Mesos slave in your cluster.
  You can stop an individual CitusDB slave in Marathon, but unless you use kill and scale it will be immediately rescheduled on the same host.
  Pros: Easier for larger clusters, if you want to quickly deploy multiple slaves.
  Cons: Harder to guarantee that CitusDB is running on only the slaves you want, as Marathon's constraints don't allow easily placing services on specific sets of nodes.

Included in this git repository is a template file you can use to generate application definitions for Marathon.
We'll edit this template to include the nodes we want a CitusDB slave on, and post the resulting JSON file to Marathon.

Switch to JSON mode in the Marathon WebUI, and paste in the contents of marathon_fixed_worker_definition.json.tpl, ensuring to change the following values:

• Any reference to node1 should be changed to the name of the node that you want to run a CitusDB slave on.
• Any reference to .consul should be changed to your Consul DNS suffix
• <your registry here> should be changed to the name of your private Docker registry (as this image isn't on the Dockerhub)
• /var/lib/docker/volumes/citus-slave can be changed (if necessary) to a different location on your Mesos slave's filesystem. This directory is where CitusDB will store it's data (rather than in a docker volume, for persistence).

Repeat this for each CitusDB slave you wish to create. Do not change the instances parameter in the Marathon configuration.
If you have four Mesos slaves, and want four CitusDB slaves, you will have to repeat this process four times - changing the node name each time.

Switch to JSON mode in the Marathon WebUI, and paste in the contents of marathon_dynamic_worker_definition.json.tpl, ensuring to change the following values:

• Any reference to .consul should be changed to your Consul DNS suffix
• <your registry here> should be changed to the name of your private Docker registry (as this image isn't on the Dockerhub)
• instances should be changed to the number of CitusDB slaves you wish to run. You can run a maximum of one CitusDB slave per physical Mesos slave.
• /var/lib/docker/volumes/citus-slave can be changed (if necessary) to a different location on your Mesos slave's filesystem. This directory is where CitusDB will store it's data (rather than in a docker volume, for persistence).

The master will be run outside of Marathon / Mesos to ensure Marathon doesn't reschedule it, and to allow for using a TCP port outside of Mesos' allocation (31000-32000).

Simply run this docker command to start a CitusDB master, and have it automatically contact the slaves:

docker run -d --restart always \
    -p 5439:5432 \
    -e SERVICE_NAME=citusdb-master \
    -e CITUS_MASTER=true \
    -e CITUS_SLAVE_NAME=citusdb \
    -e CONSUL_ADDR=172.17.42.1:8500 \
    <your registry here>/citusdb-cloud

This will start a CitusDB master accessible on port 5439 on the master, with a Consul service URL of citusdb-master.service.consul:5439

Feel free to leave a Github issue and I'll be happy to respond if it's a question specific to CitusDB on Mesos. Thanks!