Install

• 3 node Flocker-Swarm cluster

  • 1 Control Node
  • 2 Agent Nodes
  • 1 Client Node to execute your docker commands from

  If you choose to create the stack manually, please restart Docker on Agent Node 1 with a tag flocker-node==1, and Agent Node 2 with tag flocker-node==2 (by adding --label flocker-node=${node_number} to DOCKER_OPTS variable in /etc/default/docker).

Please run below steps from the client node.

• Set DOCKER_HOST to point to Swarm Manager (export DOCKER_HOST=tcp://54.86.117.247:2376).

• git clone https://github.com/myechuri/flocker-pipelinedb.git

• cd flocker-pipelinedb

• Start PipelineDB server on Agent Node 1:

docker-compose -f pipelinedb-node1.yml up

• Connect to PipelineDB server on Agent Node 1, create database twitter with static stream tweets, and a continuous view tagstream that pulls hashtags from tweets within past one hour.

ubuntu@ip-172-31-7-164:~$ psql -h 54.173.243.221  -p 5432 -U pipeline
Password for user pipeline:


pipeline=# create database twitter;
CREATE DATABASE

pipeline=# \c twitter
You are now connected to database "twitter" as user "pipeline".
twitter=# create stream tweets ( content json );
CREATE STREAM

twitter=# CREATE CONTINUOUS VIEW tagstream as SELECT json_array_elements(content #> ARRAY['entities','hashtags']) ->> 'text' AS tag FROM tweets WHERE arrival_timestamp > ( clock_timestamp() - interval '1 hour' );

CREATE CONTINUOUS VIEW

• Set Twitter OAuth information in the twitter-workload.yml. twitter tokens/keys (reference: TwitterAPI documentation).

environment:
  CONSUMER_KEY: ''
  CONSUMER_SECRET: ''
  ACCESS_TOKEN_KEY: ''
  ACCESS_TOKEN_SECRET: ''
  PIPELINE_SERVER_HOST_IP:

Run the container for a few minutes, then terminate.

docker-compose -f twitter-workload-node1.yml up

If you want it to run in the background indefinitely add the -d flag

docker-compose -f twitter-workload-node1.yml up -d

• Connect to PipelineDB server on Agent Node 1, and verify workload reached the server.

twitter=# select * from tagstream limit 5;

1. Stop PipelineDB server on Agent Node 1

docker-compose -f pipelinedb-node1.yml stop
docker-compose -f pipelinedb-node1.yml rm -f

2. Move PipelineDB server to Agent Node 2 by Docker Composing up with a manifest constraining it to Agent Node 2.

docker-compose -f pipelinedb-node2.yml up

• Connect to PipelineDB server on Agent Node 2, then select from tagstream - you should see the same output as in Step 5.

twitter=# select * from tagstream limit 5;

This demo is based on PipelineDB workflow suggested in this blogpost.

This demo is made possible by:

• Josh Berkus, Derek Nelson, Usman Masood from PipelineDB.

• ClusterHQ Engineering and Marketing.