A tool to assist in creation of the
histograph.io stack on AWS.
We create a cloud-init configuration file (see cloudinit.py for implementation details). This configuration is read by the machine on
startup and it will
- create users with SSH keys, sudo rights
- create application system user
- write and execute install scripts
(More info on cloud-init here)
This configuration is then gzipped, base64 encoded and passed to EC2
when launching an instance. Launching, tagging, waiting and other AWS
functionality is found in aws.py which implements this
using boto3.
Finally, setting up the various nodes in done using shell scripts,
see scripts/
Amazon Linux (CentOS) scripts:
scripts/install-redis.shinstall Redisscripts/install-core.shinstall histograph-corescripts/install-api.shinstalls histograph-apiscripts/install-neo4j.shinstalls Neo4J on Debian machine.scripts/install-nodejs-repository.shinstalls Node.JS repository (from https://github.com/nodesource/distributions)
Node processes are kept running using forever, which I find
slightly irritating. Some functions to create init scripts
for it are found in
scripts/library-functions.sh
Finally, after starting a machine the systems tries to login with SSH and
when sucessful it will tail /var/log/cloud-init-output.log.
Common settings are stored in cluster.yaml,
it looks like this:
users:
user-name: "~/.ssh/id_rsa.pub ssh key contents"
base-conf:
# default image, amazon linux
machine-image: ami-a6b0b7bb
# which VPC and subnet you want the instances to start in
vpc: vpc-6865cc01
subnet: subnet-71b36f0a
region: eu-central-1
# associated security group
security-group: sg-baac24d3
# default instance type
instance-type: t2.micro
hosts:
api:
# when set, this will create a system user with this name/group
app-user: histograph
# machines are assigned a static IP (for now),
# make sure this corresponds with your VPC/subnet settings
ip-address: 10.0.0.51
# override the default instance-type
instance-type: t2.small
# each of these scripts is saved to /root/ with permissions 0700
# and then executed, in the order specified here
scripts:
# this scripts registers the node repo
- install-nodejs-repository.sh
# some functions, to install node, forever, create init scripts
- library-functions.sh
# setup histograph API
- install-histograph-api.sh
core:
ip-address: 10.0.0.52
instance-type: t2.micro
app-user: histograph
scripts:
- install-nodejs-repository.sh
- library-functions.sh
- install-histograph-core.sh
redis:
ip-address: 10.0.0.53
instance-type: t2.micro
scripts:
- install-redis.sh
neo4j:
ip-address: 10.0.0.54
instance-type: m3.large
# we run Neo4J on Debian 8 (jessy)
machine-image: ami-b092aaad
scripts:
- install-neo4j.shFor each user you want to grant access to the instance: Replace 'name' with the username you wish to grant access. Replace 'key' with the public key (from the pem file you created) for the user (on aws).
The configuration and scripts work together, so beware.
This requires python3,
brew install python3
You should prefer to run python in a jail called virtualenv.
pip3 install virtualenv
Create jail in subdir venv/, ensure Python version 3.
virtualenv -p python3 venv
Enter the jail through shell magic.
source venv/bin/activate
(or source venv/bin/activate.fish if you are using
fish)
And now install the requirements
pip3 install -r requirements.txt
Ensure that you have AWS credentials setup
# ~/.aws/credentials
[default]
aws_access_key_id = YOUR_KEY
aws_secret_access_key = YOUR_SECRET
Also set up a default region (in e.g. ~/.aws/config):
# ~/.aws/config
[default]
region = eu-central-1
Now you can run the scripts.
If you have AWS environment variables set, these will be picked up by boto3.
Beware they might conflict with ~/.aws/credentials.
In bash you can clear them like this:
unset AWS_SECRET_ACCESS_KEY
unset AWS_ACCESS_KEY_ID
Creating a machine from the config above:
./aws-tool create cluster.yaml redis
By default a 'dry run' is enabled. Change the DryRun=True value to False or comment out the option in aws.py to do a proper run.
Then wait... You will hopefully see the logs. Check if all went fine, then create next instance.
./aws-tool create cluster.yaml neo4j
./aws-tool create cluster.yaml api
./aws-tool create cluster.yaml core
This is not ideal, but good enough for now.
We use two kinds of images, Debian (Neo4J) and Amazon Linux (everything else).
To find all official Debian owned images, run this command. You should prefer HVM over paravirtual and please note that image identifiers are region dependent.
aws --region eu-central-1 ec2 describe-images --owners 379101102735 \
--filters "Name=name,Values=debian-*" \
--query "Images[*].[Architecture,ImageId,VirtualizationType,Name]" \
--output text
Run the script ./setup_staging.sh. This script will clone ElasticSearch's repository and create a new cluster pointing at it.
Then it will clone the disk of Neo4j's production instance and create and launch an instance based on the cloned disk.
The id of the instance to clone is specified in the config file cluster_staging.yml, see example below.
Finally, it will launch and install the following machines:
redis-staging
core-staging
api-staging
To tear down the staging environment, run the script teardown_staging.sh.
Tearing down is not fully automated, you need to perform some actions as specified in the script's output.
The setting for the staging are stored in cluster_staging.yaml,
it looks like this:
users:
# just place your ~/.ssh/id_*.pub key contents here
# if you dont have such a public key, run `ssh-keygen`
stefano: "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAxIMaALlvQooxnPj9NiDMyhMap7IX0j4Yq/LHEZc+c4sj/KQXjovM845F/H8yj9r5Ibw4YWzkKROB9fRW+ZYcR43dlbDmlf9hZO/QYtWuk3ZC5DOvqBQ2+/Ume2sU70nyhR3u+Y60cXUhpSrr5sf1yTiQweNk8VhfezjXFCpcEUhEFtBYHiVrGY4wCIsx9IZ63Pr41A+pYiqMINXgxw/cB9s4uMIyNBG8NIiaPJC3MJgpfaX3FXLKT9BefrJP3kWBh1jTMfYyDiKCgerMh/2d5YPSiWDt1R3SPh9jQ0WgckCQsbMgl8df9Um+8iEO63iI88PIw11sEAvlE/wlUN03kQ== stefano@waag"
base-conf:
# default image, amazon linux
machine-image: ami-a6b0b7bb
subnet: subnet-1a960e73 #staging subnet
security-group: sg-b9ed94d0
vpc: vpc-6865cc01
region: eu-central-1
instance-type: t2.micro
app-user: histograph
hosts:
api-staging:
ip-address: 10.0.1.51
instance-type: t2.small
app-user: histograph
scripts:
- install-nodejs-repository.sh
- library-functions.sh
- staging/install-histograph-api.sh
core-staging:
ip-address: 10.0.1.52
instance-type: t2.micro
app-user: histograph
scripts:
- install-nodejs-repository.sh
- library-functions.sh
- staging/install-histograph-core.sh
redis-staging:
ip-address: 10.0.1.53
instance-type: t2.micro
scripts:
- install-redis.sh
# these two are cloned from production i/o created from scratch
neo4j:
ip-address: 10.0.1.54
instanceId: i-af0db813
# instance-type: m3.large
# machine-image: ami-b092aaad # debian 8 (jessy)
# scripts:
# - install-neo4j.sh
# elasticsearch:
# ip-address: 10.0.1.55
# instance-type: m3.xlarge
# machine-image: ami-b092aaad # debian 8 (jessy)
# scripts:
# - install-elasticsearch.shCopyright (C) 2015 Waag Society.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent