AWSnycast is a routing daemon for AWS route tables, to simulate an Anycast like service, and act as an extension of in-datacenter Anycast. It can also be used to provide HA NAT service.
master is currently highly unstable.
If you're crazy enough to want to use this, please use release version 0.0.8
A common pattern in AWS is to route egressing traffic (from private subnets) through a NAT instance in a public subnet. AWSnycast can manage one or more NAT instances for you automatically, allowing you to deploy one or more NAT instances per VPC.
You can deploy 2 NAT machines, in different availability zones, and configure a private routing table for each AZ. AWSnycast can then be used to manage the routes to these NAT machines for High Availability, such that by default, both machines will be active (for their AZ - to avoid cross AZ transfer costs + latency), however if one machine fails, then the remaining machine will take over it's route for the duration that it's down.
In datacenters, a common pattern is to have a /24 network for Anycast, and then in each datacenter, use systems like exabgp and bird on hosts to publish BGP routing information for individual /32 IPs of services they're running.
In AWS we can configure VPN tunnels, or Direct Connects, and publish routes to the Amazon routing tables using BGP, including the Anycast /24. AWSnycast then runs locally on your AWS nodes, healthchecks local services, and publishes more specific routes to them into the AWS route table.
This means that all your systems in AWS can use the same network for Anycast services as your in datacenter machines, and talk to services locally, as you bring them up in AWS. This is super useful for bootstrapping a new VPC (before you have any local services running), or for providing high availability.
You don't have to have a datacenter or VPN connection for AWSnycast to be useful to you, you still get route publishing based on healthchecks, and HA/failover, just not bootstrapping from in-datacenter.
N.B. Whilst publishing routes from AWS into your datacenter's BGP would be useful, at this time that is beyond the goals for this project.
You can setup routes in AWS to go to an individual instance. This means that you inject routes that are outside the addesses space for your VPC, and point them to an instance, failing over to a different instance if the machine providing the service fails.
All you have to do on the instance itself is setup a network interface which can deal with this traffic; for example, alias lo0:0 as 192.168.1.1 and disable source/destination checking for that instance in AWS.
By advertising a larger range via BGP (from VPN or Direct connect) and then injecting /32 routes for the AWS instances of individual services, you get both bootstrapping (being able to bootstrap new AWS regions from a VPN connection, as critical services such as dns and puppet/chef are always at a fixed IP address) and HA.
Good question! You can provide HA in AWS by assigning each service an ENI, and moving them between healthy instances.
There are a number of reasons I chose not to do this:
- Most AWS machine classes/sizes are fairly restricted about the number of ENIs they can have attached.
- ENIs need to be detached then re-attached - so failover isn't atomic and it makes writing reliable distributed software to do this hard without a strong consensus store. (Think detach-fight!)
- I explicitly didn't want to depend on a strong consensus store (to make this useful for VPC bootstrapping).
- ENIs are an AWS only solution, and don't/can't provide parity with existing Anycast implementations in datacenter.
Basic infrastructure services, like DNS (if you use your own DNS already), puppet or chef servers, package repositories, etc. This is not a load balancing, or SOA service discovery solution.
I'd highly recommend putting any TCP service you use AWSnycast for behind haproxy, for load balancing (as only one route can be active at a time - i.e. AWS doesn't support any form of ECMP), and to make AWSnycast's failover only needed when an instance dies, rather than an individual service instance.
If you're building an SOA, and for 'application level' rather than 'infrastructure level' services, I'd recommend checking out Consul, Smartstack and similar technologies.
In the tests/integration folder, there is Terraform code which will build a simple AWS VPC, with 2 AZs and 2 NAT machines (with HA and failover), and AWSnycast setup.
To try this, you can just run make then terraform apply in that directory to build the example network, then make sshnat to log into one of the machines, and journalctl -u awsnycast to view the logs.
Try terminating one of the machines and watch routes fail over!
FIXME!! More details about how to test / curl things here..
You can install binary release versions onto x86 Linux directly from github, e.g.
sudo wget https://github.com/bobtfish/AWSnycast/releases/download/v0.0.8/AWSnycast -O /usr/local/bin/AWSnycast
sudo chmod 700 /usr/local/bin/AWSnycast
or you can install the .deb or .rpm packages found at the same location
You need go installed to build this project (go 1.4 or 1.5).
Once you have go installed, and a GOPATH setup, you should be able to install with:
go get github.com/bobtfish/AWSnycast
go install github.com/bobtfish/AWSnycast
This will install the software to the bin directory in the first part of your GOPATH.
Run this to find it:
echo $(echo $GOPATH|cut -d: -f1)/bin/AWSnycast
You can run AWSnycast -h to get a list of helpful options:
Usage of AWSnycast:
-debug
Enable debugging
-f string
Configration file (default "/etc/awsnycast.yaml")
-noop
Don't actually *do* anything, just print what would be done
-oneshot
Run route table manipulation exactly once, ignoring healthchecks, then exit
Once you've got it fully setup, you shouldn't need any options.
To run it also needs permissions to access the AWS API. This can be done either by supplying the standard AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, or by applying an IAM Role to the instance running AWSnycast (recommended).
An example IAM Policy is shown below:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"ec2:ReplaceRoute",
"ec2:CreateRoute",
"ec2:DeleteRoute",
"ec2:DescribeRouteTables",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeInstanceAttribute"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
DescribeRouteTables + Create/Delete/ReplaceRoute are used for the core functionality, DescribeNetworkInterfaces is used to get the primary IP address of other instances which are found to be owning a route (for remote health checks), and DescribeInstanceAttribute is used to check that the local machine has src/dest checking disabled (and refusing to start if it doesn't as a safety precaution).
Note that this software does not need root permissions, and therefore should not be run as root on your system. Please run it as a normal user (or even as nobody if you're using an IAM Role).
Which routes to advertise into which route tables is configured with a YAML config file.
By default AWSnycast will look for this in /etc/awsnycast.yaml
An example config is shown below:
---
poll_time: 300 # How often to poll AWS route tables
healthchecks:
public:
type: ping
destination: 8.8.8.8
rise: 2 # How many need to succeed in a row to be up
fall: 10 # How many need to fail in a row to be down
every: 1 # How often, in seconds
localservice:
type: tcp
destination: 192.168.1.1
rise: 2
fall: 2
every: 30
config:
send: HEAD / HTTP/1.0 # String to send (optional)
expect: 200 OK # Response to get back (optional)
port: 80
remote_healthchecks:
service:
type: tcp
rise: 20 # Note we set these a lot higher than local healthchecks
fall: 20
every: 30
config:
send: HEAD / HTTP/1.0 # String to send (optional)
expect: 200 OK # Response to get back (optional)
port: 80
routetables:
# This is our AZ, so always try to takeover routes always
our_az:
find:
type: and
config:
- type: by_tag
config:
key: az
value: eu-west-1a
- type: by_tag
config:
key: type
value: private
manage_routes:
- cidr: 0.0.0.0/0 # NAT box, so manage the default route
instance: SELF
healthcheck: public
never_delete: true # Don't delete the default route if google goes down ;)
- cidr: 192.168.1.1/32 # Manage an AWSnycast service on this machine
instance: SELF
healthcheck: localservice
remote_healthcheck: service
# This is not our AZ, so only takeover routes only if they don't exist already, or
# the instance serving them is dead (terminated or stopped).
# Every backup AWSnycast instance should have if_unhealthy: true set for route tables
# it is the backup server for, otherwise multiple instances can cause routes to flap
other_azs:
find:
no_results_ok: true # This allows you to deploy the same config in regions you only have 1 AZ
type: and
config:
filters:
- type: by_tag
not: true
config:
key: az
value: eu-west-1a
- type: by_tag
config:
key: type
value: private
manage_routes:
- cidr: 0.0.0.0/0
if_unhealthy: true # Note this is what causes routes only to be taken over not present currently, or the instance with them has failed
instance: SELF
- cidr: 192.168.1.1/32
if_unhealthy: true
instance: SELF
healthcheck: localservice
remote_healthcheck: service
Healthchecks are indicated by the top level 'healthchecks' key. Values are a hash of name / definition.
The definition is composed of a few fields:
- type - required
- destination - required. The destination IP for the healthcheck. N.B. this must be an IP.
- rise - optional, how many checks need to pass in a row to become healthy. Default 2
- fall - optional, how many checks need to fail in a row to become unhealthy. Default 2
- every - required, how often in seconds to run the healthcheck
- config - optional, A hash of keys/values for the specific healthcheck type you are using
Does an ICMP ping against the destination.
Makes a TCP connection against the destination on a port. Optionally sends data and checks returned data.
- port - required, the port number to connect on
- send - optional, a string to send to the remote side
- expect - optional, a string to expect back in the response from the remote side
Indicated by the top level 'route_tabls' key. Values are a hash of name / definition.
The definition is composed of a few fields:
- find (see Finding them below)
- manage_routes (see Managing them below)
Route tables are found by various 'filters' which you can configure. Each finder has a 'type' and some 'config'.
Depending on the type of your finder, different config keys are required / valid.
All filters can optionally take a 'not' key, which if true inverts the result
Top level filters can optionally take a 'no_results_ok' key, which if true stops AWSnycast from failing if no route tables can be found. Use this when you want a setup with a backup server / az in some regions, but you want to deploy the same config in all regions.
Currently supported filters are:
Does a simple equality match on a tag. Expects config keys 'key' and 'value', whos values are exaclty matched with the tag's key and value.
Runs a series of other filters, and only matches if all of it's filters match.
Supply a list to the 'filters' config key, with values being hashes of other finders
Runs a series of other filters, and matches if any of it's filters match.
Supply a list to the 'filters' config key, with values being hashes of other finders
Matches the main route table only. Takes no config parmeters
Matches the route table associated with the subnet given in the 'subnet_id' config key
Matches any route tables which have a route to a specific (and exact) cidr (given by the 'cidr' config key).
Routes to be managed are a list of hashes, with the following keys:
- cidr - required. The address to advertise into the route table
- instance - required. The Amazon instance ID to route this cidr to. Can be SELF to mean this instance
- healthcheck - optional. The string name of the healthcheck to associate with this route. If the healthcheck doesn't pass then the route will be removed from the routing table (allowing you to failover to a wider scope Anycast route advertised from your datacenter)
- if_unhealthy - true. Only take this route over if the instance currently associated with it is unhealthy in the AWS route table (i.e. black holing traffic). This is used for backup servers in a multi-az deployment.
Release (stable) versions of AWSnycast are tagged in the repository, and go binaries (generated by Travis CI) are uploaded to github - see the CHANGELOG.md for information about what is in each release.
Note that currently this project is pre-1.0, so I reserve the right to make massive sweeping changes between versions.
Also, I've properly tested only the stuff in tests/integration, so YMMV and this may eat your lunch, and completely mess up your AWS account! You are HIGHLY recommended to become confident using the -noop mode before running this for real!
Once I've got something that I am happy deploying to production, I'll cut a 1.0.0 and switch to semver.
This project is currently under heavy development.
Here's a list of the features that I'm planning to work on next, in approximate order:
- Remote healthchecks
- Command healthcheck
- Run scripts before / after takeover of route
- Autodetect this machine's AZ
- Make us autodetect the VPC this instance is running in, and refuse to adjust routing tables in other VPCs
- Enable the use of multiple different healthchecks for a route (to only consider it down if multiple checks fail)
- Add serf gossip between instances, to allow faster and reliable failover/STONITH
- Add the ability to have external clients participate in healthchecks in the serf network.
- Add a web interface to be able to get the state as HTML or JSON (and manually initiate failovers?)
Bug reports, success(or failure) stories, questions, suggestions, feature requests and (documentation or code) patches are all very welcome.
Please feel free to ping t0m on Freenode or @bobtfish on Twitter if you'd like help configuring/using/debugging/improving this software.
Please note however that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.
Copyright Tomas Doran 2015
Apache2 - See the included LICENSE file for more details.