barrucadu / govuk-k8s Goto Github PK

View Code? Open in Web Editor NEW

0.0 3.0 0.0 116 KB

My attempt to get GOV.UK running in Kubernetes on AWS

Shell 15.01% Nix 10.44% HCL 45.26% Python 8.43% Dockerfile 0.17% Dhall 20.70%

govuk-k8s's People

Contributors

Watchers

govuk-k8s's Issues

Switch to kops?

kops is a Kubernetes provisioning/management tool. Maybe I should use it?

Two reasons:

Cross-node pod networking is being a huge pain. Sometimes it works, sometimes it doesn't. And when it doesn't, it's broken from cluster startup and nothing I do fixes it. I can see that packets are getting to the flannel.1 interface (with tcpdump), but they're not being responded to. I suspect that there's a race condition in setting up the network stack somewhere, but I haven't found it yet.
I've got a lot of terraform, NixOS config, and bash. Using kops would mean I only have a fairly small amount of all of those.

veth* interfaces are assigned link-local IPs by dhcpcd, which breaks routing to the EC2 metadata service

veth* interfaces (there seems to be one for each docker container) try to get an IP address over DHCP. dhcpcd fails to produce one, so it assigns a link-local IP in the 169.254.0.0/16 range:

[root@ip-10-0-1-220:~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 02:f9:82:d2:31:02 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.220/24 brd 10.0.1.255 scope global dynamic noprefixroute eth0
       valid_lft 3558sec preferred_lft 3108sec
    inet6 fe80::f9:82ff:fed2:3102/64 scope link
       valid_lft forever preferred_lft forever
3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
    link/ether 6a:ac:12:9e:be:1d brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::68ac:12ff:fe9e:be1d/64 scope link
       valid_lft forever preferred_lft forever
4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
    link/ether 02:42:bd:fe:3f:11 brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.1/24 brd 10.1.1.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:bdff:fefe:3f11/64 scope link
       valid_lft forever preferred_lft forever
5: veth342a6aec@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP group default
    link/ether 82:62:fc:54:a2:43 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.119.170/16 brd 169.254.255.255 scope global noprefixroute veth342a6aec
       valid_lft forever preferred_lft forever
    inet6 fe80::14f8:6bff:fe49:17a8/64 scope link
       valid_lft forever preferred_lft forever
6: veth49f729a8@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP group default
    link/ether 46:cf:f2:53:31:42 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet 169.254.70.174/16 brd 169.254.255.255 scope global noprefixroute veth49f729a8
       valid_lft forever preferred_lft forever
    inet6 fe80::44cf:f2ff:fe53:3142/64 scope link
       valid_lft forever preferred_lft forever
    inet6 fe80::58d3:9fff:feea:cd8b/64 scope link
       valid_lft forever preferred_lft forever
7: veth4d8229b0@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP group default
    link/ether be:1e:e0:b0:e0:e6 brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet 169.254.163.163/16 brd 169.254.255.255 scope global noprefixroute veth4d8229b0
       valid_lft forever preferred_lft forever
    inet6 fe80::bc1e:e0ff:feb0:e0e6/64 scope link
       valid_lft forever preferred_lft forever
    inet6 fe80::c431:51ff:feb3:4fdf/64 scope link
       valid_lft forever preferred_lft forever

These interfaces all get a corresponding route:

[root@ip-10-0-1-220:~]# ip route
default via 10.0.1.1 dev eth0 proto dhcp src 10.0.1.220 metric 202 mtu 9001
10.0.1.0/24 dev eth0 proto dhcp scope link src 10.0.1.220 metric 202 mtu 9001
10.1.0.0/24 via 10.1.0.0 dev flannel.1 onlink
10.1.1.0/24 dev docker0 proto kernel scope link src 10.1.1.1
10.1.2.0/24 via 10.1.2.0 dev flannel.1 onlink
169.254.0.0/16 dev veth342a6aec scope link src 169.254.119.170 metric 205
169.254.0.0/16 dev veth49f729a8 scope link src 169.254.70.174 metric 206
169.254.0.0/16 dev veth4d8229b0 scope link src 169.254.163.163 metric 207

Unfortunately, these new routes break access to the EC2 metadata service, which responds on 169.254.169.254.

Possible solutions:

Preventing dhcpcd from responding to the veth* interfaces

I tried just blacklisting veth*, but then kubernetes networking broke, so it seems that the veth* interfaces do need an IP.
Making dhcpcd use a different IP range as its fallback

The service doesn't have any obvious-looking arguments to change the range.
Run a DHCP server on each k8s worker

Feels a bit overkill just to assign docker containers totally arbitrary and host-local IPs, but it may be the easiest way to keep 169.254.0.0/16 free

Cluster is more "slightly available" than "highly available"

There are two major problems with availability:

Everything is in the same AZ

There's only one public subnet and only one private subnet. Everything in the same subnet lives in the same availability zone, so that's not great.

Would be better to have subnets:

public_a, public_b, public_c
private_a, private_b, private_c`

And have node_group spread things around those subnets (eg: first node goes in a, second node goes in b, third node goes in c)

There's only one K8S master

...and the easycerts thing I'm using doesn't work in a multi-master set-up, so fixing this isn't just a case of deploying more masters and letting them know about each other.

Have a script to run a cluster locally

Running "for real" in AWS is nice, but not free. My desktop computer is pretty powerful, I think I could run everything locally.

For a local Kubernetes cluster, it looks like minikube or kind is the way to go.

Concourse and a Docker registry will be easy - there are docker images available.

DNS and ingress will be trickier. I'll need a DNS server running locally which can resolve govuk-k8s.test, and make it possible for the cluster to create some sort of load balancer and add new DNS entries.

It might be easier to make a server which responds to *.*.in-cluster.govuk-k8s.test and uses the given subdomain to figure out which service / namespace to route the request to, using NodePort to expose services to the host.

Apps run in development mode

Rails development mode monitors source files for changes and reloads them. I don't know what else it does, but googling for "rails development mode" turns up a lot of people asking how to make it faster, so it seems a good thing to leave behind if not necessary.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.