A small howto on how to bring up a self-hosted kubernetes cluster
We'll use bootkube to initiate the master-components. First we'll render the assets necessary for bringing up the control plane (apiserver, controller-manger, scheduler, etc). Then we'll start the kubelets which job is it to start the assets but can't do much, because there's no API-server yet. Running bootkube once will kick things off then. At a high-level the bootstrapping process looks like this:
Image taken from the self-hosted proposal.
This is how the final cluster looks like from a kubectl perspective:
Let's start!
wget https://github.com/kubernetes-incubator/bootkube/releases/download/v0.3.9/bootkube.tar.gz
tar xvzf bootkube.tar.gz
sudo cp bin/linux/bootkube /usr/bin/
Exchange 10.7.183.59 with the node you are working on. If you have DNS available group all master node IP addresses behind a CNAME Record and provide this insted.
bootkube render --asset-dir=assets --experimental-self-hosted-etcd --etcd-servers=http://10.3.0.15:2379 --api-servers=https://10.7.183.59:443
This will generate several things:
- manifests for running apiserver, controller-manager, scheduler, flannel, etcd, dns and kube-proxy
- a
kubeconfigfile for connecting to and authenticating with the apiserver - TLS assets
wget http://storage.googleapis.com/kubernetes-release/release/v1.5.3/bin/linux/amd64/hyperkube -O ./hyperkube
sudo mv hyperkube /usr/bin/hyperkube
sudo chmod 755 /usr/bin/hyperkube
sudo mkdir -p /opt/cni/bin
wget https://github.com/containernetworking/cni/releases/download/v0.4.0/cni-amd64-v0.4.0.tbz2
sudo tar xjf cni-amd64-v0.4.0.tbz2 -C /opt/cni/bin/
sudo cp assets/auth/kubeconfig /etc/kubernetes/
sudo cp -a assets/manifests /etc/kubernetes/
sudo hyperkube kubelet --kubeconfig=/etc/kubernetes/kubeconfig \
--require-kubeconfig \
--cni-conf-dir=/etc/kubernetes/cni/net.d \
--network-plugin=cni \
--lock-file=/var/run/lock/kubelet.lock \
--exit-on-lock-contention \
--pod-manifest-path=/etc/kubernetes/manifests \
--allow-privileged \
--node-labels=master=true \
--minimum-container-ttl-duration=6m0s \
--cluster_dns=10.3.0.10 \
--cluster_domain=cluster.local \
--hostname-override=10.7.183.59
The TLS credentials generated by bootkube render in assets/tls/ are copied to a secret: assets/manifests/kube-apiserver-secret.yaml.
bootkube will serve as the temporary apiserver so the kubelet from above can start the real apiserver in a pod
sudo bootkube start --asset-dir=./assets --experimental-self-hosted-etcd --etcd-server=http://127.0.0.1:12379
bootkube should exit itself after successfully bootstrapping the master components. It's only needed for the very first bootstrapping
watch hyperkube kubectl get pods -o wide --all-namespaces
Copy the information where to find the apiserver and how to authenticate:
scp 10.7.183.59:assets/auth/kubeconfig .
sudo mkdir -p /etc/kubernetes
sudo mv kubeconfig /etc/kubernetes/
install cni binaries and download hyperkube
sudo mkdir -p /opt/cni/bin
wget https://github.com/containernetworking/cni/releases/download/v0.4.0/cni-amd64-v0.4.0.tbz2
sudo tar xjf cni-amd64-v0.4.0.tbz2 -C /opt/cni/bin/
wget http://storage.googleapis.com/kubernetes-release/release/v1.5.3/bin/linux/amd64/hyperkube -O ./hyperkube
sudo mv hyperkube /usr/bin/hyperkube
sudo chmod 755 /usr/bin/hyperkube
Start the kubelet:
sudo hyperkube kubelet --kubeconfig=/etc/kubernetes/kubeconfig \
--require-kubeconfig \
--cni-conf-dir=/etc/kubernetes/cni/net.d \
--network-plugin=cni \
--lock-file=/var/run/lock/kubelet.lock \
--exit-on-lock-contention \
--pod-manifest-path=/etc/kubernetes/manifests \
--allow-privileged \
--node-labels=master=true \
--minimum-container-ttl-duration=6m0s \
--cluster_dns=10.3.0.10 \
--cluster_domain=cluster.local \
--hostname-override=10.7.183.60
Note the only difference is the removal of --node-labels=master=true:
sudo hyperkube kubelet --kubeconfig=/etc/kubernetes/kubeconfig \
--require-kubeconfig \
--cni-conf-dir=/etc/kubernetes/cni/net.d \
--network-plugin=cni \
--lock-file=/var/run/lock/kubelet.lock \
--exit-on-lock-contention \
--pod-manifest-path=/etc/kubernetes/manifests \
--allow-privileged \
--minimum-container-ttl-duration=6m0s \
--cluster_dns=10.3.0.10 \
--cluster_domain=cluster.local\
--hostname-override=10.7.183.60
kubectl apply doesn't work for TPR at the moment. See kubernetes/kubernetes#29542. As a workaround, we use cURL to resize the cluster.
hyperkube kubectl --namespace=kube-system get cluster.etcd kube-etcd -o json > etcd.json && \
vim etcd.json && \
curl -H 'Content-Type: application/json' -X PUT --data @etcd.json http://127.0.0.1:8080/apis/etcd.coreos.com/v1beta1/namespaces/kube-system/clusters/kube-etcd
If that doesn't work, re-run until it does. See kubernetes-retired/bootkube#346 (comment)
Some Broadcom NICs panic'ed with the default Ubuntu kernel
- upgrade kernel to >
4.8because of brcm nic failure - move to
--storage-driver=overlay2instead ofaufsas docker driver - disable swap on the node (will be a fatal error in kube-1.6)
the master apiservers need to have a single address only. Possible solutions:
- use LB from the DC
- use DNS from the DC with programmable API (e.g. powerdns)
- use something like kube-keepalive-vip?
- bootstrap DNS itself (skydns, coredns)
- backup strategies (https://github.com/coreos/etcd-operator/blob/master/doc/user/spec_examples.md#three-members-cluster-that-restores-from-previous-pv-backup)
- etcd-operator failures (e.g. coreos/etcd-operator#851)
- partial failure (loosing quorum)
- permament failure (state gone completely)
- etcd needs ntp available (or another mechanism so that every node is in sync)
sudo su -
docker rm -f $(docker ps -a -q)
exit
sudo docker run --rm -it -v $(pwd)/golang/src:/go/src/ -w /go/src golang:1.7 bash
go get -u github.com/kubernetes-incubator/bootkube
cd $GOPATH/src/github.com/kubernetes-incubator/bootkube
make
./bootkube-rbac render --asset-dir assets-rbac --experimental-self-hosted-etcd --etcd-servers=http://10.3.0.15:2379 --api-servers=https://10.7.183.59:443
sudo rm -rf /etc/kubernetes/*
sudo cp -a assets-rbac/manifests /etc/kubernetes/
sudo cp assets-rbac/auth/kubeconfig /etc/kubernetes/
sudo ./bootkube-rbac start --asset-dir=./assets-rbac --experimental-self-hosted-etcd --etcd-server=http://127.0.0.1:12379
The benefit here is using a docker container instead of a kubelet binary. Also the hyperkube docker image packages and installs the cni binaries. The downside would be that in either case something needs to start the container upon a reboot of the node. Usually the something is systemd and systemd is better managing binaries than docker containers. Either way, this is how you would run a containerized kubelet:
sudo docker run \
--rm \
-it \
--privileged \
-v /dev:/dev \
-v /run:/run \
-v /sys:/sys \
-v /etc/kubernetes:/etc/kubernetes \
-v /usr/share/ca-certificates:/etc/ssl/certs \
-v /var/lib/docker:/var/lib/docker \
-v /var/lib/kubelet:/var/lib/kubelet \
-v /:/rootfs \
quay.io/coreos/hyperkube:v1.5.3_coreos.0 \
./hyperkube \
kubelet \
--network-plugin=cni \
--cni-conf-dir=/etc/kubernetes/cni/net.d \
--cni-bin-dir=/opt/cni/bin \
--pod-manifest-path=/etc/kubernetes/manifests \
--allow-privileged \
--hostname-override=10.7.183.60 \
--cluster-dns=10.3.0.10 \
--cluster-domain=cluster.local \
--kubeconfig=/etc/kubernetes/kubeconfig \
--require-kubeconfig \
--lock-file=/var/run/lock/kubelet.lock \
--containerized
Not quite working yet though. The node comes up, registeres successfully with the master and starts daemonsets. Everything comes up except flannel:
main.go:127] Failed to create SubnetManager: unable to initialize inclusterconfig: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
- https://github.com/kubernetes/community/blob/master/contributors/design-proposals/self-hosted-kubernetes.md
- https://github.com/kubernetes-incubator/bootkube
- https://github.com/coreos/etcd-operator/
- http://blog.kubernetes.io/2017/01/stronger-foundation-for-creating-and-managing-kubernetes-clusters.html
- kubernetes/kubeadm#127
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent