spiritsree / vagrant-k8s-cluster Goto Github PK

View Code? Open in Web Editor NEW

2.0 2.0 1.0 110 KB

A Kubernetes Cluster in Vagrant

License: MIT License

Shell 100.00%

k8s k8s-cluster vagrant kubernetes kubernetes-cluster

vagrant-k8s-cluster's Introduction

Vagrant-k8s-cluster

This installs a test Kubernetes cluster in vagrant using virtualbox hosts..

Requirements

Setup

You can setup the cluster and kubectl context using the setup.sh script. This configures 1 Master node and 3 Worker nodes. You can change the number of worker nodes in Vagrantfile by updating the value of NODE_COUNT.

$ ./setup.sh -h
Kubernetes cluster setup on vagrant.

Usage:
    setup.sh [-h|--help] [-n|--networking <flannel|calico|canal|weavenet>] [-c|--host-count <n>]

Arguments:
    -h|--help                                             Print usage
    -n|--networking <flannel|calico|canal|weavenet>       Kubernetes networking model to use [Default: flannel]
    -c|--host-count <n>                                   Number of worker nodes [Default: 2]

Examples:
    ./setup.sh
    ./setup.sh -n calico
    ./setup.sh -n weavenet -c 3

Destroy

You can destroy the cluster and kubectl config using destroy.sh script.

$ sh destroy.sh

Supported Networking

Other Networking

Container Runtimes

Installations

Releases

Reference

Troubleshooting

If you see an error as below while running the setup

There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["hostonlyif", "create"]

Stderr: 0%...
Progress state: NS_ERROR_FAILURE

Do a reinstall of Virtualbox and allow Oracle from System Preferences > Security & Privacy

Using Virtualbox 6.1.28 onwards need more configuration for host-only network. Details here also the Changelog

Create a file /etc/vbox/networks.conf with allowed IP ranges for Virtualbox.

$ cat /etc/vbox/networks.conf
* 172.28.128.0/24
* 192.168.56.0/24

Installing Containerd

Install containerd

# curl -sSLO https://github.com/containerd/containerd/releases/download/v1.6.14/containerd-1.6.14-linux-amd64.tar.gz
# tar xzvf containerd-1.6.14-linux-amd64.tar.gz -C /usr/local

Install runc

# curl -sSLO https://github.com/opencontainers/runc/releases/download/v1.1.3/runc.amd64
# install -m 755 runc.amd64 /usr/local/sbin/runc

Install CNI plugins

# curl -sSLO https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz
# mkdir -p /opt/cni/bin
# tar xzvf cni-plugins-linux-amd64-v1.1.1.tgz -C /opt/cni/bin

Configure containerd

# mkdir /etc/containerd
# containerd config default | sudo tee /etc/containerd/config.toml
# sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml
# curl -L https://raw.githubusercontent.com/containerd/containerd/main/containerd.service -o /etc/systemd/system/containerd.service
# systemctl daemon-reload
# systemctl enable --now containerd
# systemctl status containerd

Configure kubelet to use containerd as runtime

Edit file /var/lib/kubelet/kubeadm-flags.env and add --container-runtime=remote and --container-runtime-endpoint=unix:///run/containerd/containerd.sock.

kubeadm toold stores the CRI socket for each host as an annotation in the Node object. To change it you can execute the following command

$ kubectl edit node <node-name>

in the editor change the value of kubeadm.alpha.kubernetes.io/cri-socket from /var/run/dockershim.sock to CRI socket path of your choice, in this case (unix:///run/containerd/containerd.sock) and save the change.

Restart kubelet

# systemctl restart kubelet

Check if the runtime is changed

# kubectl get nodes -o wide

Running test

From any node run the below to test container runtime

$ critest -parallel 10 -ginkgo.succinct

Versions

Tested with below versions of the apps

Vagrant 2.3.4 (2.3.2 or higher required with Virtualbox 7.x)
VirtualBox 6.1.40/7.0.4 (6.1.28 and higher versions have issue with host-only network. Pls check the troubleshooting section for details)
yq 4.6.1
ubuntu/xenial64 (v20210623.0.0)
ubuntu/bionic64 (v20220317.0.0)

vagrant-k8s-cluster's People

Contributors

Stargazers

Watchers

Forkers

gokulsiteminder

vagrant-k8s-cluster's Issues

Sometimes the entire config won't get downloaded resulting in node display failure.

MASTERIP="172.28.128.3"
nc -C -I 8192 ${MASTERIP} 8888 | tee kube_config

node-2: error: Error loading config file "/root/.kube/config": v1.Config.AuthInfos: []v1.NamedAuthInfo: v1.NamedAuthInfo.AuthInfo: v1.AuthInfo.ClientKeyData: decode base64: illegal base64 data at input byte 884, error found in #10 byte of ...|c2pEbUVMW"}}]}|..., bigger context ...|ZUxkcXNTdlVQTC9yUlc0cy9QUHdBeVdkai84T2hCc2pEbUVMW"}}]}|...

Not able to communicate to pods running on different nodes

I1031 09:40:34.153647    6557 round_trippers.go:452]     Content-Type: text/plain; charset=utf-8
I1031 09:40:34.153651    6557 round_trippers.go:452]     X-Content-Type-Options: nosniff
I1031 09:40:34.153655    6557 round_trippers.go:452]     Content-Length: 20
I1031 09:40:34.153659    6557 round_trippers.go:452]     Date: Wed, 30 Oct 2019 22:40:33 GMT
I1031 09:40:34.163020    6557 request.go:968] Response Body: service unavailable
I1031 09:40:34.173545    6557 request.go:1171] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
I1031 09:40:34.173564    6557 cached_discovery.go:78] skipped caching discovery info due to the server is currently unable to handle the request
I1031 09:40:34.174505    6557 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.16.2 (darwin/amd64) kubernetes/c97fe50" 'https://172.28.128.3:6443/api/v1/namespaces/kube-system/pods/metrics-server-677766dccd-wjqbs'
I1031 09:40:34.178307    6557 round_trippers.go:443] GET https://172.28.128.3:6443/api/v1/namespaces/kube-system/pods/metrics-server-677766dccd-wjqbs 200 OK in 3 milliseconds
I1031 09:40:34.178356    6557 round_trippers.go:449] Response Headers:
I1031 09:40:34.178369    6557 round_trippers.go:452]     Content-Type: application/json
I1031 09:40:34.178398    6557 round_trippers.go:452]     Content-Length: 2726
I1031 09:40:34.178406    6557 round_trippers.go:452]     Date: Wed, 30 Oct 2019 22:40:33 GMT
I1031 09:40:34.178475    6557 request.go:968] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"metrics-server-677766dccd-wjqbs","generateName":"metrics-server-677766dccd-","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/pods/metrics-server-677766dccd-wjqbs","uid":"38e91d55-fb05-11e9-a97a-02dd097c7616","resourceVersion":"6349","creationTimestamp":"2019-10-30T11:05:45Z","labels":{"k8s-app":"metrics-server","pod-template-hash":"677766dccd"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"metrics-server-677766dccd","uid":"384dccfc-fb05-11e9-a97a-02dd097c7616","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"tmp-dir","emptyDir":{}},{"name":"metrics-server-token-mhngh","secret":{"secretName":"metrics-server-token-mhngh","defaultMode":420}}],"containers":[{"name":"metrics-server","image":"k8s.gcr.io/metrics-server-amd64:v0.3.6","args":["--kubelet-insecure-tls","--kubelet-preferred-address-types=InternalIP"],"resources":{},"volumeMounts":[{"name":"tmp-dir","mountPath":"/tmp"},{"name":"metrics-server-token-mhngh","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"metrics-server","serviceAccount":"metrics-server","nodeName":"node-1","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-10-30T11:33:01Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2019-10-30T11:33:41Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-10-30T11:33:41Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-10-30T11:33:02Z"}],"hostIP":"172.28.128.4","podIP":"10.244.1.3","startTime":"2019-10-30T11:33:01Z","containerStatuses":[{"name":"metrics-server","state":{"running":{"startedAt":"2019-10-30T11:33:40Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"k8s.gcr.io/metrics-server-amd64:v0.3.6","imageID":"docker-pullable://k8s.gcr.io/metrics-server-amd64@sha256:c9c4e95068b51d6b33a9dccc61875df07dc650abbf4ac1a19d58b4628f89288b","containerID":"docker://3fa2a41fe4a471cf73da05d831e28191eefa39c589a1527fd28e8340c0ec337e"}],"qosClass":"BestEffort"}}
I1031 09:40:34.184320    6557 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.16.2 (darwin/amd64) kubernetes/c97fe50" 'https://172.28.128.3:6443/api/v1/namespaces/kube-system/pods/metrics-server-677766dccd-wjqbs/log'
I1031 09:40:34.188013    6557 round_trippers.go:443] GET https://172.28.128.3:6443/api/v1/namespaces/kube-system/pods/metrics-server-677766dccd-wjqbs/log 404 Not Found in 3 milliseconds
I1031 09:40:34.188032    6557 round_trippers.go:449] Response Headers:
I1031 09:40:34.188038    6557 round_trippers.go:452]     Content-Length: 276
I1031 09:40:34.188042    6557 round_trippers.go:452]     Date: Wed, 30 Oct 2019 22:40:33 GMT
I1031 09:40:34.188045    6557 round_trippers.go:452]     Content-Type: application/json
I1031 09:40:34.188064    6557 request.go:968] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the server could not find the requested resource ( pods/log metrics-server-677766dccd-wjqbs)","reason":"NotFound","details":{"name":"metrics-server-677766dccd-wjqbs","kind":"pods/log"},"code":404}
I1031 09:40:34.188538    6557 helpers.go:199] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server could not find the requested resource ( pods/log metrics-server-677766dccd-wjqbs)",
  "reason": "NotFound",
  "details": {
    "name": "metrics-server-677766dccd-wjqbs",
    "kind": "pods/log"
  },
  "code": 404
}]```

  message: 'no response from https://10.104.74.185:443: Get https://10.104.74.185:443:
     net/http: request canceled while waiting for connection (Client.Timeout exceeded
     while awaiting headers)'

Also tried to ping other pods from within pods which failed.

Add support for other networking model like calico

Add support for other networking model like calico etc.