P.S.: I have setup 3 Nodes Kubernetes Cluster on GCP
a. 3 VMs having specification of (2 vCPUs, 2 GB memory, 10GB storage)
b. Open ports 22, 80, 443, 6443 within VPC/Compute Network
(P.S. These step are Not limited to 2 worker nodes only, If you have N
number of worker nodes, just repeat worker nodes specific steps on each worker node)
-
SSH into all 3 VMs & switch to root user using
sudo -i
-
Open URL -> Container Runtime in browser (or another tab window)
-
Select Ubuntu 16.04+ Tab to executes commands on all 3 VMs to setup Docker runtime.
Tip: Last command executed through this link should be
sudo systemctl enable docker
-
Run
docker ps
on each VM and you should see output
root@terraform-k8s-worker-1:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
root@terraform-k8s-worker-1:~#
- Execute below Prerequisite on all 3 VMs (Don't miss this step during docker runtime Setup):
sudo modprobe overlay
sudo modprobe br_netfilter
# Set up required sysctl params, these persist across reboots.
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
sudo sysctl --system
-
Open kubernetes documentation link -> Kubernetes Documentation and search
install kubeadm
in search box on left hand side -
Open link from Search Result -> Installing kubeadm | Kubernetes
On this page Jump on to section Installing kubeadm, kubelet and kubectl
Select Tab Ubuntu, Debian Or HypriotOS and execute following set of commands from this section on all 3 VMs:
sudo apt-get update && sudo apt-get install -y apt-transport-https curl curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list deb https://apt.kubernetes.io/ kubernetes-xenial main EOF sudo apt-get update sudo apt-get install -y kubelet kubeadm kubectl sudo apt-mark hold kubelet kubeadm kubectl
-
Restarting the kubelet is required:
systemctl daemon-reload
systemctl restart kubelet
Important!: To setup master node in the cluster please Switch to Master Node VM terminal
- Search for Creating a cluster with kubeadm to run
kubeadm init
on Master node VM , Open link from search result --> Creating a cluster with kubeadm | Kubernetes
On this page Jump on to section More information
Execute kubeadm initilisation command
kubeadm init
# You will see output in console like..
[init] Using Kubernetes version: vX.Y.Z
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
.
.
.
.
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a Pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join <control-plane-host>:<control-plane-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash>
- Copy the following
kubeadm init
command output, into handy text editor likenotepad/notepad++/sublimetext
etc
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a Pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join <control-plane-host>:<control-plane-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash>
-
To make kubectl work for your non-root user from Master Node, run these commands
- Exit from root-user for example
root@terraform-k8s-master-0:~# exit
logout
bapus_pro_2020@terraform-k8s-master-0:~$
- Run as a normal user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
- Now run
kubectl get nodes
command, there should be only one node (i.e. Master node in Not Ready Status)
bapus_pro_2020@terraform-k8s-master-0:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
terraform-k8s-master-0 Not Ready master 40s v1.19.2
bapus_pro_2020@terraform-k8s-master-0:~$
-
Now to apply Network policy to establish network communication within cluster between master node & worker node. And get Master node in ready state. Here we will setup Calico network policy
- Open Calico Network Policy to setup on Master node
Tip: To open calico network policy documentation through Google search, In Google search type "calico.yaml kubernetes" and open first link "Install Calico networking and network policy for on-premises ..." & Jump to Section Install Calico with Kubernetes API datastore, 50 nodes or less
- Now Run
kubectl apply -f <filename_or_url>
command on Master node
curl https://docs.projectcalico.org/manifests/calico.yaml -O kubectl apply -f calico.yaml
OR
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
- Check Network policy apply run result. Execute
kubectl get nodes -w
& wait for command to output ready status
kubectl get nodes -w NAME STATUS ROLES AGE VERSION terraform-k8s-master-0 Not Ready master 2m9s v1.19.2 terraform-k8s-master-0 Not Ready master 2m10s v1.19.2 terraform-k8s-master-0 Not Ready master 2m11s v1.19.2 terraform-k8s-master-0 Ready master 2m12s v1.19.2 terraform-k8s-master-0 Ready master 2m13s v1.19.2 terraform-k8s-master-0 Ready master 2m14s v1.19.2 . . #Now hit Control+c
- Now run
kubectl get nodes
command,
bapus_pro_2020@terraform-k8s-master-0:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION terraform-k8s-master-0 Ready master 2m40s v1.19.2 bapus_pro_2020@terraform-k8s-master-0:~$
Now Join each worker Nodes to Master node in the cluster by running kubeadm join
command from Worker node VM
Important!: To join worker node in the cluster, Please Switch to Worker Node's VM terminal (Repeat this step for each worker node)
- Copy
kubeadm join xxxxxx
command from text editor previously copied askubeadm init
command output & run on each Worker Node i.e.
kubeadm join <control-plane-host>:<control-plane-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash>
Example:
root@terraform-k8s-worker-1:~# kubeadm join 10.142.0.4:6443 --token fia5f8.2x5ud7uhwgxlcfb2 \
> --discovery-token-ca-cert-hash sha256:42b95610799f923b5ffbc770dd5951a7870aa53efb1ba28938bec69e208e30fd
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
root@terraform-k8s-worker-1:~#
Repeat this step for each Worker Node
- Now Switch to Master Node VM terminal and run
kubectl get nodes
command
bapus_pro_2020@terraform-k8s-master-0:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
terraform-k8s-master-0 Ready master 3m5s v1.19.2
terraform-k8s-worker-0 Ready <none> 3s v1.19.2
terraform-k8s-worker-1 Ready <none> 4s v1.19.2
bapus_pro_2020@terraform-k8s-master-0:~$