Divided into following sections :

• Managing Partitions and File Systems
• Logical Volume Management and RAID Levels
• User and Group Administration, SUDO and Permissions
• Network Configuration and Troubleshooting
• Managing SELinux
• Booting Procedure and Kernel parameters
• Job Automation
• Administrating Remote Systems (SSH)
• Memory Management (Swap)
• Software Management
• Backup and Restore
• Managing Installed Services
• Managing Process
• FTP (File Transfer Protocol) Server
• NFS (Network File System) Server, Autofs and LDAP Client
• Samba Server
• NTP (Network Time Protocol) or Chrony
• DNS (Domain Naming System)
• DHCP (Dynamic Host Configuration Protocol)
• Web Server (Apache)
• Mail Server
• iSCSI (Remote Storage)
• MySQL Server and MariaDB
• Log Server and Log Files
• Configuring IPtables and Firewall
• Virtualization
• General Questions
• Kickstart Installation and PXE (Network) Installation
• Veritas Volume Manager and Veritas Cluster
• RedHat Cluster

1. What is partition?
   
   • A partition is a contiguous set of blocks on a drive that are treated as independent disk.
2. What is partitioning?
   

• Partitioning means to divide a single hard drive into many logical drives. 3.Why we have multiple partitions?
• Encapsulate our data. Since file system corruption is limited to that partition only. So we can save our data from accidents.
• We can increase the disk space efficiency. Depending on our usage we can format the partition with different block sizes. So we can reduce the wastage of the disk.
• We can limit the data growth by assigning the disk quotas. 4.What is the structure of the disk partition?
• The first sector of the O/S disk contains the MBR (Master Boot Record). The MBR is divided into 3 parts and it's size is 512 bytes.
• The first part is IPL (Initial Program Loader) and it contains the Secondary Boot Loader. So, IPL is responsible for booting the O/S and it's size is 446 bytes.
• The second part is PTI (Partition Table Information). It contains the number of partitions on the disk, sizes of the partitions and type of the partitions. 5.Explain the disk partition criteria?
• Every disk can have max. 4 partitions. The 4 partitions are 3 Primary partitions and 1 Extended partition.
• The MBR and O/S will install in Primary partition only.
• The Extended partition is a special partition and can be further divided into multiple logical partitions.

6. How to identify the disks? In Linux different types of disks will be identified by different naming conventions.

• IDE) drives will be shown as /dev/hda, /dev/hdb, /dev/hdc, ...etc., and the partitions are /dev/hda1, /dev/hda2, /dev/hda3, ...etc.,
• iSCSI/SCSI and SATA drives will be shown as /dev/sda, /dev/sdb, /dev/sdc, ...etc., and the partitions are /dev/sda1, /dev/sda2, /dev/sda3, ...etc.,
• Virtual drives will be shown as /dev/vda, /dev/vdb, /dev/vdc, ...etc., and the partitions are /dev/vda1, /dev/vda2, /dev/vda3, ...etc., IDE -----> Integrated Drive Electronics.iSCSI -----> Internet Small Scale System Interface. SCSI -----> Small Scale System Interface.

7. What is file system?
   • It is a method of storing the data in an organized fashion on the disk. Every partition on the disk except MBR and Extended partition should be assigned with some file system in order to make them to store the data. File system is applied on the partition by formatting it with a particular type of file system.
8. What are the different types of file systems supported in Linux?
   • The Linux supported file systems are ext2, ext3, ext4, xfs, vfat, cdfs, hdfs, iso9660 ...etc., The ext2, ext3, ext4 file systems are widely used in RHEL-6 and xfs file system is introduced on RHEL-7. The vfat file system is used to maintain a common storage between Linux and Windows O/S.
   • The cdfs file system is used to mount the CD-ROMs and the hdfs file system is used to mount DVDs. The iso9660 file system is used to read CD/DVD.iso image format files in Linux O/S.
9. What is mounting and in how many types can we mount the partitions?

• Attaching a directory to the file system in order to access the partition and it's file system is known as mounting.
• In general the sub directories under /mnt directory are the mount points to mount the file systems. There two types of mountings in Linux/Unix.
• Temporary Mounting :
  • In a temporary mounting first we create a directory and mount the partition on that directory. But this type mounting will last only till the system is up and once it is rebooted the mounting will be lost. Example:# mount <directory name (mount point)>
• Permanent Mounting :
• In this also first we create the directory and open the /etc/fstab file and make an entry as below, Whenever the system reboots mount the partitions according to entries in /etc/fstab file. So, these type of mountings are permanently even after the system is rebooted. # mount -a to mount the partitions without reboot)

All the questions in this article are based on asked in interview .

Divided into following sections :

• Administration
• Compute
• Storage
• Network
• Security
• Monitoring
• Logging

Q: How to do maintenance activity on K8 node?

A: Maintenance activity are inevitable part of administration, you may need to do the patching or apply some security fixes on K8. Mark the node unschedulable and then drain the PODs which are present on K8 node.

• kubectl cordon
• kubectl drain --ignore-daemonsets

It's important to include the --ignore-daemonsets for any daemonset running on this node. Just in case if any statefulset is running on this node, and if no more node is available to maintain the count of statesful set then statesfulset POD will be in pending status.

Q: What is role of a pause container?

A: Pause container servers as the parent container for all the containers in your POD.

• It serves as the basis of Linux namespace sharing in the POD.
• PID 1 for each POD to reap the zombie processes.

https://www.ianlewis.org/en/almighty-pause-container

Q: Why we need service mesh?

A: A service mesh ensures that communication among containerized and often ephemeral application infrastructure services is fast, reliable, and secure. The mesh provides critical capabilities including service discovery, load balancing, encryption, observability, traceability, authentication and authorization, and support for the circuit breaker pattern.

Q: How to control the resource usage of a POD?

A: With requests and limits resource usage of a POD can be control.

request: the amount of resources being requested for a container. If a container exceeds its request for resources, it may be throttled back down to it’s request.

limit: an upper cap on the resources a container is able to use. If it tries to exceed this limit it may be terminated if Kubernetes decides that another container needs the resources. If you’re sensitive to pod restarts, it makes sense to have the sum of all container resource limits equal or less than the total resource capacity for your cluster.

https://www.noqcks.io/notes/2018/02/03/understanding-kubernetes-resources/

Q: What are the units of CPU and memory in POD definition?

A: CPU is in milicores and memory in bytes. CPU can be easily throttled but not memory.

Q: Where else we can set a resource limit?

A: You may also set resource limit on a namespace. This is helpful in scenarios where people have habit of not defining the resource limits in POD definition.

Q: How will you update the version of K8?

A: Before doing the update of K8, it's important to read the release notes to understand the changes introduced in newer version and whether version update will also update the etcd.

https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-12/

Q: Difference between helm and K8 operator?

A: An Operator is an application-specific controller that extends the Kubernetes API to create, configure and manage instances of complex stateful applications on behalf of a Kubernetes user. It builds upon the basic Kubernetes resource and controller concepts, but also includes domain or application-specific knowledge to automate common tasks better managed by computers. On the other hand, helm is a package manager like yum or apt-get.

Q: Explain the role of CRD (Custom Resource Definition) in K8?

A: A custom resource is an extension of the Kubernetes API that is not necessarily available in a default Kubernetes installation. It represents a customization of a particular Kubernetes installation. However, many core Kubernetes functions are now built using custom resources, making Kubernetes more modular.

Q: What are the various K8 related services running on nodes and role of each service?

A: Mainly K8 cluster consists of two type of nodes: master and executor

• master services:

  • kube-apiserver: Master API service which acts like a door to K8 cluster.
  • kube-scheduler: Schedule PODs according to available resources on executor nodes.
  • kube-controller-manager: controller is a control loop that watches the shared state of the cluster through the apiserver and makes changes attempting to move the current state towards the desired state

• executor node: (These also runs on master node)

  • kube-proxy: The Kubernetes network proxy runs on each node. This reflects services as defined in the Kubernetes API on each node and can do simple TCP, UDP, and SCTP stream forwarding or round robin TCP, UDP, and SCTP forwarding across a set of backends.
  • kubelet: kubelet takes a set of PodSpecs that are provided through various mechanisms (primarily through the apiserver) and ensures that the containers described in those PodSpecs are running and healthy

Q: Recommended way of managing the access to multiple clusters?

A: kubectl looks for the config file, multiple clusters access information can be specified in this config file. kubectl config commands can be used to manage the access to these clusters.

Q: What is PDB (Pod Disruption Budget)?

A: A PDB specifies the number of replicas that an application can tolerate having, relative to how many it is intended to have. For example, a Deployment which has a .spec.replicas: 5 is supposed to have 5 pods at any given time. If its PDB allows for there to be 4 at a time, then the Eviction API will allow voluntary disruption of one, but not two pods, at a time. This is applicable for voluntary disruptions.

Q: In what situations daemonsets are normally used?

A: Daemonset are used to start the PODs on every node in cluster. It's used generally to run the monitoring or logging agents which are supposed to run on every executor node in cluster.

Q: When stateful sets are preferred?

A: When you are running the applications which require quorum basically the applications which are not truely stateless for those applications stateful sets are required.

Q: What's init container and when it can be used?

A: init containers will set a stage for you before running the actual POD.

• Wait for some time before starting the app Container with a command like sleep 60.
• Clone a git repository into a volume.

Q: What are the application deployment strategies?

A: In this agile world there is continuous demand of upgrading the applciations, we have multiple options for deploying the new version of app:

1. Recreate: Old style, existing application version is destroyed and new version is deployed. Significant amount of downtime.
2. Rolling update: Gradually bringing down the existing deployment and introducing the new versions. You decide how many instances can be upgraded at single point of time.
3. Shadow: Traffic going to existing version of application is replicated to new version to see it's working. Istio provide this pattern.
4. A/B Testing using Istio: Running multiple variants of application together and determines the best one based on user traffic. It's more for managment decisions.
5. Blue/Green : Blue is mainly about switching the traffic from one version of app to another version.
6. Canary deployment : In which certain percentage of traffic is shifted from one version to another. If things work well we will keep on increasing the traffic shift. It's different from the rolling update in which existing version count is reduced gradually.

Q: How to troubleshoot if the POD is not getting scheduled?

A: There are many factors which can led to unstartable POD. Most common one is running out of resources, use the commands like kubectl desribe <POD> -n <Namespace> to see the reason why POD is not started. Also, keep an eye on kubectl get events to see all events coming from the cluster.

Q: How to run a POD on particular node?

A: Various methods are available to achieve it.

• nodeName: specify the node name in POD spec, it will try to run the POD on specific node.

• nodeSelector : you may assign a specific lable to node which have special resources and use the same label in POD spec so that POD will run only on that node.

• nodeaffinities: requiredDuringSchedulingIgnoredDuringExecution, preferredDuringSchedulingIgnoredDuringExecution are hard, soft requirements for running the POD on specific nodes. This will be replacing nodeSelector in future. It depends on the node labels.

Q: How to ensure PODs are colocated to get performance benefits?

A: podAntiAffinity and podAffinity are the affinity concept to not keep and keep the PODs on same node. Key point to note is that it depends on the POD labels.

Q: What are the taints and toleration?

A: Taints allow a node to repel a set of pods. You can set taints on the node and only the POD which have tolerations matching the taints condition will be able to run on those nodes. This is useful in the case when you allocated node for one user and don't want to run the PODs from other users on that node.

Q: How to provide persistent storage for POD?

A: Persistent volumes are used for persistent POD storage. They can be provision statically or dynamically.

Static : A cluster administrator creates a number of PVs. They carry the details of the real storage which is available for use by cluster users.

Dynamically : Administrator creates a PVC (Persistent volume claim) specifying the existing storage class and volume created dynamically based on PVC.

Q: How two containers running in a single POD have single IP address?

A: Kubernetes implements this by creating a special container for each pod whose only purpose is to provide a network interface for the other containers. These is one pause container which is responsible for namespace sharing in the POD. Generally, people ignore the existance of this pause container but actually this container is the heart of network and other functionalities of POD. It provides a single virtual interface which is used by all containers running in a POD.

Q: What are the various ways to provide external world connectivity to K8?

A: By default POD should be able to reach the external world but for vice-versa we need to do some work. Following options are available to connect with POD from outer world.

• Nodeport (it will expose one port on each node to communicate with it)
• Load balancers (L4 layer of TCP/IP protocol)
• Ingress (L7 layer of TCP/IP Protocol)

One another method is kube-proxy which can be used to expose a service with only cluster IP on local system port.

$ kubectl proxy --port=8080 $ http://localhost:8080/api/v1/proxy/namespaces//services/:/

https://medium.com/google-cloud/kubernetes-nodeport-vs-loadbalancer-vs-ingress-when-should-i-use-what-922f010849e0

Q: What's the difference between nodeport and load balancer?

A: nodport relies on the IP address of your node. Also, you can use the node ports only from the range 30000–32767, on another hand load balancer will have it's own IP address. All the major cloud providers supports creating the LB for you if you specify LB type while creating the service. On baremetal based clusters, metallb is promising.

Q: When we need ingress instead of a LB?

A: For each service you have one LB. You can have single ingress for multiple services. This will allow you do both path based and subdomain based routing to backend services. You can do the SSL termination at ingress.

Q: How POD to POD communication works?

A: For POD to POD communication, it's always recommended to use the K8 service DNS instead of POD IP because PODs are ephemeral and their IPs can get change after the redeployment.

If the two PODs are running on a same host then physical interface will not come into the picture.

• Packet will leave POD1 virtual network interface and go to docker bridge (cbr0).
• Docker bridge will forward the packet to right POD2 which is running on same host.

If two PODs are running on a different host then physical interface of both host machines will come into the picture. Let's consider a scenario in which CNI is not used.

POD1 = 192.168.2.10/24 (node1, cbr0 192.168.2.1) POD2 = 192.168.3.10/24 (node2, cbr1 192.168.3.1)

• POD1 will send the traffic destined for POD2 to it's GW (cbr0) because both are in different subnet.
• GW doesn't know about 192.168.3.0/24 network hence it will forward the traffic to physical interface of node1.
• node1 will forward the traffic to it's own physical rourter/gateway.
• That physical router/GW should have the route for 192.168.3.0/24 network to route the traffic to node2.
• Once traffic reaches node2, it pass that traffic to POD2 through cbr1

If the Calico CNI it's responsible for adding the routes for cbr (docker bridge IP address) in all nodes.

Q: How POD to service communication works?

A: PODs are ephemeral their IP address can change hence to communicate with POD in reliable way service is used as a proxy or load balancer. A service is a type of kubernetes resource that causes a proxy to be configured to forward requests to a set of pods. The set of pods that will receive traffic is determined by the selector, which matches labels assigned to the pods when they were created. K8 provides an internal cluster DNS that resolves the service name.

Service is using different internal network than POD network. netfilter rules which are injected by kube-proxy are used to redirect the request actually destined for service IP to right POD.

Q: How does service knows about healthy endpoints?

A: kubelet running on worker node is responsible for detecting the unhealth endpoints, it passes that information to API server then eventually this information is passed to kube-proxy which wil adjust the netfilter rules accordingly.

I highly recommend reading the following series to get solid understanding about the K8 networking.

https://medium.com/google-cloud/understanding-kubernetes-networking-pods-7117dd28727

https://medium.com/google-cloud/understanding-kubernetes-networking-services-f0cb48e4cc82

https://medium.com/google-cloud/understanding-kubernetes-networking-ingress-1bc341c84078

Q: What are the various things can be done to increase the K8 security?

A: This is a huge topic, I am sharing some thoughts on it.

• By default, POD can communicate with any other POD, we can setup network policies to limit this communication between the PODs.
• RBAC (Role based access control) to narrow down the permissions.
• Use namespaces to establish security boundaries.
• Set the admission control policies to avoid running the priviledged containers.
• Turn on audit logging.

Q: How to monitor K8 cluster?

A: Prometheus is used for K8 monitoring. Prometheus ecosystem consists of multiple components.

• main Prometheus server which scrapes and stores time series data
• client libraries for instrumenting application code
• a push gateway for supporting short-lived jobs
• special-purpose exporters for services like HAProxy, StatsD, Graphite, etc.
• an alertmanager to handle alerts
• various support tools

Q: How to make prometheus HA?

A: You may run multiple instances of prometheus HA but grafana can use only of them as a datasource. You may put load balancer in front of multiple prometheus instances, use sticky sessions and failover if one of the prometheus instance dies. This make things complicated. Thanos is another project which solve these challenges.

Q: What are other challenges with prometheus?

A: Desipte of being very good at K8 monitoring, prometheus still have some issues:

• Prometheus HA support.
• No downsampling is available for collected metrics over the period of time.
• No support for object storage for long term metric retention.

All of these challenges are again overcome by Thanos.

Q: What's prometheus operator?

A: The mission of the Prometheus Operator is to make running Prometheus on top of Kubernetes as easy as possible, while preserving configurability as well as making the configuration Kubernetes native.

Q: How to get the central logs from POD?

A: This architecture depends upon application and many other factors. Following are the common logging patters

• Node level logging agent
• Streaming sidecar container
• Sidecar container with logging agent
• Export logs directly from the application

In our setup, filebeat and journalbeat are running as daemonset. Logs collected by these are dumped to kafka topic which are eventually dumped to ELK stack.

Same can be achieved using EFK stack and fluentd-bit.