NOTE: This is a temporary repo to allow users to preview the upcoming "Sysbox pods" feature. The feature is only present via this repo. It's not yet in the Sysbox or Sysbox Enterprise repos, but it's expected to land there in a few weeks (ideally after we've received sufficient feedback).

Once the feature is released, this current repo will cease to exist and the documentation and other artifacts will be moved to the Sysbox repo.

Having said this, we would appreciate if you refer to Sysbox repo for any GitHub star that you may be willing to throw at us -- always welcomed!

The Sysbox pods feature enables deployment of Kubernetes (K8s) pods using the Sysbox runtime.

In essence, Sysbox pods value-proposition is twofold: extended functionality and enhanced security.

With Sysbox, K8s can deploy strongly isolated (rootless) pods that can run not just microservices, but also workloads such as systemd, Docker, Podman (WIP) and even K8s, seamlessly.

Prior to Sysbox, running such pods required using privileged (insecure) containers and very complex pod setups and entrypoints. This is insecure (privileged pods allow users inside the pod to easily compromise the K8s host) and puts a lot of complexity on the K8s cluster admin & users.

You can now use Sysbox for improving the security of your K8s pods by replacing your existing privileged pods, and for deploying VM-like environments inside pods (quickly and efficiently, without actually using virtual machines).

With Sysbox this insecurity and complexity go away: the pods are strongly isolated, and Sysbox absorbs all the complexity of setting up the pod correctly to run the above-mentioned workloads.

Here is a video showing how to use K8s + Sysbox to deploy a rootless pod that runs systemd and Docker inside:

https://asciinema.org/a/401488?speed=1.5

• Kubernetes Version Requirements
• Kubernetes Worker Node Requirements
• Installation
• Sysbox Pods Deployment
• Kubernetes Manifests
• Sysbox Container Images
• Host Volume Mounts
• Uninstallation
• Troubleshooting
• Contact
• Thank You

Sysbox is only supported on Kubernetes v1.20.* at this time.

The reason for this is that Sysbox currently requires the presence of the CRI-O runtime v1.20, as the latter introduces support for "rootless pods" (i.e., pods that use the Linux user-ns). Since the version of CRI-O and K8s must match, the K8s version must also be v1.20.*.

Once you have a K8s v1.20 cluster up and running, you need to install CRI-O and Sysbox on one or more worker nodes.

Prior to installing Sysbox, ensure each node where you will install Sysbox meets the following requirement:

• The node's OS must be Ubuntu Focal or Bionic (with a 5.0+ kernel).

Installation is done via a couple of daemonsets which "drop" the CRI-O and Sysbox binaries onto the desired K8s nodes and perform all associated config:

• Install CRI-O on the desired K8s worker nodes:

kubectl label nodes <node-name> crio-install=yes
kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/rbac/crio-deploy-rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/daemonset/crio-deploy-k8s.yaml

• Install Sysbox on the same nodes:

kubectl label nodes <node-name> sysbox-install=yes
kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/rbac/sysbox-deploy-rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/daemonset/sysbox-deploy-k8s.yaml
kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/runtime-class/sysbox-runtimeclass.yaml

Then deploy pods with Sysbox see next section.

NOTES:

• The installation will temporarily disrupt all pods on the K8s worker nodes where CRI-O and Sysbox were installed, for up to 1 minute, as they require the kubelet to restart.

• Sysbox can be installed in all or some of the Kubernetes cluster worker nodes, according to your needs.

• Installing Sysbox on a node does not imply all pods on the node are deployed with Sysbox. You can choose which pods use Sysbox via the pod's spec (see next section). Pods that don't use Sysbox continue to use the default low-level runtime (i.e., the OCI runc) or any other runtime you choose.

• Pods deployed with Sysbox are managed via K8s just like any other pods; they can live side-by-side with non Sysbox pods and can communicate with them according to your K8s networking policy.

Deploying pods with Sysbox is easy: you only need a couple of things in the pod spec.

For example, here is a sample pod spec using the ubuntu-bionic-systemd-docker image. It creates a rootless pod that runs systemd as init (pid 1) and comes with Docker (daemon + CLI) inside:

apiVersion: v1
kind: Pod
metadata:
  name: ubu-bio-systemd-docker
  annotations:
    io.kubernetes.cri-o.userns-mode: "auto:size=65536"
spec:
  runtimeClassName: sysbox-runc
  containers:
  - name: ubu-bio-systemd-docker
    image: registry.nestybox.com/nestybox/ubuntu-bionic-systemd-docker
    command: ["/sbin/init"]
  restartPolicy: Never

There are two key pieces of the pod's spec that tie it to Sysbox:

• "runtimeClassName": Tells K8s to deploy the pod with Sysbox (rather than the default OCI runc). The pods will be scheduled only on the nodes that support Sysbox.

• "io.kubernetes.cri-o.userns-mode": Tells CRI-O to launch this as a rootless pod (i.e., root user in the pod maps to an unprivileged user on the host) and to allocate a range of 65536 Linux user-namespace user and group IDs. This is required for Sysbox pods.

Also, for Sysbox pods you typically want to avoid sharing the process (pid) namespace between containers in a pod. Thus, avoid setting shareProcessNamespace: true in the pod's spec, especially if the pod carries systemd inside (as otherwise systemd won't be pid 1 in the pod and will fail).

Depending on the size of the pod's image, it may take several seconds for the pod to deploy on the node. Once the image is downloaded on a node however, deployment should be very quick (few seconds).

The K8s manifests used for setting up Sysbox can be found here.

The pod in the prior example uses the ubuntu-bionic-systemd-docker, but you can use any container image you want. Sysbox places no requirements on the container image.

Nestybox has several images which you can find here:

https://hub.docker.com/u/nestybox

Those same images are in the Nestybox GitHub registry (ghcr.io/nestybox/<image-name>).

We usually rely on registry.nestybox.com as an image front-end so that docker image pulls are forwarded to the most suitable repository without impacting our users.

Some of those images carry systemd only, others carry systemd + Docker, other carry systemd + K8s (yes, you can run K8s inside rootless pods deployed by Sysbox).

To mount host volumes into a K8s pod deployed with Sysbox, the K8s worker node's kernel must include the shiftfs kernel module.

NOTE: shiftfs is currently only supported on Ubuntu kernels and it's installed automatically by the sysbox-deploy-k8 daemonset.

The need for shiftfs arises because such Sysbox pods are rootless, meaning that the root user inside the pod maps to a non-root user on the host (e.g., pod user ID 0 maps to host user ID 296608). Without shiftfs, host directories or files which are typically owned by users IDs in the range 0->65535 will show up as nobody:nogroup inside the pod.

The shiftfs module solves this problem, as it allows Sysbox to "shift" user and group IDs inside the pod, such that files owned by users 0->65536 on the host also show up as owned by users 0->65536 inside the pod.

Once shiftfs is installed, Sysbox will detect this and use it when necessary. As a user you don't need to know anything about shiftfs; you just setup the pod with volumes as usual.

For example, the following spec creates a Sysbox pod with ubuntu-bionic + systemd + docker and mounts host directory /root/somedir into the pod's /mnt/host-dir.

apiVersion: v1
kind: Pod
metadata:
  name: ubu-bio-systemd-docker
  annotations:
    io.kubernetes.cri-o.userns-mode: "auto:size=65536"
spec:
  runtimeClassName: sysbox-runc
  containers:
  - name: ubu-bio-systemd-docker
    image: registry.nestybox.com/nestybox/ubuntu-bionic-systemd-docker
    command: ["/sbin/init"]
    volumeMounts:
      - mountPath: /mnt/host-dir
        name: host-vol
  restartPolicy: Never
  volumes:
  - name: host-vol
    hostPath:
      path: /root/somedir
      type: Directory

When this pod is deployed, Sysbox will automatically enable shiftfs on the pod's /mnt/host-dir. As a result that directory will show up with proper user-ID and group-ID ownership inside the pod.

With shiftfs you can even share the same host directory across pods, even if the pods each get exclusive Linux user-namespace user-ID and group-ID mappings. Each pod will see the files with proper ownership inside the pod (e.g., owned by users 0->65536) inside the pod.

To uninstall Sysbox:

kubectl delete runtimeclass sysbox-runc
kubectl delete -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/daemonset/sysbox-deploy-k8s.yaml
kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/daemonset/sysbox-cleanup-k8s.yaml
kubectl delete -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/daemonset/sysbox-cleanup-k8s.yaml
kubectl delete -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/rbac/sysbox-deploy-rbac.yaml

To uninstall CRI-O:

kubectl delete -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/daemonset/crio-deploy-k8s.yaml
kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/daemonset/crio-cleanup-k8s.yaml
kubectl delete -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/daemonset/crio-cleanup-k8s.yaml
kubectl delete -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/rbac/crio-deploy-rbac.yaml

NOTE: The uninstallation will temporarily disrupt all pods on the nodes where CRI-O and Sysbox were installed, for up to 1 minute, as they require the kubelet to restart.

Please contact us if you hit any problems (see contact info below).

The troubleshooting doc has some useful info too.

Slack: Nestybox Slack Workspace

Email: [email protected]

We are available from Monday-Friday, 9am-5pm Pacific Time.

We thank you very much for giving sysbox-pods an early try. We appreciate any feedback you can provide to us!