NOTE: This is a temporary repo to allow users to preview the upcoming "Sysbox pods" feature. The feature is only present via this repo. It's not yet in the Sysbox or Sysbox Enterprise repos, but it's expected to land there in a few weeks (ideally after we've received sufficient feedback).
Once the feature is released, this current repo will cease to exist and the documentation and other artifacts will be moved to the Sysbox repo.
Having said this, we would appreciate if you refer to Sysbox repo for any GitHub star that you may be willing to throw at us -- always welcomed!
The Sysbox pods feature enables deployment of Kubernetes (K8s) pods using the Sysbox runtime.
In essence, Sysbox pods value-proposition is twofold: extended functionality and enhanced security.
With Sysbox, K8s can deploy strongly isolated (rootless) pods that can run not just microservices, but also workloads such as systemd, Docker, Podman (WIP) and even K8s, seamlessly.
Prior to Sysbox, running such pods required using privileged (insecure) containers and very complex pod setups and entrypoints. This is insecure (privileged pods allow users inside the pod to easily compromise the K8s host) and puts a lot of complexity on the K8s cluster admin & users.
You can now use Sysbox for improving the security of your K8s pods by replacing your existing privileged pods, and for deploying VM-like environments inside pods (quickly and efficiently, without actually using virtual machines).
With Sysbox this insecurity and complexity go away: the pods are strongly isolated, and Sysbox absorbs all the complexity of setting up the pod correctly to run the above-mentioned workloads.
Here is a video showing how to use K8s + Sysbox to deploy a rootless pod that runs systemd and Docker inside:
https://asciinema.org/a/401488?speed=1.5
- Kubernetes Version Requirements
- Kubernetes Worker Node Requirements
- Installation
- Sysbox Pods Deployment
- Kubernetes Manifests
- Sysbox Container Images
- Host Volume Mounts
- Uninstallation
- Troubleshooting
- Contact
- Thank You
Sysbox is only supported on Kubernetes v1.20.* at this time.
The reason for this is that Sysbox currently requires the presence of the CRI-O runtime v1.20, as the latter introduces support for "rootless pods" (i.e., pods that use the Linux user-ns). Since the version of CRI-O and K8s must match, the K8s version must also be v1.20.*.
Once you have a K8s v1.20 cluster up and running, you need to install CRI-O and Sysbox on one or more worker nodes.
Prior to installing Sysbox, ensure each node where you will install Sysbox meets the following requirement:
- The node's OS must be Ubuntu Focal or Bionic (with a 5.0+ kernel).
Installation is done via a couple of daemonsets which "drop" the CRI-O and Sysbox binaries onto the desired K8s nodes and perform all associated config:
- Install CRI-O on the desired K8s worker nodes:
kubectl label nodes <node-name> crio-install=yes
kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/rbac/crio-deploy-rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/daemonset/crio-deploy-k8s.yaml
- Install Sysbox on the same nodes:
kubectl label nodes <node-name> sysbox-install=yes
kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/rbac/sysbox-deploy-rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/daemonset/sysbox-deploy-k8s.yaml
kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/runtime-class/sysbox-runtimeclass.yaml
Then deploy pods with Sysbox see next section.
NOTES:
-
The installation will temporarily disrupt all pods on the K8s worker nodes where CRI-O and Sysbox were installed, for up to 1 minute, as they require the kubelet to restart.
-
Sysbox can be installed in all or some of the Kubernetes cluster worker nodes, according to your needs.
-
Installing Sysbox on a node does not imply all pods on the node are deployed with Sysbox. You can choose which pods use Sysbox via the pod's spec (see next section). Pods that don't use Sysbox continue to use the default low-level runtime (i.e., the OCI runc) or any other runtime you choose.
-
Pods deployed with Sysbox are managed via K8s just like any other pods; they can live side-by-side with non Sysbox pods and can communicate with them according to your K8s networking policy.
Deploying pods with Sysbox is easy: you only need a couple of things in the pod spec.
For example, here is a sample pod spec using the ubuntu-bionic-systemd-docker
image. It creates a rootless pod that runs systemd as init (pid 1) and comes
with Docker (daemon + CLI) inside:
apiVersion: v1
kind: Pod
metadata:
name: ubu-bio-systemd-docker
annotations:
io.kubernetes.cri-o.userns-mode: "auto:size=65536"
spec:
runtimeClassName: sysbox-runc
containers:
- name: ubu-bio-systemd-docker
image: registry.nestybox.com/nestybox/ubuntu-bionic-systemd-docker
command: ["/sbin/init"]
restartPolicy: Never
There are two key pieces of the pod's spec that tie it to Sysbox:
-
"runtimeClassName": Tells K8s to deploy the pod with Sysbox (rather than the default OCI runc). The pods will be scheduled only on the nodes that support Sysbox.
-
"io.kubernetes.cri-o.userns-mode": Tells CRI-O to launch this as a rootless pod (i.e., root user in the pod maps to an unprivileged user on the host) and to allocate a range of 65536 Linux user-namespace user and group IDs. This is required for Sysbox pods.
Also, for Sysbox pods you typically want to avoid sharing the process (pid)
namespace between containers in a pod. Thus, avoid setting
shareProcessNamespace: true
in the pod's spec, especially if the pod carries
systemd inside (as otherwise systemd won't be pid 1 in the pod and will fail).
Depending on the size of the pod's image, it may take several seconds for the pod to deploy on the node. Once the image is downloaded on a node however, deployment should be very quick (few seconds).
The K8s manifests used for setting up Sysbox can be found here.
The pod in the prior example uses the
ubuntu-bionic-systemd-docker
, but you can use any container
image you want. Sysbox places no requirements on the container image.
Nestybox has several images which you can find here:
https://hub.docker.com/u/nestybox
Those same images are in the Nestybox GitHub registry (ghcr.io/nestybox/<image-name>
).
We usually rely on registry.nestybox.com
as an image front-end so that docker
image pulls are forwarded to the most suitable repository without impacting our
users.
Some of those images carry systemd only, others carry systemd + Docker, other carry systemd + K8s (yes, you can run K8s inside rootless pods deployed by Sysbox).
To mount host volumes into a K8s pod deployed with Sysbox, the K8s worker node's
kernel must include the shiftfs
kernel module.
NOTE: shiftfs is currently only supported on Ubuntu kernels and it's installed automatically by the sysbox-deploy-k8 daemonset.
The need for shiftfs arises because such Sysbox pods are rootless, meaning that
the root user inside the pod maps to a non-root user on the host (e.g., pod user
ID 0 maps to host user ID 296608). Without shiftfs, host directories or files
which are typically owned by users IDs in the range 0->65535 will show up as
nobody:nogroup
inside the pod.
The shiftfs module solves this problem, as it allows Sysbox to "shift" user and group IDs inside the pod, such that files owned by users 0->65536 on the host also show up as owned by users 0->65536 inside the pod.
Once shiftfs is installed, Sysbox will detect this and use it when necessary. As a user you don't need to know anything about shiftfs; you just setup the pod with volumes as usual.
For example, the following spec creates a Sysbox pod with ubuntu-bionic + systemd +
docker and mounts host directory /root/somedir
into the pod's /mnt/host-dir
.
apiVersion: v1
kind: Pod
metadata:
name: ubu-bio-systemd-docker
annotations:
io.kubernetes.cri-o.userns-mode: "auto:size=65536"
spec:
runtimeClassName: sysbox-runc
containers:
- name: ubu-bio-systemd-docker
image: registry.nestybox.com/nestybox/ubuntu-bionic-systemd-docker
command: ["/sbin/init"]
volumeMounts:
- mountPath: /mnt/host-dir
name: host-vol
restartPolicy: Never
volumes:
- name: host-vol
hostPath:
path: /root/somedir
type: Directory
When this pod is deployed, Sysbox will automatically enable shiftfs on the pod's
/mnt/host-dir
. As a result that directory will show up with proper user-ID and
group-ID ownership inside the pod.
With shiftfs you can even share the same host directory across pods, even if the pods each get exclusive Linux user-namespace user-ID and group-ID mappings. Each pod will see the files with proper ownership inside the pod (e.g., owned by users 0->65536) inside the pod.
To uninstall Sysbox:
kubectl delete runtimeclass sysbox-runc
kubectl delete -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/daemonset/sysbox-deploy-k8s.yaml
kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/daemonset/sysbox-cleanup-k8s.yaml
kubectl delete -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/daemonset/sysbox-cleanup-k8s.yaml
kubectl delete -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/rbac/sysbox-deploy-rbac.yaml
To uninstall CRI-O:
kubectl delete -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/daemonset/crio-deploy-k8s.yaml
kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/daemonset/crio-cleanup-k8s.yaml
kubectl delete -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/daemonset/crio-cleanup-k8s.yaml
kubectl delete -f https://raw.githubusercontent.com/nestybox/sysbox-pods-preview/master/k8s-manifests/rbac/crio-deploy-rbac.yaml
NOTE: The uninstallation will temporarily disrupt all pods on the nodes where CRI-O and Sysbox were installed, for up to 1 minute, as they require the kubelet to restart.
Please contact us if you hit any problems (see contact info below).
The troubleshooting doc has some useful info too.
Slack: Nestybox Slack Workspace
Email: [email protected]
We are available from Monday-Friday, 9am-5pm Pacific Time.
We thank you very much for giving sysbox-pods an early try. We appreciate any feedback you can provide to us!