NOTE: This project is deprecated. All the work to enable confidential computing for VM TEEs is being pursued upstream in the Kata containers project.
Details on weekly meetings is available here.
Overview
Raksh (protect
) project is created with the aim to secure Kubernetes deployed
workload along with its specification (POD or Deployment YAML) by leveraging VM based Trusted Execution Environment (TEE).
Simply put Raksh makes it easier to use VM based TEE with containers in a Kubernetes cluster.
The initial development was primarily to leverage Protected Execution Facility (PEF) capability provided by Power (Power9) processors. However it should be pretty straight forward to adapt it for any other VM based TEEs, for example AMD Secure Encrypted Virtualisation (SEV).
PEF introduces a concept of secure VM (SVM). Anything running inside the VM is protected. More details on PEF is available here Ref: https://www.youtube.com/watch?v=pKh_mPPo9X4
Our goal was to leverage the PEF functionality with Kubernetes to provide a more secure option for our clients to deploy their containerized application. Since the protection and isolation is provided by the virtualization layer (KVM with support for SVM), the natural choice was to leverage Kata containers runtime as the basis. There are already examples of Kata integration with different virtualization technologies for improved security and isolation (firecracker etc).
We also had a need for strong coupling between the container image and the VM image and go a step further to protect the application specification (K8s POD/Deployment YAML) as well. More details on how we protect the application specification is available in the following doc
We leverage Kata containers. However we use a modified Kata agent with the following functionalities:
- Support for decrypting the application specification inside the VM
- Creating the containers based on the decrypted specification
Tying VM based TEE (eg. PEF) with a VM container runtime (Kata) provides us best of both worlds:
- Data in use protection
- Increased isolation
- No change in application code (like required by process-based TEE)
For more details please refer to the following links
Team
Harshal Patil
Manjunath Kumatagi
Nitesh Konkar
Pradipta Banerjee
Suhail Anjum
Prerequisites
- Fedora/RHEL/CentOS or Ubuntu host with KVM support
- Golang 1.12+
- Docker or Podman for building the containers
- Kubernetes with Kata Containers version 1.9 as runtime. Ensure Runtimeclass name is set as "kata-containers"
- CRIO or containerd
How to build
# Clone the repository
$ mkdir -p $GOPATH/src/github.com/ibm
$ cd $GOPATH/src/github.com/ibm
$ git clone https://github.com/ibm/raksh.git
$ git checkout -b 1.9.1-raksh origin/1.9.1-raksh
# For building binaries
$ cd raksh
$ make build-binary
# Install rakshctl binary
$ install -D build/_output/bin/rakshctl /usr/bin/
# Build and push docker images to external registry:
$ REGISTRY=docker.io && docker login $REGISTRY
$ REGISTRY=docker.io ORG=projectraksh make build-image
$ REGISTRY=docker.io ORG=projectraksh make push-image
$ REGISTRY=docker.io ORG=projectraksh IMAGE=sc-scratch make push-manifest
$ VERSION=latest REGISTRY=docker.io ORG=projectraksh IMAGE=sc-scratch make push-manifest
$ REGISTRY=docker.io ORG=projectraksh IMAGE=securecontainer-operator make push-manifest
$ VERSION=latest REGISTRY=docker.io ORG=projectraksh IMAGE=securecontainer-operator make push-manifest
Building the Kata (Raksh) agent and initrd
Follow the steps mentioned here to build the agent and initrd
Quick Start
Deploy the securecontainer-operator
Create image registry secret for your setup
$ kubectl create secret docker-registry regcred --docker-server=<image-registry> --docker-username=<user-name> --docker-password=<password> --docker-email=<email>
# Setup Service Account
$ kubectl create -f deploy/service_account.yaml
# Setup RBAC
$ kubectl create -f deploy/role.yaml
$ kubectl create -f deploy/role_binding.yaml
# Setup the CRD
$ kubectl create -f deploy/crds/securecontainers.k8s.io_securecontainerimageconfigs_crd.yaml
$ kubectl create -f deploy/crds/securecontainers.k8s.io_securecontainerimages_crd.yaml
$ kubectl create -f deploy/crds/securecontainers.k8s.io_securecontainers_crd.yaml
# Deploy the securecontainer-operator operator:
$ kubectl create -f deploy/operator.yaml
Note: Set the image-registry and change the image name for the securecontainer-operator in deploy/operator.yaml to override
Delete the securecontainer-operator
$ kubectl delete -f deploy/operator.yaml
$ kubectl delete -f deploy/crds/securecontainers.k8s.io_securecontainerimageconfigs_crd.yaml
$ kubectl delete -f deploy/crds/securecontainers.k8s.io_securecontainerimages_crd.yaml
$ kubectl delete -f deploy/crds/securecontainers.k8s.io_securecontainers_crd.yaml
$ kubectl delete -f deploy/role.yaml
$ kubectl delete -f deploy/role_binding.yaml
$ kubectl delete -f deploy/service_account.yaml
How to build the SecureContainerImage
$ rakshctl image create --image <SecureContainerImage_RESOURCE_NAME> --initrd <PATH_TO_KATA-INITRD_IMAGE> --vmlinux <PATH_TO_KATA-KERNEL> --symmKeyFile <PATH_TO_SYMM_KEY_FILE> --filename <PATH_TO_DEPLOYMENT_FILE> --scratch <IMAGE_REGISTRY/ORG/SCRATCH_IMAGE_NAME> --push <IMAGE_REGISTRY/ORG/IMAGE_NAME>
$ rakshctl image create --image nginx-securecontainerimage --initrd /usr/share/kata-containers/kata-containers-initrd.img --vmlinux /usr/share/kata-containers/vmlinuz.container --symmKeyFile /root/key_file --filename /securecontainers/sample/nginx.yaml --scratch docker.io/projectraksh/sc-scratch:latest --push docker.io/projectraksh/nginx-securecontainerimage:latest
Note: For Intel, use compressed kernel image- vmlinuz. For Power, use uncompressed kernel image- vmlinux.
To generate symmKey use:openssl rand -rand /dev/urandom 32 > symmKey
Create SecureContainerImageConfig custom resource
$ kubectl create -f imageconfig.yaml
Example:
$ cat imageconfig.yaml
apiVersion: securecontainers.k8s.io/v1alpha1
kind: SecureContainerImageConfig
metadata:
name: nginx-securecontainerimageconfig
spec:
imageDir: /var/lib/kubelet/secure-images
runtimeClassName: kata-containers
Create SecureContainerImage custom resource:
$ kubectl create -f securecontainerimage.yaml
Example:
$ cat securecontainerimage.yaml
apiVersion: securecontainers.k8s.io/v1alpha1
kind: SecureContainerImage
metadata:
name: nginx-securecontainerimage
spec:
vmImage: projectraksh/nginx-securecontainerimage:latest
imagePullSecrets:
- name: regcred
SecureContainerImageConfigRef:
name: nginx-securecontainerimageconfig
Note: Replace the vmImage with proper image build in step How to build the SecureContainerImage
Convert the user workload to secure workload and deploy
Example workload:
$ cat nginx.yaml
apiVersion: v1
kind: Pod
metadata:
labels:
app: nginx
namespace: default
name: nginx
spec:
containers:
- image: nginx:latest
imagePullPolicy: IfNotPresent
name: nginx
ports:
- containerPort: 80
protocol: TCP
The above command will generate a secure workload in nginx-sc.yaml file
$ cat nginx-sc.yaml
---
apiVersion: v1
data:
nginx: ZZ9P65ZlLoK/f9WZDFLq4j8F4piH9yAANmHg+CwC/5oFatk35E77p+DXYY9DX1HU3OqyZ++7+UHV/7XoWuUfgO0p0eVRT8nkF2VZRRSwWIgeBH7RdIYluPqt0TAQt4AFdOc3E1bFyKheWtT+l/JBJLmjSFTSKaZMaV+hb9Ev3WYVd7VDLoI5fhh9v2LE8bH1GfPFRKo=
kind: ConfigMap
metadata:
creationTimestamp: null
name: secure-configmap-nginx
namespace: default
---
apiVersion: securecontainers.k8s.io/v1alpha1
kind: SecureContainer
metadata:
creationTimestamp: null
name: secure-nginx
object:
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
app: nginx
name: nginx
namespace: default
spec:
containers:
- image: projectraksh/sc-scratch:latest
imagePullPolicy: IfNotPresent
name: nginx
ports:
- containerPort: 80
protocol: TCP
resources: {}
volumeMounts:
- mountPath: /etc/raksh
name: secure-volume-nginx
readOnly: true
imagePullSecrets:
- name: regcred
volumes:
- configMap:
items:
- key: nginx
path: raksh.properties
name: secure-configmap-nginx
name: secure-volume-nginx
status: {}
spec:
SecureContainerImageRef:
name: nginx-securecontainerimage
status: {}
Deploy the secure workload:
$ kubectl create -f nginx-sc.yaml