emil-jacero / kode-operator Goto Github PK

Separate test categories (storage, statefulset, security, etc) into multiple files to make it easier for contributions

The first attempt was based on Envoy proxy Basic Auth. The problem with that implementation is that it doesn't redirect the user to a login page if they don't have a Authorization header set.

The new implementation should make an attempt at utilizing both envoy's own basic-auth feature and Ext_authz to work.
It should use a custom made ext_authz plugin to redirect the user to a login page (basic auth) if they are missing the Authorization header.
It should then use the built in Envoy basic-auth feature to authenticate the user.

If this method fails, write a ext_authz extension that manages the whole flow (authentication and login).

Tests

• Should configure Envoy with basic-auth if EnvoyProxyConfig.AuthConfig.AuthType=basic

A user should be able to add their SSH keys as a secret or chose to generate a pair.
This could be used to SSH to other systems from within the container.

Documentation

Add warning that they should not use their regular SSH keys. They should rather create new ones first and upload as secret or generate with the controller.

Take inspiration from the fluxv2 project for how to implement.

The test suite is done, write tests that test all the basic functionality that has been implemented

Tests

• Should create and delete Kode resource.
• Should update (restart) the Kode instance if changes are detected (User, Password, ExistingSecret, Home, Workspace, UserConfig, Privileged, InitPlugins).
• Should NOT update running Kode instances if the related KodeTemplate is changed.
• Should increase PVC size when Kode.Storage.Resources is increased (size decrease is not allowed).
• Should set the PVC ReclaimPolicy to Retain if Kode.Storage.KeepVolume is set to true.
• Should retain the persistent volume if Kode.Storage.KeepVolume is set to true.
• Should delete the PVC if Kode.Storage.KeepVolume is set to false.
• Should use an existing persistent volume claim if Kode.Storage.ExistingVolumeClaim is defined.
• Should run all InitPlugins (InitContainers) after the built in Envoy initContainers.
• Should run KodeTemplate.InitPlugins before Kode.InitPlugins.
• Should create an Envoy sidecar container when KodeTemplate references a EnvoyProxyConfig
• Should set HTTPFilters and Clusters settings to the Envoy sidecar container if specified.

If a user defindes Kode.username and Kode.password it should create a secret and put the values into that secret for use in the statefulSet and initContainers.
If a user defines Kode.existingSecret it should just use that secret.

A user authenticates using OIDC, the JWT token is used to identify and allow them access to the Kode instance.
This feature will allow Envoy to pass the JWT token to the container so that it can be used for the user downstream.

If this does not work, look into simplifying the reauthentication process inside the container so that they can use their identity downstream.

A user might want to keep their data even if they delete their Kode instance. Use a finalizer on the PVC to control deletion.

The same logic can be used for the Recycle feature #23

To allow Kode to create an EntryPoint with a subdomain we have two options.

1. Create DNS wildcard and have fun! This is relatively simple but prevents the administrator from using different gateways or ingresses if needed because the DNS record is pointing to a single IP address.
2. Use an automated DNS operator like external-dns to manage DNS records. This will ensure that an administrator can deploy multiple entrypoints resources with different ingresses or gatways and IP addresses.

When omitting EnvoyProxyReference in "production" it allows it. However when i attempt to omit it for an integration test it fails.

Error

  [FAILED] Expected success, but got an error:
      <*errors.StatusError | 0x14000407540>: 
      KodeClusterTemplate.kode.jacero.io "test-kodetemplate-without-envoy" is invalid: spec.envoyProxyRef.kind: Unsupported value: "": supported values: "EnvoyProxyConfig", "EnvoyProxyClusterConfig"
      {
          ErrStatus: {
              TypeMeta: {Kind: "", APIVersion: ""},
              ListMeta: {
                  SelfLink: "",
                  ResourceVersion: "",
                  Continue: "",
                  RemainingItemCount: nil,
              },
              Status: "Failure",
              Message: "KodeClusterTemplate.kode.jacero.io \"test-kodetemplate-without-envoy\" is invalid: spec.envoyProxyRef.kind: Unsupported value: \"\": supported values: \"EnvoyProxyConfig\", \"EnvoyProxyClusterConfig\"",
              Reason: "Invalid",
              Details: {
                  Name: "test-kodetemplate-without-envoy",
                  Group: "kode.jacero.io",
                  Kind: "KodeClusterTemplate",
                  UID: "",
                  Causes: [
                      {
                          Type: "FieldValueNotSupported",
                          Message: "Unsupported value: \"\": supported values: \"EnvoyProxyConfig\", \"EnvoyProxyClusterConfig\"",
                          Field: "spec.envoyProxyRef.kind",
                      },
                  ],
                  RetryAfterSeconds: 0,
              },
              Code: 422,
          },
      }
  In [BeforeAll] at: /Users/emil/devel/emil/kode-operator/test/integration/controller/kode_integration_test.go:105 @ 07/08/24 09:57:40.873

Config

envoyProxyRef is omitted.

---
apiVersion: kode.jacero.io/v1alpha1
kind: KodeTemplate
metadata:
  labels:
    app.kubernetes.io/name: kode-operator
    app.kubernetes.io/managed-by: kustomize
  name: kodetemplate-codeserver-sample
  namespace: default
spec:
  type: code-server
  image: linuxserver/code-server:latest
  tz: "Europe/Stockholm"
  puid: 1000
  pgid: 1000
  defaultHome: /config
  defaultWorkspace: workspace

Look into using both virtink and kubevirt.

This feature should detect if the user is inactive and "recycle" the container. In practice this would require the container to report on the activity in some way.

For code-server, this is done through the /healthz endpoint.
For other containers this would be different. For example in the webtop scenario KasmVNC has inactive_user_session_timeout but that might not expose an endpoint that can be used.

Might be a good idea to look into a way to detect the inactivity via the Envoy Proxy in some way.

• Bug
• Feature request
• Question

Fix and improve the status updater. The status manager should be generalized so that it can be used by other processes.

The kode specific status must be compatible

For developers

• How do i contribute?
• How do i setup my environment? ;)

The first attempt was based on Envoy proxy Basic Auth. The problem with that implementation is that it doesn't redirect the user to a login page if they don't have a Authorization header set.

The new implementation should be a sidecar container that Envoy proxy redirects all requests to using Ext Authz. The sidecar container will handle the basic auth authentication and authorization.

Just like code-server this should be supported.

Webtop is developed and maintained by linuxserver.io. It is a linux desktop in a container.

It will be an ephemeral desktop running in kubernetes. It will be an alternative to a more long lived desktop solution like virtual machines and citrix.

A requested feature is to have a CRD that simplifies the publishing of Kode instances.

Each Kode resource should belong to a single user, and only that user. This could be accomplished in many different ways.

Subdomains

Assign a unique subdomain for each user's instance. For example, user1.kode.example.com, user2.kode.example.com.
A sub-path for each Kode instance is required.

Advantages:

• Clear isolation at the DNS level.
• Easy to implement with Ingress API or Gateway API.
• Scales well with a large number of users.
• Can use wildcard SSL certificates to secure all subdomains.

Disadvantages:

• Requires DNS configuration to support wildcard subdomains.
• May have more complex DNS and SSL management.

URL Paths

Use URL paths to differentiate users, such as kode.example.com/user1, kode.example.com/user2.
A sub-path for each Kode instance is required.

Advantages:

• Easier to manage a single domain.
• Simpler DNS and SSL certificate management.
• Also scales well with large number of users.

Disadvantages:

• Requires careful configuration to ensure no path conflicts.
• May require more complex routing rules in the Ingress API or Gateway API.
• Less isolation compared to subdomains.

Unique Ports

Assign a unique port for each user's instance, such as kode.example.com:3001, kode.example.com:3002.
A sub-path for each Kode instance is required.

Advantages:

• Simple and straightforward implementation.
• No need for complex DNS configuration.

Disadvantages:

• Not user-friendly; users have to remember port numbers.
• Limited number of ports available, not scalable for a large number of users.
• Might require firewall adjustments.

Summary & Reflection

There are more options like utilizing a service-mesh within Kubernetes to route traffic, but that would require this project to integrate with each service-mesh out there and that would take to much time.

The Unique port option is simple but a bit ugly. I foresee some use-cases where this might be wanted. For example in single cluster deployments, and where there are no easy way to configure DNS and access to certificates.

The Subdomain and URL Paths options are both the standard elsewhere in the industry and are relatively easy to implement.
I might also add support for External-DNS as an option to just using wildcard DNS records.

Even when all the dependencies are up to date the settings is not accepted.

The project

• Summary: What does it offer?
• Why: Problem statement and Why does it exist?
• Features: A list of killer features!
• Link to Project site and documentation

For users

• How do i use it?
• How do i submit features requests?
• How do i submit bug reports?

For developers

• How do i contribute?
• How do i setup my environment? ;)