Problem

konductor terminal is causing graphical artifacting in cli

workaround

run command fish or bash to open a non-affected shell session

Need to add onramp documentation to support user choice of Pulumi state backend.

Inspired by issue #23

Discussed in discord #dev-kargo

So we can see what's happening in our homelab

I have installed Kasten using helm

helm install k10 kasten/k10 --namespace=kasten-io --create-namespace

There are many helm values available - https://docs.kasten.io/latest/

In order for this to be successful we need persistent storage available to our cluster.

Add a draw.io diagram depicting the platform infrastructure tech stack from metal all the way up.

I would like to request that the official client be podman or nerdctl to comply with a fully OCI compliant toolchain, I want to build containers nonroot with nonroot running processes and the cli is compatible with docker.

Enhance cilium configuration to replace MetalLB with Cilium L2 Advertisement & IPAM.

tasks

• helm chart
• helm values
• CiliumL2AnnouncementPolicy
• CiliumLoadBalancerIPPool
• temp test pod and service

Reference

• https://isovalent.com/blog/post/migrating-from-metallb-to-cilium

CCIO should publish/update/propagate community resources including Code of Conduct, Contributing.md, etc to facilitate community collaboration, psychological safety, diversity, and participation.

Add Kubevirt to Kargo IaC

Reference

• last known good configuration ccio/kargo3/kubevirt

Problem

Kargo has no auth/access control integration. Add via Open Unison.

https://openunison.github.io/documentation/ouctl/

Convert repository to protected 'main' branch

Pre-MVP development is currently proceeding with direct commits to main branch. This is poor git hygene and will not scale with project and community growth.

Recommend

1. activate protected main branch feature
2. setup dev / edge / main branches as [dev/test/prod] promotion channels
3. setup release publishing from main > vX.XX.XX semantic versioned branches for pinning deployments to stable releases
4. document n-1 support & maintenance + accept PRs for minor revision updates to version release branches.

problem

github actions kind ci/cd runner is failing

I would like to be able to run CI jobs as ephemeral containers with dagger in this project. https://dagger.io/ I believe it fits in neatly within the philosophy and purpose of Kargo.

Installation for local dev container, after make kind, user is prompted for pulumi access token.

Pulumi account and valid access token should probably be listed in the requirements so that it's setup before installation.

Add a LAN DNS service for the lab.

Reference

• https://github.com/usrbinkat/homelab-apps/blob/main/ops/coredns/values.yaml

Add Storage Providers

Requirements

1. A simple and reliable mvp storage provider that can be simply toggled on/off via bool in pulumi config and requires minimal/no user configuration.
2. A highly configurable and available storage provider supporting sane hdd, ssd, nvme storage class device matching and also serve the platform undercloud block devices as storage classes to tenant workloads and tenant kubernetes clusters as well.
3. A NFS storage provisioner for easy long term persistent storage served from appliance or provisioned NAS.

Selection

1. rancher hostpath provisioner
   • example pulumi iac usrbinkat/homelab-apps/ops/local-path-provisioner
2. rook ceph operator
   • last known good rook helm values rook ceph values
3. rook ceph cluster(s)
   • last known good rook helm values rook ceph values
4. nfs democratic-csi provisioner
   • example helm values usrbinkat/homelab-apps/ops/democratic-csi

https://github.com/usrbinkat/pulumi-ts-substacks-example?tab=readme-ov-file

In my environment I have access to an NFS shared storage device so with my environment I am looking at deploying via helm the CSI-Driver-NFS option.

The helm command I am using is below.

helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs --namespace kube-system --version v4.6.0 --set externalSnapshotter.enabled=true --set controller.runOnControlPlane=true --set controller.replicas=2

helm get values -n kube-system csi-driver-nfs

USER-SUPPLIED VALUES:
controller:
  replicas: 2
  runOnControlPlane: true
externalSnapshotter:
  enabled: true

A helm install is not enough we also need a StorageClass, Snapshotclass

Just documenting here so we can add to Kargo if requested and needed.

Problem:

Currently, platform development cycle velocity is capped at the speed of bare metal iteration which is significantly slower and more cumbersome than developing on virtual infrastructure.

Recommend:

Write the official virtualized platform developer workflow IaC and docs to deliver a low effort nested platform in platform development environment for Kargo contributors.

Need to rootcause & resolve breaking change from 1.14.5 to 1.15.1

https://github.com/ContainerCraft/Kargo/blob/main/src/kargo/cilium.py#L16

Add Multus via cluster network addons operator

Tasks

• add cnao operator
• cnao config
• add br0 resource
• fix cilium strict mode + multus linux bridge bug
• fix cilium cni missing bridge, macvlan, macvtap, etc cni binaries

Reference

• example helm values: ccio/helm/charts/cnao

Steps Required

• TODO: complete RFE

When installing cert manager the cluster-selfsigned-issuer ClusterIssuer fails to create because the cert-manager webhooks aren't up yet. Need to wait until the webhooks are ready before creating this object. Here's the logs:

kubernetes:cert-manager.io/v1:ClusterIssuer (cluster-selfsigned-issuer-root):
    error: resource "urn:pulumi:localkargo::kargo::kubernetes:cert-manager.io/v1:ClusterIssuer::cluster-selfsigned-issuer-root" was not successfully created by the Kubernetes API server : Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-b86abdb0-webhook.cert-manager.svc:443/mutate?timeout=10s": tls: failed to verify certificate: x509: certificate signed by unknown authority

  pulumi:pulumi:Stack (kargo-localkargo):
    2024-03-14 14:30:59,136 - INFO - Fetching URL: https://raw.githubusercontent.com/cilium/charts/master/index.yaml
    2024-03-14 14:30:59,775 - INFO - Fetching URL: https://raw.githubusercontent.com/cilium/charts/master/index.yaml
    2024-03-14 14:31:00,813 - INFO - Fetching URL: https://charts.jetstack.io/index.yaml

    error: update failed

As a primary feature of the Kargo platform, CAPI is a necessary MVP for building/destroying tenant k8s clusters as a core service of the platform.

Reference

• Last known good capi deployment artifacts ubk/homelab-apps/ops/capi

Problem

Trying to eliminate a built in namespace resource from a remote k8s.core.v1.ConfigFile manifest using transformations.

Logging indicates the condition is correctly detected but the namespace resource is still deployed.

Code permalink

    # Define the transformation to remove Namespace creation and ensure correct namespace for other resources
    # TODO: fix transformation to remove namespace creation (currently producing duplicate namespace resource)
    def remove_namespace_transform(args):
        if args['kind'] == "Namespace":
            pulumi.log.info(f"Skipping creation of duplicate Namespace: {args['metadata']['name']}")
            return None  # Skip the creation of this resource if it's a duplicate
        else:
            if 'metadata' in args:
                args['metadata']['namespace'] = ns_name
        pulumi.log.info(f"Transforming resource of namespace/kind: {ns_name}/{args['kind']}")
        return args

Bash

Kargo on  mvp/usrbinkat/refactor [!] via  usrbinkat@ci via 🐍 v3.10.12 
🐋 ❯ pulumi up --skip-preview --refresh=true
Updating (ci)

View in Browser (Ctrl+O): https://app.pulumi.com/usrbinkat/kargo/ci/updates/165

     Type                                                                  Name                                    Status              Info
     pulumi:pulumi:Stack                                                   kargo-ci                                                    4 messages
     ├─ pulumi:providers:kubernetes                                        k8sProvider                                                 
     ├─ kubernetes:core/v1:Namespace                                       cert-manager                                                
     │  └─ kubernetes:helm.sh/v3:Release                                   cert-manager                                                
     │     └─ kubernetes:cert-manager.io/v1:ClusterIssuer                  cluster-selfsigned-issuer-root                              [diff: ~metadata]
     │        └─ kubernetes:cert-manager.io/v1:Certificate                 cluster-selfsigned-issuer-ca                                
     │           └─ kubernetes:cert-manager.io/v1:ClusterIssuer            cluster-selfsigned-issuer                                   [diff: ~metadata]
     ├─ kubernetes:core/v1:Namespace                                       kubevirt                                                    
     │  └─ kubernetes:yaml:ConfigFile                                      kubevirt-operator                                           
     │     ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole          kubevirt/kubevirt-operator                                  [diff: ~metadata]
     │     ├─ kubernetes:core/v1:ServiceAccount                            kubevirt/kubevirt-operator                                  
     │     ├─ kubernetes:kubevirt.io/v1:KubeVirt                           kubevirt                                                    
     │     ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding   kubevirt/kubevirt-operator                                  [diff: ~metadata]
     │     ├─ kubernetes:apps/v1:Deployment                                kubevirt/virt-operator                                      
     │     ├─ kubernetes:core/v1:Namespace                                 kubevirt                                                    
     │     ├─ kubernetes:scheduling.k8s.io/v1:PriorityClass                kubevirt/kubevirt-cluster-critical                          [diff: ~metadata]
     │     ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole          kubevirt/kubevirt.io:operator                               [diff: ~metadata]
     │     ├─ kubernetes:rbac.authorization.k8s.io/v1:RoleBinding          kubevirt/kubevirt-operator-rolebinding                      
     │     ├─ kubernetes:rbac.authorization.k8s.io/v1:Role                 kubevirt/kubevirt-operator                                  
     │     └─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition  kubevirt/kubevirts.kubevirt.io                              [diff: ~metadata]
 ~   ├─ kubernetes:core/v1:Endpoints                                       kubernetes                              refresh (0.13s)     [diff: +apiVersion,kind,metadata]
     └─ custom:x:KubernetesApiEndpointIp                                   kubernetes-endpoint-service-address                         

Diagnostics:
  pulumi:pulumi:Stack (kargo-ci):
    Using helm release version: cert-manager/1.14.5
    Using KubeVirt version: kubevirt/1.2.0
    Using emulation for KubeVirt in developer mode
    Skipping creation of duplicate Namespace: kubevirt

Outputs:
    kubernetes-endpoint-service-address: "172.18.0.2"

Resources:
    21 unchanged

Duration: 10s

Problem

Makefile is a less integrated developer automation tool. Taskfile offers several more advanced environment variable handling features which is a primary motivation for adoption.

Proposal

Convert Makefile and Github Actions Workflow tasks to use Taskfile instead.

Requirements:

• 100% Feature Parity coverage in Taskfile for current Makefile features
• CONTRIBUTOR.md section explaining dev and test workflow
• CI .github/workflows/kind.yaml passes all tests
• Docstrings in code where taskfile documentation may be helpful for future maintainers

Resources:

• https://taskfile.dev
• https://taskfile.dev/faq

Problem:

MVP codebase is inconsistent and unpredictable having been written as an un-planned prototype codebase.

Recommendations:

Refactor, and redesign codebase and repository layout to properly implement maintainable and predictable structure for future growth and wider contribution.

the CDI is a shim to support distribution of virtual machine disk images via OCI using rudimentary Dockerfile COPY approach.

Need to adopt until UOR shim deprecates CDI for artifact distribution.

Reference

• last known good deployment ccio/kargo3/cdi

Feature Request

Add support for Kata Containers / Firecracker micro VMs

Reference

• https://github.com/siderolabs/extensions#container-runtimes
• https://github.com/kata-containers/kata-containers/blob/main/tools/packaging/kata-deploy/README.md

Problem

Currently CI testing Kargo deployment on Kind Kubernetes instead of Talos-in-Docker due to failure of Talos to successfully bring the talos cluster API online and bootstrap the kubernetes cluster.

Inlets sub represents ~50% of homelab Public IP total cost of ownership via it's implementation/subscription model when factored into monthly PCP IP + instance / k8s entry level pricing tiers.

Evaluate ngrok and tailscale as replacements and or alternative supported options.

Reference

• Pulumi Ngrok Deployment IaC