Git Product home page Git Product logo

function-kcl's Introduction

Crossplane Composition Functions using KCL

Go Report Card GoDoc License

Introduction

Crossplane KCL function allows developers to use KCL (a DSL) to write composite logic without the need for repeated packaging of crossplane functions, and we support package management and the KRM KCL specification, which allows for OCI/Git source and the reuse of KCL's module ecosystem.

Check out this blog to learn more. Here's a simple example:

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: example
spec:
  compositeTypeRef:
    apiVersion: example.crossplane.io/v1beta1
    kind: XR
  mode: Pipeline
  pipeline:
    - step: basic
      functionRef:
        name: function-kcl
      input:
        apiVersion: krm.kcl.dev/v1alpha1
        kind: KCLRun
        source: |
          # Read the XR
          oxr = option("params").oxr
          # Patch the XR with the status field
          dxr = {
              **oxr
              status.dummy = "cool-status"
          }
          # Construct a bucket
          bucket = {
              apiVersion = "s3.aws.upbound.io/v1beta1"
              kind = "Bucket"
              metadata.annotations: {
                  "krm.kcl.dev/composition-resource-name" = "bucket"
              }
              spec.forProvider.region = option("oxr").spec.region
          }
          # Return the bucket and patched XR
          items = [bucket, dxr]
    - step: automatically-detect-ready-composed-resources
      functionRef:
        name: function-auto-ready

Install the KCL Function to Cluster

cat <<EOF | kubectl apply -f -
apiVersion: pkg.crossplane.io/v1beta1
kind: Function
metadata:
  name: kcl-function
spec:
  package: xpkg.upbound.io/crossplane-contrib/function-kcl:latest
EOF

Using this Function

Source Support

To use a KCLRun as the function config, the KCL source must be specified in the source field. Additional parameters can be specified in the params field. The params field supports any complex data structure as long as it can be represented in YAML. Besides, the function can load KCL codes from inline source, OCI source, Git source and FileSystem source.

  • Inline source example
apiVersion: krm.kcl.dev/v1alpha1
kind: KCLRun
metadata:
  name: basic
spec:
  source: |
    {
        apiVersion = "s3.aws.upbound.io/v1beta1"
        kind = "Bucket"
        metadata.annotations: {
            "krm.kcl.dev/composition-resource-name" = "bucket"
        }
        spec.forProvider.region = option("oxr").spec.region
    }
  • OCI source example
apiVersion: krm.kcl.dev/v1alpha1
kind: KCLRun
metadata:
  name: basic
spec:
  source: oci://ghcr.io/kcl-lang/crossplane-xnetwork-kcl-function
  • Git source example
apiVersion: krm.kcl.dev/v1alpha1
kind: KCLRun
metadata:
  name: basic
spec:
  source: github.com/kcl-lang/modules/crossplane-xnetwork-kcl-function
  • FileSystem source example
apiVersion: krm.kcl.dev/v1alpha1
kind: KCLRun
metadata:
  name: basic
spec:
  source: ./path/to/kcl/file.k

Read the Function Requests and Values through the option Function

oxr

Custom Parameters

You can define your custom parameters in the params field and use option("params").custom_key to get the custom_value.

apiVersion: krm.kcl.dev/v1alpha1
kind: KCLRun
metadata:
  name: basic
spec:
  params:
     custom_key: custom_value
  source: oci://ghcr.io/kcl-lang/crossplane-xnetwork-kcl-function

Expect Output

A KRM YAML list means that each document must have an apiVersion, kind through the items field or a single YAML output.

  • Using the items field
apiVersion: krm.kcl.dev/v1alpha1
kind: KCLRun
metadata:
  name: basic
spec:
  source: |
    items = [{
        apiVersion: "ec2.aws.upbound.io/v1beta1"
        kind: "Instance"
        metadata.name = "instance1"
        spec.forProvider.region: "us-east-2"
    }, {
        apiVersion: "ec2.aws.upbound.io/v1beta1"
        kind: "Instance"
        metadata.name = "instance2"
        spec.forProvider.region: "us-east-2"
    }]
  • Single YAML output
apiVersion: krm.kcl.dev/v1alpha1
kind: KCLRun
metadata:
  name: basic
spec:
  source: |
    {
        apiVersion: "ec2.aws.upbound.io/v1beta1"
        kind: "Instance"
        metadata.name = "instance"
        spec.forProvider.ami: "ami-0d9858aa3c6322f73"
        spec.forProvider.instanceType: "t2.micro"
        spec.forProvider.region: "us-east-2"
    }

Target Support

The KCL function can target various types of objects:

  • Default: create new resources and set fields on the XR.
  • Resources: create new resources.
  • PatchDesired: set fields on existing DesiredComposed Resources.
  • PatchResources: set fields on existing resources fields. These resources will then be added to the desired resources map.
  • XR: set fields on the XR.

This is controlled by fields on the KCLRun

apiVersion: krm.kcl.dev/v1alpha1
kind: KCLRun
metadata:
  name: basic
spec:
  # default: Default
  target: Default | PatchDesired | PatchResources | Resources | XR
  source: |
    # Omit the source field
    ...

Extract Data from a Specific Composed Resource

To extract data from a specific composed resource by using the resource name, we can use the option("params").ocds variable, ocds is a mapping that its key is the resource name and its value is the observed composed resource like the example.

apiVersion: krm.kcl.dev/v1alpha1
kind: KCLRun
metadata:
  name: show-ocds
spec:
  source: |
    {
        metadata.name = "ocds"
        spec.ocds = option("params").ocds
        spec.user_kind = option("params").ocds["test-user"]?.Resource.Kind
        spec.user_metadata = option("params").ocds["test-user"]?.Resource.metadata
        spec.user_status = option("params").ocds["test-user"]?.Resource.status
    }

Composite Resource Connection Details

To return desired composite resource connection details, include a KCL config that produces the special CompositeConnectionDetails resource like the example:

apiVersion: krm.kcl.dev/v1alpha1
kind: KCLRun
metadata:
  name: basic
spec:
  source: |
    details = {
        apiVersion: "meta.krm.kcl.dev/v1alpha1"
        kind: "CompositeConnectionDetails"
        data: {
            "connection-secret-key": "connection-secret-value"
        }
    }
    # Omit other composite logics.
    # Input the details resource into the return resource list.
    items = [
        details
        # Omit other return resources.
    ]

Note: The value of the connection secret value must be base64 encoded. This is already the case if you are referencing a key from a managed resource's connectionDetails field. However, if you want to include a connection secret value from somewhere else, you will need to use the base64.encode function:

apiVersion: krm.kcl.dev/v1alpha1
kind: KCLRun
metadata:
  name: basic
spec:
  source: |
    import base64
    
    # Omit other logic
    ocds = option("params").ocds
    details = {
        apiVersion: "meta.krm.kcl.dev/v1alpha1"
        kind: "CompositeConnectionDetails"
        data: {
            "server-endpoint" = base64.encode(ocds["my-server"].Resource.status.atProvider.endpoint)
        }
    }

To mark a desired composed resource as ready, use the krm.kcl.dev/ready annotation:

apiVersion: krm.kcl.dev/v1alpha1
kind: KCLRun
metadata:
  name: basic
spec:
  source: |
    # Omit other logic
    user = {
        apiVersion: "iam.aws.upbound.io/v1beta1"
        kind: "User"
        metadata.name = "test-user"
        metadata.annotations: {
            "krm.kcl.dev/ready": "True"
        }
    }

Patching the XR status field

You can read the XR, patch it with the status field and return the new patched XR in the item result like this

apiVersion: krm.kcl.dev/v1alpha1
kind: KCLRun
metadata:
  name: basic
spec:
  source: |
    # Read the XR
    oxr = option("params").oxr
    # Patch the XR with the status field
    dxr = {
        **oxr
        status.dummy = "cool-status"
    }
    items = [dxr] # Omit other resources

Library

You can directly use KCL standard libraries such as regex.match, math.log.

Tutorial

See here to study more features such as conditions and loops in KCL.

Examples

More examples can be found here

Debugging the KCL Function in Cluster

Logs are emitted to the Function's pod logs. Look for the Function pod in crossplane-system.

Levels

Info   # default
Debug  # run with --debug flag

Developing

# Run code generation - see input/generate.go
$ go generate ./...

# Run tests - see fn_test.go
$ go test ./...

# Build the function's runtime image - see Dockerfile
$ docker build . --tag=kcllang/crossplane-kcl

# Build a function package - see package/crossplane.yaml
$ crossplane xpkg build -f package --embed-runtime-image=kcllang/crossplane-kcl

# Push a function package to the registry
$ crossplane --verbose xpkg push -f package/*.xpkg xpkg.upbound.io/crossplane-contrib/function-kcl:latest

function-kcl's People

Contributors

empath-nirvana avatar peefy avatar renovate[bot] avatar samirmarin avatar ytsarev avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

function-kcl's Issues

`"kcl" returned a fatal result: failed to run kcl function pipelines: Permission denied (os error 13)` with `v0.5.2` release

What happened?

Status:
  Conditions:
    Last Transition Time:  2024-04-16T16:50:32Z
    Message:               cannot compose resources: pipeline step "kcl" returned a fatal result: failed to run kcl function pipelines: Permission denied (os error 13)
    Reason:                ReconcileError
    Status:                False
    Type:                  Synced
  Warning  ComposeResources   7s (x5 over 17s)  defined/compositeresourcedefinition.apiextensions.crossplane.io  cannot compose resources: pipeline step "kcl" returned a fatal result: failed to run kcl function pipelines: Permission denied (os error 13)
k -n upbound-system logs -f deploy/crossplane-contrib-function-kcl-b6e7ced47cd4
2024/04/16 16:44:51 kclvm.go:41: [WARN] install kclvm failed: mkdir /go: permission denied
2024/04/16 16:44:51 kclvm.go:53: [WARN] install kclvm failed: open /go/init.lock: no such file or directory
2024/04/16 16:44:51 kclvm.go:58: [WARN] install kclvm failed: mkdir /go: permission denied
{"level":"info","ts":1713285921.3011003,"caller":"fn/fn.go:32","msg":"Running Function","tag":""}
{"level":"info","ts":1713285921.402543,"caller":"fn/fn.go:32","msg":"Running Function","tag":""}

How can we reproduce it?

Run basic kcl example e2e with XR and Composition (not crossplane beta render but the full XR instantiation )

What environment did it happen in?

Function version: xpkg.upbound.io/crossplane-contrib/function-kcl:v0.5.2

The function works as expected with the previous release of v0.5.1

Container uses 5GB of ram sitting mostly idle

Looking at my datadog metrics, it seems to be consuming 5GB of ram at all times (using, not requested), this is more than any of my very heavily used crossplane providers (most of which use less than 1GB of ram. Is there anything that can be done to reduce memory consumption? I'm only syncing one composition with this currently so it's very lightly used.

published version doesn't work

Hey ๐Ÿ‘‹

Thanks for the interesting function!

What happened?

Looks like there are no published function, which I can try? Not sure if that's docker image problem or my local machine setup. And I'm not sure how to list versions available.

So I'm trying to run examples/resources/loop with functions.yaml changed to:

apiVersion: pkg.crossplane.io/v1beta1
kind: Function
metadata:
  name: kcl-function
spec:
  package: kcllang/crossplane-kcl

which IIUC should run function in docker image, pulled from somewhere? ๐Ÿค”

but all I get is:

โžœ  loop git:(main) make run
crossplane beta render xr.yaml composition.yaml functions.yaml -r
crossplane: error: cannot render composite resource: cannot start Function "kcl-function": cannot start Docker container: Error response from daemon: unable to find user nonroot: no matching entries in passwd file

I'm running on arm64 macos with docker desktop

Thanks!

Function Pod High memory usage / Leak

What happened?

The function-kcl pod increases in memory usage over time, until it exhausts the node:
image

How can we reproduce it?

Not really sure, seems like it happens consistently

What environment did it happen in?

Function version: 0.8.0
Crossplane version: 1.15.2

How do you patch the XR status?

I've got a pipeline that creates resources, but I need to bubble status back up to the XR.

Is there an example of how to accomplish basically an ToCompositeFieldPath patch back to the XR using this function or with some other function?

The patch and transform function needs to know the name of the resource and that's not really deterministic.

arm64 architecture support

What problem are you facing?

We are trying to switch the architecture of some machines from amd64 to arm64 and found that function pod starts CrashLoopBackOff-ing as soon as there's a single XR/CompositeResource referencing it. We switched back to an amd node and it works properly again. Could it be that arm64 arch is unsupported?

Info:

  • Function version: v0.7.2
  • Currently using AWS EKS:v1.28.8-eks-ae9a62a
  • Nodes:
    • ARM64: AWS m6g.2xlarge instance
    • AMD64: AWS m7i-flex.2xlarge instance (Everything ok here)
crossplane beta trace -o wide <xrd>/<claim>
# NAME               RESOURCE   SYNCED   READY   STATUS
# <xrd>/<claim>                 True     False   Waiting: Claim is waiting for composite resource to become Ready
# โ””โ”€ <xrd>/<xr>                 False    -       ReconcileError: cannot compose resources: cannot run Composition pipeline step "kcl-render": cannot run Function "function-kcl": rpc error: code = Unavailable desc = last connection error: connection error: desc = "transport: Error while dialing: dial tcp x.x.x.x:9443: connect: connection refused"
KCL Function Pod - Describe
kubectl describe po -l pkg.crossplane.io/function=function-kcl
Name:             function-kcl-c44536185222-66c9f554fd-m8wht
Namespace:        crossplane
Priority:         0
Service Account:  function-kcl-c44536185222
Node:             ...
Start Time:       Fri, 10 May 2024 11:21:05 +0200
Labels:           pkg.crossplane.io/function=function-kcl
                  pkg.crossplane.io/revision=function-kcl-c44536185222
                  pod-template-hash=66c9f554fd
Annotations:      vpc.amazonaws.com/pod-ips: x.x.x.x
Status:           Running
IP:               x.x.x.x
IPs:
  IP:           x.x.x.x
Controlled By:  ReplicaSet/function-kcl-c44536185222-66c9f554fd
Containers:
  package-runtime:
    Container ID:   containerd://031093bf8ba71fa1ff3daa11ffd3df236d605794365adc4753851789f044d2d8
    Image:          xpkg.upbound.io/crossplane-contrib/function-kcl:v0.7.2
    Image ID:       xpkg.upbound.io/crossplane-contrib/function-kcl@sha256:c4453618522289c708d15cb12cd0f35ba0c612d1c6c4d7dcd095aa12ef97d4e5
    Ports:          8080/TCP, 9443/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Fri, 10 May 2024 11:23:16 +0200
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Fri, 10 May 2024 11:23:01 +0200
      Finished:     Fri, 10 May 2024 11:23:01 +0200
    Ready:          True
    Restart Count:  2
    Environment:
      TLS_SERVER_CERTS_DIR:             /tls/server
    Mounts:
      /tls/server from tls-server-certs (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4q9sf (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  tls-server-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  function-kcl-tls-server
    Optional:    false
  kube-api-access-4q9sf:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:         BestEffort
Node-Selectors:    nodegroup=crossplane
Tolerations:       dedicated=crossplane:NoSchedule
                   node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                   node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  2m32s                default-scheduler  Successfully assigned crossplane/function-kcl-c44536185222-66c9f554fd-m8wht to <node>
  Warning  BackOff    35s                  kubelet            Back-off restarting failed container package-runtime in pod function-kcl-c44536185222-66c9f554fd-m8wht_crossplane(eba0d4c1-b0b0-4c06-b829-0051b7c7590e)
  Normal   Pulled     21s (x3 over 2m31s)  kubelet            Container image "xpkg.upbound.io/crossplane-contrib/function-kcl:v0.7.2" already present on machine
  Normal   Created    21s (x3 over 2m31s)  kubelet            Created container package-runtime
  Normal   Started    21s (x3 over 2m31s)  kubelet            Started container package-runtime
KCL Function Pod - Logs with stacktrace
{"level":"info","ts":1715337555.3369577,"caller":"fn/fn.go:32","msg":"Running Function","tag":""}
panic: fork/exec /usr/local/bin/kclvm_cli: exec format error
goroutine 36 [running]:
 kcl-lang.io/kcl-go/pkg/runtime.(*Runtime).Start(0x400042c8c0)
      /go/pkg/mod/kcl-lang.io/[email protected]/pkg/runtime/runtime.go:54 +0x1a4
 kcl-lang.io/kcl-go/pkg/runtime.initRuntime(0x0?)
      /go/pkg/mod/kcl-lang.io/[email protected]/pkg/runtime/init.go:48 +0x14c
 kcl-lang.io/kcl-go/pkg/runtime.GetRuntime.func1()
      /go/pkg/mod/kcl-lang.io/[email protected]/pkg/runtime/init.go:26 +0x20
 sync.(*Once).doSlow(0x4000bd5530?, 0x0?)
      /usr/local/go/src/sync/once.go:74 +0x100
 sync.(*Once).Do(...)
      /usr/local/go/src/sync/once.go:65
 kcl-lang.io/kcl-go/pkg/runtime.GetRuntime()
      /go/pkg/mod/kcl-lang.io/[email protected]/pkg/runtime/init.go:26 +0x40
 kcl-lang.io/kcl-go/pkg/service.NewKclvmServiceClient()
      /go/pkg/mod/kcl-lang.io/[email protected]/pkg/service/client_kclvm_service.go:21 +0x1c
 kcl-lang.io/kcl-go/pkg/kcl.runWithHooks({0x400094ac28, 0x0, 0x0}, {0x64996d0, 0x1, 0x1}, {0x400094ae48, 0x1, 0x1})
      /go/pkg/mod/kcl-lang.io/[email protected]/pkg/kcl/api.go:477 +0xf4
 kcl-lang.io/kcl-go/pkg/kcl.run(...)
      /go/pkg/mod/kcl-lang.io/[email protected]/pkg/kcl/api.go:486
 kcl-lang.io/kcl-go/pkg/kcl.RunWithOpts({0x400094ae48?, 0x400094ae48?, 0x0?})
      /go/pkg/mod/kcl-lang.io/[email protected]/pkg/kcl/api.go:373 +0x50
 kcl-lang.io/kpm/pkg/api.RunWithOpt(0x4000867700)
      /go/pkg/mod/kcl-lang.io/[email protected]/pkg/api/kpm_run.go:89 +0x2d0
 kcl-lang.io/cli/pkg/options.(*RunOptions).Run(0x40000da000)
      /go/pkg/mod/kcl-lang.io/[email protected]/pkg/options/run.go:137 +0x19c
 kcl-lang.io/krm-kcl/pkg/edit.RunKCL({0x4000088788?, 0x1?}, {0x40006c7200?, 0x4000a348c0?}, 0x4000866940)
      /go/pkg/mod/kcl-lang.io/[email protected]/pkg/edit/bootstrap.go:52 +0x1ac
 kcl-lang.io/krm-kcl/pkg/edit.(*SimpleTransformer).Transform(0x400094b1d0, {0x4000088788?, 0x1?, 0x0?})
      /go/pkg/mod/kcl-lang.io/[email protected]/pkg/edit/transformer.go:46 +0xb0
 kcl-lang.io/krm-kcl/pkg/kio.Filter.Filter({0x400042c000}, {0x4000088788, 0x1, 0x1})
      /go/pkg/mod/kcl-lang.io/[email protected]/pkg/kio/filter.go:39 +0x160
 sigs.k8s.io/kustomize/kyaml/kio.Pipeline.ExecuteWithCallback({{0x400094b7b0, 0x1, 0x1}, {0x400094b7a0, 0x1, 0x1}, {0x400094b790, 0x1, 0x1}, 0x0}, ...)
      /go/pkg/mod/sigs.k8s.io/kustomize/[email protected]/kio/kio.go:137 +0x250
 sigs.k8s.io/kustomize/kyaml/kio.Pipeline.Execute({{0x400094b7b0, 0x1, 0x1}, {0x400094b7a0, 0x1, 0x1}, {0x400094b790, 0x1, 0x1}, 0x0})
      /go/pkg/mod/sigs.k8s.io/kustomize/[email protected]/kio/kio.go:104 +0x6c
 main.(*Function).RunFunction(0x40006f57a0, {0x1a59d60?, 0x400011b978?}, 0x40000dc3c0)
      /fn/fn.go:133 +0xe28
 github.com/crossplane/function-sdk-go/proto/v1beta1._FunctionRunnerService_RunFunction_Handler({0x1a59d60, 0x40006f57a0}, {0x255f9f0, 0x400013e1e0}, 0x40007da200, 0x0)
      /go/pkg/mod/github.com/crossplane/[email protected]/proto/v1beta1/run_function_grpc.pb.go:104 +0x1c0
 google.golang.org/grpc.(*Server).processUnaryRPC(0x4000636a00, {0x255f9f0, 0x400013e150}, {0x256ac80, 0x4000737c80}, 0x40005ee900, 0x400084aae0, 0x649c730, 0x0)
      /go/pkg/mod/google.golang.org/[email protected]/server.go:1369 +0xb58
 google.golang.org/grpc.(*Server).handleStream(0x4000636a00, {0x256ac80, 0x4000737c80}, 0x40005ee900)
      /go/pkg/mod/google.golang.org/[email protected]/server.go:1780 +0xb20
 google.golang.org/grpc.(*Server).serveStreams.func2.1()
      /go/pkg/mod/google.golang.org/[email protected]/server.go:1019 +0x8c
 created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 82
      /go/pkg/mod/google.golang.org/[email protected]/server.go:1030 +0x13c

How could this Function help solve your problem?

I don't know whether there's currently a way to make the KCL function work with arm nodes. If not, would it be possible to add support for arm64 architecture?


PS: Excuse me if I picked the wrong labels for the issue, not sure whether you would considered it a bug or it's just unsupported for the moment.

[Enhancement] Support annotation to set the plain http for the crossplane kcl function

What problem are you facing?

OCI Source is managed by function kcl and should be able to be set through the fields of KCLRun, in theory something like this perhaps.

    functionRef:
      name: kcl-function
    input:
      apiVersion: krm.kcl.dev/v1alpha1
      kind: KCLRun
      metadata:
        name: basic
        annotations:
          kcl-something/allow-insecure-source: True
      spec:
        target: Resources
        source: oci://localhost:7900/my-composition-kcl:0.0.1

How could this Function help solve your problem?

Kind not considered in Items list

What happened?

I tried to create two resources with different kinds, but the same name.

        source: |
          oxr = option("params").oxr
          envconf = option("params").ctx["apiextensions.crossplane.io/environment"]
          items = [
            {
              apiVersion = "eks.aws.upbound.io/v1beta1"
              kind = "Cluster"
              metadata.name = oxr.metadata.labels["crossplane.io/claim-name"]
              metadata.labels = { "aws.platform.ripple.com/observedcluster" = "true" }
              spec = { 
                forProvider.region = oxr.spec.parameters.region
                providerConfigRef.name = envconf.name
                managementPolicies = ["Observe"]
              }
            },
            {
                apiVersion = "apiextensions.crossplane.io/v1alpha1"
                kind = "EnvironmentConfig"
                metadata.name = oxr.metadata.labels["crossplane.io/claim-name"]
                metadata.labels = { "aws.platform.ripple.com/observedcluster" = "true" }
            }
          ]

Output:

apiVersion: apiextensions.crossplane.io/v1alpha1
kind: EnvironmentConfig
metadata:
  annotations:
    crossplane.io/composition-resource-name: cluster-josh-x1
  generateName: cluster-josh-x1-
  labels:
    aws.platform.ripple.com/observedcluster: "true"
    crossplane.io/composite: cluster-josh-x1
  name: cluster-josh-x1
  ownerReferences:
    - apiVersion: aws.platform.ripple.com/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: ObservedCluster
      name: cluster-josh-x1
      uid: ""
spec:
  forProvider:
    region: us-west-2
  managementPolicies:
    - Observe
  providerConfigRef:
    name: dev2-us-west-2

I only get whatever the final object was instead of a Cluster and an EnvironmentConfig

How can we reproduce it?

Attempt to create two different items with the same name.

Observe that the output is actually the merging of the two yaml objects instead of two distinct objects.

What environment did it happen in?

Function version: v0.7.0

Example KCL to pass resource data between created cloud resources

I'm writing KCL code to create Azure resources using the KCL crossplane function as part of a composition. I need to supply some of one resource's "atProvider" data to another resource's "forProvider" data.

In the past this would be achieved by using a patch with the patch and transform function but I would prefer to keep this encapsulated within the KCL code if possible.

Should I use the option("params").ocds some how to access the named resource and obtain its atProvider data? If so, can you provide an example of accessing a a resource this way. Please also let me know if this is not the preferred way of doing this.

Support non-SSL OCI registry for composition local development

What problem are you facing?

Attempting to use crossplane beta render on windows to render out a kcl package from a local docker registry:2 that has no ssl enabled.

I've tried using a relative and absolute file path to my module's main.k file but I'm always informed of "no such file or directory" so I have no idea of the required path format (for windows) or what the file path is relative to. That would be a separate issue to this.

If I use oci:// scheme, the kcl function expects to pull the package using https even though I have explicitly set "DefaultOciPlainHttp":true in my local kpm.json file.

I can push my package to the registry using kcl mod push oci://localhost:7900/my-composition-kcl no problem.

If in the KCLRun I use a source oci://host.docker.internal:7900/my-composition-kcl:0.0.1 I get the following error reported back from the crossplane beta render:

crossplane: error: cannot render composite resource: pipeline step "normal" returned a fatal result: failed to run kcl function pipelines: failed to select latest version from 'host.docker.internal:7900/my-composition-kcl:0.0.1'

Get "https://host.docker.internal:7900/v2/my-composition-kcl/tags/list": http: server gave HTTP response to HTTPS client

Could we allow the KCLRun to use the local oci registry settings, specifically DefaultOciPlainHttp to use http instead of https for download?

How could this Function help solve your problem?

Allow me to continue to use a non-SSL local registry

Extract data from a specific resource

What problem are you facing?

I'm not sure how to extract data from a specific composed resource by using kind and metadata.name. I'm guessing that it should be inside ocds but I'm not sure what the structure of it is. Is there an example I could use?

P.S. I apologize if this is already explained in one of the examples (and I missed it).

Write to Composite Resource

What problem are you facing?

I don't see the option to write back to the Composite resource. According to https://github.com/crossplane-contrib/function-kcl?tab=readme-ov-file#guides-for-developing-kcl option can be used only to read, not to write.

How could this Function help solve your problem?

Add a feature equivalent to the ToCompositeFieldPath Patch and Transform type. An example:

    - type: ToCompositeFieldPath
      fromFieldPath: spec.forProvider.manifest.spec.rules[0].host
      toFieldPath: status.host 

Ability to decouple `metadata.name` from `crossplane.io/composition-resource-name` and `crossplane.io/external-name` annotations

What problem are you facing?

Currently, in this function the metadata.name is a single entry point and identifier that is getting automatically propagated to crossplane.io/composition-resource-name and crossplane.io/external-name annotations.

That is convenient but frequently undesired behavior.

The situations where it is required for them to have different values:

  • I want to have the same metada.name across different resource kinds. Currently if I try to do it with function-kcl the resources with the same metada.name are getting overrides by the last one, because the same crossplane.io/composition-resource-name is getting set automatically.
    Example:
            providerConfigTypes = ["helm", "kubernetes"]
            providerConfigs = [{
              apiVersion = "{}.crossplane.io/v1alpha1".format(t)
              kind = "ProviderConfig"
              metadata.name = id
              metadata.annotations = {
                  "krm.kcl.dev/ready": "True"
              }
              spec.credentials = {
                  secretRef = {
                    name = "{}-ekscluster".format(uid)
                    namespace = connectionSecretNamespace
                    key = "kubeconfig"
                  }
                  source = "Secret"
                }
            } for t in providerConfigTypes]

Only last kubernetes ProviderConfig is getting rendered in this case

apiVersion: kubernetes.crossplane.io/v1alpha1
kind: ProviderConfig
metadata:
  annotations:
    crossplane.io/composition-resource-name: configuration-aws-eks

because the function is setting identical crossplane.io/composition-resource-name annotation

apiVersion: apigatewayv2.aws.upbound.io/v1beta1
kind: DomainName
metadata:
  annotations:
    crossplane.io/external-name: example-email.upbound-providers.io
  name: example
spec:
  forProvider:
...

In this case metada.name != crossplane.io/external-name and it is mandatory.
It is relatively frequent case when we need to override external-name.

How could this Function help solve your problem?

Respect when user sets custom values like

metadata.annotations = {
    "crossplane.io/composition-resource-name" = "custom-composition-name"
    "crossplane.io/external-name" = "custom-external-name"
}

Currently they are getting overridden by metadata.name value and it breaks the desired behavior

Function unable to run intermittently when multiple claims applied to cluster

What happened?

When applying multiple claims to a cluster using a single manifest file (using "---" separator between claims) Crossplane initially reports the following error when describing most of the composites:

"Message: cannot compose resources: cannot run Composition pipeline step "normal": cannot run Function "kcl-function": rpc error: code = DeadlineExceeded desc = context deadline exceeded"

Eventually all composites are reconciled correctly but it would appear like the function cannot handle concurrent execution perhaps? This has the effect of degrading the performance of reconciling many resources at once.

NOTE: Applying each claim separately and waiting for a result works without the error occurring.

How can we reproduce it?

Create a composite and apply several claims in one go. Observe each of the composites to see the error message.

What environment did it happen in?

Function version: v0.4.0
Crossplane: v1.15.0
Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.0

Add additional package metadata

What happened?

Now that this function-kcl has been published to the marketplace at https://marketplace.upbound.io/functions/crossplane-contrib/function-kcl, we should improve the metadata of this package to provide a more rich experience with more details.

How can we reproduce it?

Navigate to https://marketplace.upbound.io/functions/crossplane-contrib/function-kcl, and see:

  • Overview says "No overview provided."
  • Name is kcl-function, even though the repo is function-kcl.
  • Source code field is missing
  • maybe others too?
  • note that an icon is missing, but the marketplace will be adding that on the backend, so don't worry about that

We can compare this to another function like https://marketplace.upbound.io/functions/crossplane-contrib/function-auto-ready, which has more of these details defined. As an example, the source of its metadata can be found in https://github.com/crossplane-contrib/function-auto-ready/blob/main/package/crossplane.yaml.

What environment did it happen in?

Function version: v0.2.0

DatabaseInstance wrong value type

What happened?

While trying to create / update a DatabaseInstance.sql.gcp.upbound.io with database flags e.g.

items = [
            {
              apiVersion: "sql.gcp.upbound.io/v1beta1"
              kind: "DatabaseInstance"
              spec: {
                forProvider: {
                  project: "test-project"
                  settings: [
                    {
                      databaseFlags: [
                        {
                          name: "log_checkpoints"
                          value: "on"
                        }
                      ]
                    }
                  ]
                }
              }
            }
        ]

The update / create fails with:

defined/compositeresourcedefinition.apiextensions.crossplane.io  cannot compose resources: cannot apply composed resource "alir-test-crossplane-db-instance": failed to create typed patch object (/alir-test-crossplane-db-instance; sql.gcp.upbound.io/v1beta1, Kind=DatabaseInstance): errors:
  .spec.forProvider.settings[0].databaseFlags[0].value: expected string, got &value.valueUnstructured{Value:true}

This does not happen without function-kcl.
I saw a similar issue reported here maybe it is related: crossplane-contrib/provider-upjet-aws#1261

What environment did it happen in?

Function version: latest
Function revision: kcl-function-011644505e8f
Cloud: GCP
Crossplane version: 1.15.2

Meta type error when building the package

What happened?

When try to build the package, I get the following error:

crossplane: error: failed to build package: not exactly one package meta type

How can we reproduce it?

It was working. But suddenly just started to fail. The command is from README.md development chapter:

# Build a function package - see package/crossplane.yaml
$ crossplane xpkg build -f package --embed-runtime-image=kcllang/crossplane-kcl

What environment did it happen in?

This happens as a result of local development.

KCLInput vs KCLRun

In the Crossplane Marketplace one can find the KCLInput.template.fn.crossplane.io/v1beta1 specification (here).

However, all the provided examples here use KCLRun.krm.kcl.dev/v1alpha1.

Both CRDs have not the same spec (For instance, you cannot set target: Default in KCLInput), so I was wondering:

  • Is there any available info about the difference? I think that more clues about this could avoid a bit of confusion (Like mine :( )
  • Are they supposed to share the same spec? (but maybe they are different because the XP CRDs need to be in a XP group?)
    • In this case, spec diffs (Example: spec.target - KCLInput spec vs KCLRun doc) )

Thanks!

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

Detected dependencies

dockerfile
Dockerfile
  • golang 1
docker/amd64/Dockerfile
  • golang 1
docker/arm64/Dockerfile
  • golang 1
github-actions
.github/workflows/ci.yaml
  • actions/checkout v4
  • actions/setup-go v5
  • docker/setup-qemu-action v3
  • docker/setup-buildx-action v3
  • actions/checkout v4
  • docker/build-push-action v5
  • actions/upload-artifact v4
  • actions/checkout v4
  • actions/download-artifact v4
  • docker/login-action v3
  • ubuntu 22.04
  • ubuntu 22.04
  • ubuntu 22.04
gomod
go.mod
  • go 1.22.2
  • dario.cat/mergo v1.0.0
  • github.com/alecthomas/kong v0.9.0
  • github.com/crossplane/crossplane-runtime v1.15.1
  • github.com/crossplane/function-sdk-go v0.2.0
  • github.com/google/go-cmp v0.6.0
  • github.com/pkg/errors v0.9.1
  • google.golang.org/protobuf v1.34.1
  • gopkg.in/yaml.v2 v2.4.0
  • k8s.io/apimachinery v0.30.1
  • kcl-lang.io/krm-kcl v0.8.7-0.20240516090518-ba06813424c8@ba06813424c8
  • sigs.k8s.io/controller-tools v0.15.0
  • sigs.k8s.io/yaml v1.4.0
  • oras.land/oras-go v1.2.5
  • oras.land/oras-go/v2 v2.5.0

  • Check this box to trigger a request for Renovate to run again on this repository

[Enhancement] setting composition resource status

What happened?

I want to provide values to custom composition resource status fields, similar to how patch and transform do it using ToCompositeFieldPath. I am defining extra properties on my CompositeResourceDefinition's status property and attempting to set the property value using the below code:

_dxr = option("params").dxr
_dxr.status.myExtraProp = "sample"

However, the function fails to compile complaining of an UndefinedType:

    Message:               cannot compose resources: pipeline step "normal" returned a fatal result: failed to run kcl function pipelines: failed to compile the kcl package
EvaluationError
failed to update the dict. An iterable of key-value pairs was expected, but got UndefinedType. Check if the syntax for updating the dictionary with the attribute 'myExtraProp' is correct

How can we reproduce it?

Define a custom status property called myprop in xrd and attempt to set its value on dxr.status.myprop.

What environment did it happen in?

Function version: xpkg.upbound.io/crossplane-contrib/function-kcl:v0.3.4

How to modify composite resource connection details from kcl function?

I am developing a composite using KCL function code to create resources needed for an Azure workload identity, namely a UserAssignedIdentity, a FederatedIdentityCredential and a K8 Object for a service account. The PrincipalId of the UserAssignedIdentity is needed by other composites to facilitate the creation of RoleAssignment resources for their own access needs. So I need to surface the PrincipalId property from the composite somehow. NOTE: RoleAssignment only accepts a PrincipalId and has no selector option.

The problem is UserAssignedIdentity does not surface any information in connection details so writeConnectionSecretToRef produces a secret with no data, and of course this then provides no data to propagate up to the composite itself.

The status.atProvider data surfaced by the UserAssignedIdentity in the observed ocds dictionary does contain all the data needed of course but I'm not sure how to surface this to the composite. Theoretically I could write my own secret with the required data but was wondering if there was a way of still using a composite's connectiondetails so the secret is written by crossplane. Any ideas?

[Track] Crossplane KCL Provider

What problem are you facing?

I'm wondering if a KCL Crossplane Provider would be beneficial as well, similar to the Helm provider [https://marketplace.upbound.io/providers/crossplane-contrib/provider-helm/v0.18.1]. This would allow more direct access to Kubernetes resources such as Secrets which would be useful for pull secrets to access KCL modules that exists in OCI registries, Git, etc.

How could this Function help solve your problem?

I think the KCL Function is great and I believe if there was a KCL Provider to accompany it in order to leverage external KCL modules, it would for sure be a very powerful combination.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.