fidelity / kraan Goto Github PK

View Code? Open in Web Editor NEW

90.0 90.0 15.0 14.77 MB

Kraan is a Kubernetes Controller that manages the deployment of HelmReleases to a cluster.

Home Page: https://fidelity.github.io/kraan/

License: Apache License 2.0

Dockerfile 0.45% Makefile 4.45% Go 85.14% Shell 9.55% Mustache 0.40%

addons helm-controller k8s

kraan's People

Contributors

Stargazers

Watchers

Forkers

sbernheim nab-gha arghya88 mukteshkrmishra aarjay4001 samhays michael12312 kishoregv ysnr-capco rxacevedo syllogy lvchongen ekmixon luis-pinto-fanduel fidelity-contributions

kraan's Issues

Improve test coverage for repos package

linter configuration suppresses a number of linters at present.

We need to revisit this at some stage and decide which linters to suppress globally and fix code to pass other linters or add comments as to why certain lint errors and warning are ok to ignore. See https://github.com/fidelity/kraan/blob/master/golangci-lint.yml#L185

Metrics reported for deleted layers

Description
Metrics are produced for non-existent layers.

To Reproduce
Steps to reproduce the behavior:

Deploy a layer
Check that metrics are produced correctly.
Delete the layer
Metrics (in particular reconcile_condition) are still produced for the deleted layer

Expected behavior
Once I delete a layer I expect no metrics to be produced from that point onwards.

Logs

root@node1:/# curl http://10.123.123.156:8080/metrics | grep reconcile_condition
# HELP reconcile_condition The current condition status of a GitOps Toolkit resource reconciliation.
# TYPE reconcile_condition gauge
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer1",status="Deleted",type="Deployed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer1",status="Deleted",type="Failed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer1",status="False",type="Deployed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer1",status="False",type="Failed"} 1
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer1",status="True",type="Deployed"} 1
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer1",status="True",type="Failed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer1",status="Unknown",type="Deployed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer1",status="Unknown",type="Failed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer2",status="Deleted",type="Deployed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer2",status="Deleted",type="Failed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer2",status="False",type="Deployed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer2",status="False",type="Failed"} 1
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer2",status="True",type="Deployed"} 1
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer2",status="True",type="Failed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer2",status="Unknown",type="Deployed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer2",status="Unknown",type="Failed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer3",status="Deleted",type="Deployed"} 1
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer3",status="Deleted",type="Failed"} 1
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer3",status="False",type="Deployed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer3",status="False",type="Failed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer3",status="True",type="Deployed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer3",status="True",type="Failed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer3",status="Unknown",type="Deployed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer3",status="Unknown",type="Failed"} 0

adi@WSL$ k get addonslayers.kraan.io
NAME                  VERSION             SOURCE                      PATH                            STATUS     MESSAGE
layer1                0.4.1               layer1-git                  ./add-ons/helm-releases         Deployed   AddonsLayer version 0.4.1 is Deployed, All HelmReleases deployed
layer2                1.0.2               layer2-git                  ./add-ons/helm-releases         Deployed   AddonsLayer version 1.0.2 is Deployed, All HelmReleases deployed

Additional context
As you can see above, layer 3 has been deleted (long time before querying the metrics endpoint) but the reconcile_condition metric is still produced for it. I'm trying to create an alert for reconcile_condition{type="Deployed", status="True"} < 0, but it keeps triggering for layers that don't exist anymore.

Another observation here would be that the name of the layer is in the namespace field instead of the name field. I'd appreciate if that could be fixed as well. 😄

Watch source controller

Implement functionality to watch for changes to the addons sub directory in the repository and retrieve the files if a change is made.

The Source Controller will store a copy of the addons repository based on a defined tag in a tar file held in its local storage.
Kraan will obtain Source Controller to retrieve that tag and then obtain the tar file and unpack it.

Consider removing the K8s Client from reconciler

see #11 (comment)

apply dry run not working

I noticed that apply dry run is not working, when a helm release already exists on the server it is returning the helm release state as it is on the cluster rather than in the yaml file.

cat /tmp/kraan-5ww3fu/addons-config/./testdata/addons/bootstrap/microservice1.yaml

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: microservice-1
  namespace: bootstrap
  annotations:
    fluxcd.io/automated: "false"
spec:
  releaseName: microservice-test-1
  test:
    enable: true
    ignoreFailures: false
    timeout: 300
  chart:
    git: https://github.com/fidelity/kraan
    path: testdata/charts/podinfo
    ref: issue-36
  values:
    preHookBackoffLimit: 1
    preHookActiveDeadlineSeconds: 60
    preHookRestartPolicy: Never
    preHookDelaySeconds: 10
    preHookSucceed: "true"
    testHookBackoffLimit: 1
    testHookActiveDeadlineSeconds: 60
    testHookRestartPolicy: Never
    testHookDelaySeconds: 10
    testHookSucceed: "true"
    podinfo:
      service:
        enabled: true
        type: ClusterIP
      replicaCount: 1
      message: -Microservice Test 1

dry run apply show chart ref master rather than issue-9

kubectl apply -R -f /tmp/kraan-5ww3fu/addons-config/./testdata/addons/bootstrap/microservice1.yaml --dry-run -o json
W0908 09:28:26.148859   84008 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
{
    "apiVersion": "helm.fluxcd.io/v1",
    "kind": "HelmRelease",
    "metadata": {
        "annotations": {
            "fluxcd.io/automated": "false",
            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"helm.fluxcd.io/v1\",\"kind\":\"HelmRelease\",\"metadata\":{\"annotations\":{\"fluxcd.io/automated\":\"false\"},\"name\":\"microservice-1\",\"namespace\":\"bootstrap\"},\"spec\":{\"chart\":{\"git\":\"https://github.com/fidelity/kraan\",\"path\":\"testdata/charts/podinfo\",\"ref\":\"master\"},\"releaseName\":\"microservice-test-1\",\"test\":{\"enable\":true,\"ignoreFailures\":false,\"timeout\":300},\"values\":{\"podinfo\":{\"message\":\"-Microservice Test 1\",\"replicaCount\":1,\"service\":{\"enabled\":true,\"type\":\"ClusterIP\"}},\"preHookActiveDeadlineSeconds\":60,\"preHookBackoffLimit\":1,\"preHookDelaySeconds\":10,\"preHookRestartPolicy\":\"Never\",\"preHookSucceed\":\"true\",\"testHookActiveDeadlineSeconds\":60,\"testHookBackoffLimit\":1,\"testHookDelaySeconds\":10,\"testHookRestartPolicy\":\"Never\",\"testHookSucceed\":\"true\"}}}\n"
        },
        "creationTimestamp": "2020-09-08T08:06:52Z",
        "generation": 1,
        "labels": {
            "kraan/owner": "bootstrap"
        },
        "name": "microservice-1",
        "namespace": "bootstrap",
        "ownerReferences": [
            {
                "apiVersion": "kraan.io/v1alpha1",
                "blockOwnerDeletion": true,
                "controller": true,
                "kind": "AddonsLayer",
                "name": "bootstrap",
                "uid": "a06a88c4-61b9-436f-a14b-9d79c35b081b"
            }
        ],
        "resourceVersion": "856421",
        "selfLink": "/apis/helm.fluxcd.io/v1/namespaces/bootstrap/helmreleases/microservice-1",
        "uid": "cd663723-b3ac-47d0-a174-4e6cb1eee009"
    },
    "spec": {
        "chart": {
            "git": "https://github.com/fidelity/kraan",
            "path": "testdata/charts/podinfo",
            "ref": "master"
        },
        "releaseName": "microservice-test-1",
        "rollback": {},
        "test": {
            "enable": true,
            "ignoreFailures": false,
            "timeout": 300
        },
        "values": {
            "podinfo": {
                "message": "-Microservice Test 1",
                "replicaCount": 1,
                "service": {
                    "enabled": true,
                    "type": "ClusterIP"
                }
            },
            "preHookActiveDeadlineSeconds": 60,
            "preHookBackoffLimit": 1,
            "preHookDelaySeconds": 10,
            "preHookRestartPolicy": "Never",
            "preHookSucceed": "true",
            "testHookActiveDeadlineSeconds": 60,
            "testHookBackoffLimit": 1,
            "testHookDelaySeconds": 10,
            "testHookRestartPolicy": "Never",
            "testHookSucceed": "true"
        }
    },
    "status": {
        "conditions": [
            {
                "lastTransitionTime": "2020-09-08T08:06:52Z",
                "lastUpdateTime": "2020-09-08T08:27:25Z",
                "message": "Chart fetch failed for Helm release 'microservice-test-1' in 'bootstrap'.",
                "reason": "ChartFetchFailed",
                "status": "False",
                "type": "ChartFetched"
            },
            {
                "lastTransitionTime": "2020-09-08T08:06:52Z",
                "lastUpdateTime": "2020-09-08T08:27:25Z",
                "message": "Chart fetch failed for Helm release 'microservice-test-1' in 'bootstrap'.",
                "reason": "ChartFetchFailed",
                "status": "False",
                "type": "Released"
            }
        ],
        "observedGeneration": 1,
        "phase": "ChartFetchFailed"
    }
}

showing the helm release currently applied, has master as ref

kubectl -n bootstrap get helmreleases.helm.fluxcd.io microservice-1 -o json
{
    "apiVersion": "helm.fluxcd.io/v1",
    "kind": "HelmRelease",
    "metadata": {
        "annotations": {
            "fluxcd.io/automated": "false",
            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"helm.fluxcd.io/v1\",\"kind\":\"HelmRelease\",\"metadata\":{\"annotations\":{\"fluxcd.io/automated\":\"false\"},\"name\":\"microservice-1\",\"namespace\":\"bootstrap\"},\"spec\":{\"chart\":{\"git\":\"https://github.com/fidelity/kraan\",\"path\":\"testdata/charts/podinfo\",\"ref\":\"master\"},\"releaseName\":\"microservice-test-1\",\"test\":{\"enable\":true,\"ignoreFailures\":false,\"timeout\":300},\"values\":{\"podinfo\":{\"message\":\"-Microservice Test 1\",\"replicaCount\":1,\"service\":{\"enabled\":true,\"type\":\"ClusterIP\"}},\"preHookActiveDeadlineSeconds\":60,\"preHookBackoffLimit\":1,\"preHookDelaySeconds\":10,\"preHookRestartPolicy\":\"Never\",\"preHookSucceed\":\"true\",\"testHookActiveDeadlineSeconds\":60,\"testHookBackoffLimit\":1,\"testHookDelaySeconds\":10,\"testHookRestartPolicy\":\"Never\",\"testHookSucceed\":\"true\"}}}\n"
        },
        "creationTimestamp": "2020-09-08T08:06:52Z",
        "generation": 1,
        "labels": {
            "kraan/owner": "bootstrap"
        },
        "name": "microservice-1",
        "namespace": "bootstrap",
        "ownerReferences": [
            {
                "apiVersion": "kraan.io/v1alpha1",
                "blockOwnerDeletion": true,
                "controller": true,
                "kind": "AddonsLayer",
                "name": "bootstrap",
                "uid": "a06a88c4-61b9-436f-a14b-9d79c35b081b"
            }
        ],
        "resourceVersion": "856421",
        "selfLink": "/apis/helm.fluxcd.io/v1/namespaces/bootstrap/helmreleases/microservice-1",
        "uid": "cd663723-b3ac-47d0-a174-4e6cb1eee009"
    },
    "spec": {
        "chart": {
            "git": "https://github.com/fidelity/kraan",
            "path": "testdata/charts/podinfo",
            "ref": "master"
        },
        "releaseName": "microservice-test-1",
        "rollback": {},
        "test": {
            "enable": true,
            "ignoreFailures": false,
            "timeout": 300
        },
        "values": {
            "podinfo": {
                "message": "-Microservice Test 1",
                "replicaCount": 1,
                "service": {
                    "enabled": true,
                    "type": "ClusterIP"
                }
            },
            "preHookActiveDeadlineSeconds": 60,
            "preHookBackoffLimit": 1,
            "preHookDelaySeconds": 10,
            "preHookRestartPolicy": "Never",
            "preHookSucceed": "true",
            "testHookActiveDeadlineSeconds": 60,
            "testHookBackoffLimit": 1,
            "testHookDelaySeconds": 10,
            "testHookRestartPolicy": "Never",
            "testHookSucceed": "true"
        }
    },
    "status": {
        "conditions": [
            {
                "lastTransitionTime": "2020-09-08T08:06:52Z",
                "lastUpdateTime": "2020-09-08T08:27:25Z",
                "message": "Chart fetch failed for Helm release 'microservice-test-1' in 'bootstrap'.",
                "reason": "ChartFetchFailed",
                "status": "False",
                "type": "ChartFetched"
            },
            {
                "lastTransitionTime": "2020-09-08T08:06:52Z",
                "lastUpdateTime": "2020-09-08T08:27:25Z",
                "message": "Chart fetch failed for Helm release 'microservice-test-1' in 'bootstrap'.",
                "reason": "ChartFetchFailed",
                "status": "False",
                "type": "Released"
            }
        ],
        "observedGeneration": 1,
        "phase": "ChartFetchFailed"
    }
}

Delete the helm release from Cluster

kubectl -n bootstrap delete helmreleases.helm.fluxcd.io microservice-1 
helmrelease.helm.fluxcd.io "microservice-1" deleted
kubectl -n bootstrap get helmreleases.helm.fluxcd.io microservice-1 -o json
Error from server (NotFound): helmreleases.helm.fluxcd.io "microservice-1" not found

dry run again, this time ok

kubectl apply -R -f /tmp/kraan-5ww3fu/addons-config/./testdata/addons/bootstrap/microservice1.yaml --dry-run -o json
W0908 09:29:29.261976   85555 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
{
    "apiVersion": "helm.fluxcd.io/v1",
    "kind": "HelmRelease",
    "metadata": {
        "annotations": {
            "fluxcd.io/automated": "false",
            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"helm.fluxcd.io/v1\",\"kind\":\"HelmRelease\",\"metadata\":{\"annotations\":{\"fluxcd.io/automated\":\"false\"},\"name\":\"microservice-1\",\"namespace\":\"bootstrap\"},\"spec\":{\"chart\":{\"git\":\"https://github.com/fidelity/kraan\",\"path\":\"testdata/charts/podinfo\",\"ref\":\"issue-36\"},\"releaseName\":\"microservice-test-1\",\"test\":{\"enable\":true,\"ignoreFailures\":false,\"timeout\":300},\"values\":{\"podinfo\":{\"message\":\"-Microservice Test 1\",\"replicaCount\":1,\"service\":{\"enabled\":true,\"type\":\"ClusterIP\"}},\"preHookActiveDeadlineSeconds\":60,\"preHookBackoffLimit\":1,\"preHookDelaySeconds\":10,\"preHookRestartPolicy\":\"Never\",\"preHookSucceed\":\"true\",\"testHookActiveDeadlineSeconds\":60,\"testHookBackoffLimit\":1,\"testHookDelaySeconds\":10,\"testHookRestartPolicy\":\"Never\",\"testHookSucceed\":\"true\"}}}\n"
        },
        "name": "microservice-1",
        "namespace": "bootstrap"
    },
    "spec": {
        "chart": {
            "git": "https://github.com/fidelity/kraan",
            "path": "testdata/charts/podinfo",
            "ref": "issue-36"
        },
        "releaseName": "microservice-test-1",
        "test": {
            "enable": true,
            "ignoreFailures": false,
            "timeout": 300
        },
        "values": {
            "podinfo": {
                "message": "-Microservice Test 1",
                "replicaCount": 1,
                "service": {
                    "enabled": true,
                    "type": "ClusterIP"
                }
            },
            "preHookActiveDeadlineSeconds": 60,
            "preHookBackoffLimit": 1,
            "preHookDelaySeconds": 10,
            "preHookRestartPolicy": "Never",
            "preHookSucceed": "true",
            "testHookActiveDeadlineSeconds": 60,
            "testHookBackoffLimit": 1,
            "testHookDelaySeconds": 10,
            "testHookRestartPolicy": "Never",
            "testHookSucceed": "true"
        }
    }
}

Improve source controller sync processing

At present the mapping function that processes the gitrepository.source.toolkit.fluxcd.io custom resources gets the current repository contents from the source controller, identifies the addon layers that use it and creates a link to the addon layer's data directory then requeues the addon layers for processing.

This works but has some issues. The process of getting the repository data from the source controller and unpacking it into the local filesystem involves deleting the existing copy of the data to avoid deleted files being retained. This creates a window when the symbolic link used by the addon layers to access this data points to a non existent directory. This causes the addon layer to be marked as failed. The subsequent reprocessing of the addon layer will find the data and continue but this generates errors and status updates that are alarming for the user.

The repos package that is used to process the gitrepository.source.toolkit.fluxcd.io custom resources contains a Repos object that keeps track of repos. This contains entries for each revision of a repo. A better solution would be for the Repos to contain Repo objects that relate to gitrepository.source.toolkit.fluxcd.io custom resources and introduce a 'RepoRevision' object that relates to a particular git commit in the repo.

I also propose that the mapping function should just manage the Repos, adding, deleting Repo objects to the map in Repos and requeuing any addon layers that use that Repo.

We should add code to the addon layers reconciliation processing to look for the Repo they use in Repos and then look in the Repo for a RepoRevision matching the Repo's current revision. If this is not present it will be created and data retrieved from the source controller. Then the symbolic link for the addon layers data can be updated to point at the new revision or adjusted if need be to point at pre-existing directory, which will occur if multiple addon layers are using the same gitrepository.source.toolkit.fluxcd.io custom resource and one has created and updated a new revision.

Restructure reconcile processing

After discussion with @richardcase I propose restructuring the reconcile processing to remove the status driven case statement and replace this with a single stream of processing that performs prerequisite checks and then executes required processing.

reconcile processing unit tests

Add unit tests for controllers package functions

hard code kraan-controller in template but build kraan/kraan-controller-prerelease:v0.1.20-dev

We hard code kraan-controller in

kraan/chart/templates/kraan/deployment.yaml

Line 38 in 711059b

 image: {{ .Values.kraan.kraanController.image.repository }}/kraan-controller:{{ .Values.kraan.kraanController.image.tag | default .Chart.AppVersion }} 

but in ci we build kraan/kraan-controller-prerelease:v0.1.20-dev, seems we should have a value to override this so we can use that build?

Am I missing something?

@saada thoughts

AddonsLayer: Conditions should be unique

Is your feature request related to a problem? Please describe.
I would like to use kubectl wait addonslayers --for=condition=Deployed --all to wait for layers to complete. This is not possible today due to duplication of conditions per version. So if v1 has Deployed=false and v2 has Deployed=true, the wait command still fails as it likely chooses the first condition it finds.

Describe the solution you'd like
If the intention was to use Conditions as a log, we should instead use the object's Events instead. Conditions are not intended to be logs. They should be unique per condition type.

Describe alternatives you've considered
I tried using the AddonsLayers.status.state but that is not accessible via kubectl wait.

Implement application metrics

Add metrics to track AddonsLayer, GitRepository and HelmRelease processing

Deploy kraan-controller image to public repository

We need to deploy the kraan-controller image to a public repository so users can deploy it to clusters.
github packages has been suggested but so far I've been unable to get an image deployed to it.

Process non HelmRelease objects

Add owner information and process kubernetes objects

Test setup script for private repositories

The setup script has been tested using public image repositories but the use of image pull secrets and a private repository needs to be tested.

Tidy logging

Tidying up logging levels used and investigate how to address logging of structures as json.
If we leave it to logr implementation to log a structure it doesn't seem to log nested items.
If we convert to json first then the json is embdeded in json

Update reason and message fields with details of apply progress

At present when a layer is being applied the reason field of the status condition is set to AddonsLayer is being applied.
It would be informative to populate the reason and message fields with more details, i.e. the name of the helm release we are waiting for and the most recent events. For example, given a helm release deployment failing due to test failures we could get events for helm releases

kubectl get events -n mgmt -o json | jq -r '.items[] | select(."involvedObject"."kind" == "HelmRelease") '
{
  "apiVersion": "v1",
  "count": 14,
  "eventTime": null,
  "firstTimestamp": "2020-09-09T09:14:08Z",
  "involvedObject": {
    "apiVersion": "helm.fluxcd.io/v1",
    "kind": "HelmRelease",
    "name": "microservice-1",
    "namespace": "mgmt",
    "resourceVersion": "1050516",
    "uid": "21f2c5bc-f211-4514-8b55-c1decffa965b"
  },
  "kind": "Event",
  "lastTimestamp": "2020-09-09T09:38:09Z",
  "message": "(combined from similar events): synchronization of release 'microservice-test-1' in namespace 'mgmt' failed: test failed: pod microservice-test-1-podinfo-grpc-test-12rqo failed",
  "metadata": {
    "creationTimestamp": "2020-09-09T09:14:08Z",
    "name": "microservice-1.163312b203b2a122",
    "namespace": "mgmt",
    "resourceVersion": "1057305",
    "selfLink": "/api/v1/namespaces/mgmt/events/microservice-1.163312b203b2a122",
    "uid": "398c8c58-42df-43e5-82d0-9fd2192c1592"
  },
  "reason": "FailedReleaseSync",
  "reportingComponent": "",
  "reportingInstance": "",
  "source": {
    "component": "helm-operator"
  },
  "type": "Warning"
}

it would be helpful if the reason could be updated with something like

waiting for  'microservice-test-1' in namespace 'mgmt', FailedReleaseSync

and the message with

 "(combined from similar events): synchronization of release 'microservice-test-1' in namespace 'mgmt' failed: test failed: pod microservice-test-1-podinfo-grpc-test-12rqo failed"

Problem with AddonsLayer Processing Order

Seeing an issue where the 'apps' layer applies before the layers it depends on.

I delete HelmReleases and update AddonsLayers as follows:

sed -i s/0.1.13/0.1.14/g $REPOS_PATH/addons-config/testdata/addons/addons.yaml;kubectl delete helmreleases.helm.fluxcd.io -A --all;kubectl apply -f $REPOS_PATH/addons-config/testdata/addons/addons.yaml;watch kubectl get al

This causes all layers to process. Layer 'apps' that depends on 'base' and 'mgmt' is immediately processed and starts applying.
From debugging output it appear that the old 0.1.13 version of the custom resource is being processed and seeing the existing version 0.1.13 of 'base' and 'mgmt' layers are deployed so it proceeds to apply.

This reconcile is in response to the helm release being deleted.

2020-09-09T12:42:37.010+0100	DEBUG	hr sync	HR update, requeuing	{"Name": "apps", "HR": "apps/microservice-1", "at": "controllers.indexHelmReleaseByOwner() - addons_controller.go(267)"}
....
2020-09-09T12:42:37.510+0100	DEBUG	controllers.AddonsLayer	apply required	{"Name": "apps", "Spec": {"source":{"name":"addons-config","namespace":"gitops-system","path":"./testdata/addons/apps"},"prereqs":{"k8sVersion":"v1.16","dependsOn":["[email protected]","[email protected]"]},"interval":"0s","version":"0.1.13"}, "Status": {"conditions":[{"type":"Deployed","status":"False","version":"0.1.08","lastTransitionTime":"2020-09-09T10:37:22Z","reason":"AddonsLayer is Deployed"},{"type":"Deployed","status":"False","version":"0.1.09","lastTransitionTime":"2020-09-09T10:44:05Z","reason":"AddonsLayer version 0.1.08 is Deployed"},{"type":"Deployed","status":"False","version":"0.1.09","lastTransitionTime":"2020-09-09T10:44:09Z","reason":"AddonsLayer version 0.1.09 is Deployed"},{"type":"Deployed","status":"False","version":"0.1.11","lastTransitionTime":"2020-09-09T10:54:53Z","reason":"AddonsLayer version 0.1.11 is Deployed"},{"type":"Applying","status":"False","version":"0.1.11","lastTransitionTime":"2020-09-09T11:01:36Z","reason":"AddonsLayer is being applied","message":"The applying status means the manager is either applying the yaml files or waiting for the HelmReleases to successfully deploy."},{"type":"Deployed","status":"False","version":"0.1.12","lastTransitionTime":"2020-09-09T11:01:48Z","reason":"AddonsLayer version 0.1.12 is Deployed"},{"type":"Applying","status":"False","version":"0.1.12","lastTransitionTime":"2020-09-09T11:17:31Z","reason":"AddonsLayer is being applied","message":"The applying status means the manager is either applying the yaml files or waiting for the HelmReleases to successfully deploy."},{"type":"Deployed","status":"False","version":"0.1.12","lastTransitionTime":"2020-09-09T11:17:45Z","reason":"AddonsLayer version 0.1.12 is Deployed"},{"type":"Applying","status":"False","version":"0.1.12","lastTransitionTime":"2020-09-09T11:19:11Z","reason":"AddonsLayer is being applied","message":"The applying status means the manager is either applying the yaml files or waiting for the HelmReleases to successfully deploy."},{"type":"Deployed","status":"True","version":"0.1.13","lastTransitionTime":"2020-09-09T11:19:19Z","reason":"AddonsLayer version 0.1.13 is Deployed"}],"state":"Deployed","version":"0.1.13"}, "at": "controllers.(*AddonsLayerReconciler).processApply() - addons_controller.go(110)"}

Need to do more debugging to understand how to address this scenario.

publish godocs

What is the best way to generate and publish godocs?
Makefile currently implements godocdown https://github.com/robertkrimen/godocdown
However this utility is not maintained and has some issues

Use helm repository in chart references rather than git repository

At present the helm releases in the testdata uses helm charts in a git repository, we should support the use of helm charts from helm repositories

Add metrics

Add application metrics

Use scratch or distroless image

Add processing done flag

See comments on #54

kraan-controller fails to detect Helm Release deployment failures

When running tests using new script to run kraan-controller on workstation against local files (https://github.com/fidelity/kraan/pull/35/files#diff-1818c11d00240758e549449086e232c6) Helm release in bootstrap layer are failing, see #36 but kraan-controller marks the layer deployed.

kubectl get addonslayers.kraan.io bootstrap -o json| jq -r '.spec,.status'
{
  "hold": false,
  "interval": "1m",
  "prereqs": {
    "k8sVersion": "v1.16"
  },
  "source": {
    "name": "addons-config",
    "namespace": "gitops-system",
    "path": "./testdata/addons/bootstrap"
  },
  "version": "0.1.01"
}
{
  "conditions": [
    {
      "lastTransitionTime": "2020-09-02T17:01:04Z",
      "message": "The k8sVersion status means the manager has detected that the AddonsLayer needs a higher version of the Kubernetes API than the current version running on the cluster.",
      "reason": "AddonsLayer is waiting for the required K8sVersion",
      "status": "True",
      "type": "K8sVersion",
      "version": "0.1.01"
    },
    {
      "lastTransitionTime": "2020-09-03T08:16:40Z",
      "message": "source directory (/repos/addons-config/./testdata/addons/bootstrap) not found for AddonsLayer bootstrap",
      "reason": "AddonsLayer processsing has failed",
      "status": "True",
      "type": "Failed",
      "version": "0.1.01"
    },
    {
      "lastTransitionTime": "2020-09-03T09:56:14Z",
      "message": "The applying status means the manager is either applying the yaml files or waiting for the HelmReleases to successfully deploy.",
      "reason": "AddonsLayer is being applied",
      "status": "True",
      "type": "Applying",
      "version": "0.1.01"
    },
    {
      "lastTransitionTime": "2020-09-03T10:10:43Z",
      "message": "error from kubectl while parsing source directory (/tmp/kraan-LQ1JMB/addons-config/./testdata/addons/bootstrap) for AddonsLayer bootstrap: error executing kubectl command '/usr/local/bin/kubectl apply -R -f /tmp/kraan-LQ1JMB/addons-config/./testdata/addons/bootstrap -o json' : exit status 1",
      "reason": "AddonsLayer processsing has failed",
      "status": "True",
      "type": "Failed",
      "version": "0.1.01"
    },
    {
      "lastTransitionTime": "2020-09-03T10:20:16Z",
      "message": "The applying status means the manager is either applying the yaml files or waiting for the HelmReleases to successfully deploy.",
      "reason": "AddonsLayer is being applied",
      "status": "True",
      "type": "Applying",
      "version": "0.1.01"
    },
    {
      "lastTransitionTime": "2020-09-03T10:20:17Z",
      "reason": "AddonsLayer is Deployed",
      "status": "True",
      "type": "Deployed",
      "version": "0.1.01"
    },
    {
      "lastTransitionTime": "2020-09-03T10:40:45Z",
      "message": "The pruning status means the manager is pruning objects removed from this layer",
      "reason": "AddonsLayer is being pruned",
      "status": "True",
      "type": "Pruning",
      "version": "0.1.01"
    },
    {
      "lastTransitionTime": "2020-09-03T10:40:48Z",
      "message": "The applying status means the manager is either applying the yaml files or waiting for the HelmReleases to successfully deploy.",
      "reason": "AddonsLayer is being applied",
      "status": "True",
      "type": "Applying",
      "version": "0.1.01"
    },
    {
      "lastTransitionTime": "2020-09-03T10:40:52Z",
      "reason": "AddonsLayer is Deployed",
      "status": "True",
      "type": "Deployed",
      "version": "0.1.01"
    }
  ],
  "state": "Deployed",
  "version": "0.1.01"
}

Controller Manager list not returning items

The controller manager is meant to monitor helm releases and retain a cache of them that we can use mgr.client.List() to retrieve.
This is not working, see #37.

Need to workout how to make this work.

Applier tests require kubectl executable

The apply package tests are not mocking the kubectl package fully thus require a kubectl executable to be present

Support for layer integration test job

Is your feature request related to a problem? Please describe.
Need to be able perform some tests after all resources in a layer are deployed to verify the addons in that layer are working correctly together.

Describe the solution you'd like
The ability to configure a helmrelease that runs after other helmreleases in the layer have been successfully deployed

Describe alternatives you've considered
None.

Additional context
The integration test job needs to be run even if there have been no changes to helmreleases in the layer.

Package gitops toolkit as Helm Release

The source controller component of the gitops toolkit is required by Kraan. To support users who want everything to be deployed using Helm Releases we need to package this as a Helm Release.

Add deployment of new chart to ci

As discussed with @saada we should add a step to the ci to generate and deploy a new chart version if the version in https://github.com/fidelity/kraan/blob/master/chart/Chart.yaml#L6 has been updated.

Use controller manager sync rather than re-queue

@richardcase suggested using the controller manager sync capability rather than re-queuing custom resources.
This will mean the interval field in the spec is not required and the sync period will be set at controller runtime.
This change will mean every instance of the AddonsLayer custom resources will be processed by the controller every 'sync period'

Use Helm-Controller

deploy and use helm-controller instead of helm-operator

Helm Operator not deploying helm releases

From helm operator logs

ts=2020-09-03T10:20:30.907312991Z caller=release.go:85 component=release release=single-test targetNamespace=bootstrap resource=bootstrap:helmrelease/microservice1 helmVersion=v3 error="failed to prepare chart for release: chart unavailable: fatal: bad revision 'owner-refs', full output:\n fatal: bad revision 'owner-refs'\n"

See https://github.com/fidelity/kraan/pull/35/files#diff-439ae7262cbdf5c619a721946100b290 for test data being used.

Integration Tests

At present we have integration tests in packages.
These should be moved to top level tests/integration directory.
Also integration tests should not use external resources (e.g K8s cluster), these interactions should be mocked fake implementations used.
The integration tests should test integration between the packages in the project so no mocks of project packages should be used.

Support source path that has no valid yaml files in.

Support source path that has no valid yaml files in. This is required to support the scenario where all HelmReleases have been moved to other layers but this layer still needs to be processed in order to perform prune processing.

Applier Implementation

Implement CI using github actions

Need to implement CI tests

Add printer columns to Custom Resource Definition

Adding printer column settings to CRD will improve the output from kubectl get ...

Layers not getting processed

During demo I noticed that occasionally a requeue after delay does not work
This should be addressed by implementing #8
in the mean time, restarting the kraan-controller causes the layers to be reprocessed

Fix parsing of sync period command line argument

The sync period command line argument is being processed as a string then parsed to a duration. Flags package has a method for handling duration arguments directly

Implement json format logging

Want to generate json format logs

Liveness and Readiness probes are periodically failing at present.

The Kraan controller implements readiness and liveness tests using the controller manager's default checks (https://github.com/fidelity/kraan/blob/master/pkg/main/main.go#L105) but these periodically fail so are currently disabled in the deployment yaml, this needs to be fixed.

layer processing order not correct

Seeing an issue where the 'apps' layer applies before the layers it depends on.

I delete HelmReleases and update AddonsLayers as follows:

sed -i s/0.1.13/0.1.14/g $REPOS_PATH/addons-config/testdata/addons/addons.yaml;kubectl delete helmreleases.helm.fluxcd.io -A --all;kubectl apply -f $REPOS_PATH/addons-config/testdata/addons/addons.yaml;watch kubectl get al

This reconcile is in response to the helm release being deleted.

2020-09-09T12:42:37.010+0100	DEBUG	hr sync	HR update, requeuing	{"Name": "apps", "HR": "apps/microservice-1", "at": "controllers.indexHelmReleaseByOwner() - addons_controller.go(267)"}
....
2020-09-09T12:42:37.510+0100	DEBUG	controllers.AddonsLayer	apply required	{"Name": "apps", "Spec": {"source":{"name":"addons-config","namespace":"gitops-system","path":"./testdata/addons/apps"},"prereqs":{"k8sVersion":"v1.16","dependsOn":["[email protected]","[email protected]"]},"interval":"0s","version":"0.1.13"}, "Status": {"conditions":[{"type":"Deployed","status":"False","version":"0.1.08","lastTransitionTime":"2020-09-09T10:37:22Z","reason":"AddonsLayer is Deployed"},{"type":"Deployed","status":"False","version":"0.1.09","lastTransitionTime":"2020-09-09T10:44:05Z","reason":"AddonsLayer version 0.1.08 is Deployed"},{"type":"Deployed","status":"False","version":"0.1.09","lastTransitionTime":"2020-09-09T10:44:09Z","reason":"AddonsLayer version 0.1.09 is Deployed"},{"type":"Deployed","status":"False","version":"0.1.11","lastTransitionTime":"2020-09-09T10:54:53Z","reason":"AddonsLayer version 0.1.11 is Deployed"},{"type":"Applying","status":"False","version":"0.1.11","lastTransitionTime":"2020-09-09T11:01:36Z","reason":"AddonsLayer is being applied","message":"The applying status means the manager is either applying the yaml files or waiting for the HelmReleases to successfully deploy."},{"type":"Deployed","status":"False","version":"0.1.12","lastTransitionTime":"2020-09-09T11:01:48Z","reason":"AddonsLayer version 0.1.12 is Deployed"},{"type":"Applying","status":"False","version":"0.1.12","lastTransitionTime":"2020-09-09T11:17:31Z","reason":"AddonsLayer is being applied","message":"The applying status means the manager is either applying the yaml files or waiting for the HelmReleases to successfully deploy."},{"type":"Deployed","status":"False","version":"0.1.12","lastTransitionTime":"2020-09-09T11:17:45Z","reason":"AddonsLayer version 0.1.12 is Deployed"},{"type":"Applying","status":"False","version":"0.1.12","lastTransitionTime":"2020-09-09T11:19:11Z","reason":"AddonsLayer is being applied","message":"The applying status means the manager is either applying the yaml files or waiting for the HelmReleases to successfully deploy."},{"type":"Deployed","status":"True","version":"0.1.13","lastTransitionTime":"2020-09-09T11:19:19Z","reason":"AddonsLayer version 0.1.13 is Deployed"}],"state":"Deployed","version":"0.1.13"}, "at": "controllers.(*AddonsLayerReconciler).processApply() - addons_controller.go(110)"}

Need to do more debugging to understand how to address this scenario.

Describe alternatives you've considered
I could create my own HelmChart with the custom yaml resources but it's cumbersome and unnecessary.

Add go generate to makefiles

Update makefiles to get mock... generated when package source changes

Create Developers Guide

Suggest we create a dev guide for the project like https://github.com/paulcarlton-ww/go-utils/blob/master/docs/dev-guide.md to explain how to buil, test and run etc.