For charts like Istio and Longhorn today, we have multiple release lines that need to be maintained for a given package in a single branch.

Currently, the workaround is to keep the chart name the same and create a brand new package to track the separate release line. e.g.

assets/longhorn-1.1 tracks the 1.1.x release line
assets/longhorn-1.2 tracks the 1.2.x release line

but because both are named longhorn internally, the index.yaml has them as the same chart (since it is indexed by chart name, not package name).

Instead, we should be able to have something like packages/longhorn/1.1/* and packages/longhorn/1.2/* which allows each of those directories to output a chart that is all stored under assets/longhorn instead.

Epic Issue: #60

When a local directory is specified, the last location in the path should be replaced by dependencyName instead of appending dependencyName. At the current moment, this would result in something like file://./charts/gitjob => charts/gitjob/gitjob instead of charts/gitjob.

Instead of having a packageVersion appended to the version presumed from the chart, we will deprecate the packageVersion flag entirely. Instead, the version will be specified in the package.yaml itself.

This will involve removing all references of it from the code, including but not limited to:

If we want to introduce the version on make charts

e.g. the version will that we patch to will be ignored, even if it is patched like in

https://github.com/rancher/charts/blob/d688738828eba441a4af9621e5bbed449aa744a8/packages/rancher-monitoring/generated-changes/patch/Chart.yaml.patch#L55-L56

We just need to modify

A chart like rancher-istio should be able to create rancher-istio-crd based on CRDs introduced in its subchart, rancher-kiali-server. That way we can avoid releasing rancher-kiali-server-crd when trying to install rancher-istio in favor of just installing rancher-istio-crd.

This issue is an EPIC to track several changes in charts-build-scripts + rancher/charts that need to be addressed.

1. #62
2. #61
3. #54
4. #63

Currently, every call to make prepare and make patch repulls the tgz file or Github repository, which is really slow.

We should modify this to in some way cache the charts-original, so simultaneous calls to make prepare and make patch don't have to keep re-pulling and deleting the upstream chart.

This caching should never happen on a make charts, make sync, or make validate; only on make prepare and make patch.

Without checking for this, you could accidentally run make prepare followed by make patch and have a .orig file show up on your overlay without any error.

How would I use this for new charts? What does it do?

Add the following field to Charts and AdditionalCharts to support forking manifests

manifests:
- url: <required> # UpstreamOptions
  subdirectory: <optional> # UpstreamOptions
  commit: <optional> # UpstreamOptions
  filepath: <optional> # a specific file in the subdirectory to pull in the manifest for
  manifestPath: <required> # Where to drop the manifests, directories along path will be created if they don't exist

For upstream charts that we fork, we need to make sure that any licenses from the upstream repository can be included in the chart on a make charts.

Add a set of additional flags that can modify the templates/ directory of a CRD chart on prepare:

# Under additionalCharts[i].crdOptions
prepareOptions:
  syncVersionWithMainChart: # whether to automatically update the version on the templates/Chart.yaml to match the main chart's version        

I was using the following package.yaml:

url: https://github.com/k8snetworkplumbingwg/sriov-network-operator.git
subdirectory: deployment/sriov-network-operator
commit: v1.0.0
packageVersion: 01
additionalCharts:
  - workingDir: charts-crd
    crdOptions:
      templateDirectory: crd-template
      crdDirectory: templates
      addCRDValidationToMainChart: true

Suddenly, make prepare failed because a patch was not able to apply correctly, even though it was working yesterday. After investigating, I realized that even if make prepare did not throw any error, it was not cloning the repo with the tag v1.0.0 but master. If I modify the v1.0.0 with something obviously wrong like: commit: va76182b5f6f8a, make prepare works even though it ends up cloning master:

./bin/charts-build-scripts prepare
INFO[0000] Pulling k8snetworkplumbingwg/sriov-network-operator@va76182b5f6f8a1[path=deployment/sriov-network-operator] from upstream into charts 
INFO[0022] Loading dependencies for chart               
INFO[0022] Applying changes from generated-changes      
INFO[0022] Loading dependencies for chart               
INFO[0022] Copying CRDs from charts/crds to charts-crd/templates 
INFO[0022] Deleting charts/crds/k8s.cni.cncf.io_networkattachmentdefinitions_crd.yaml 
INFO[0022] Deleting charts/crds/sriovnetwork.openshift.io_sriovibnetworks.yaml 
INFO[0022] Deleting charts/crds/sriovnetwork.openshift.io_sriovnetworknodepolicies.yaml 
INFO[0022] Deleting charts/crds/sriovnetwork.openshift.io_sriovnetworknodestates.yaml 
INFO[0022] Deleting charts/crds/sriovnetwork.openshift.io_sriovnetworks.yaml 
INFO[0022] Deleting charts/crds/sriovnetwork.openshift.io_sriovoperatorconfigs.yaml 
INFO[0022] Adding templates/validate-install-crd.yaml to main chart 

I understand that the script does not work with tags and only with commits. But if I write a wrong commit, it would be good if it failed or threw a warning saying that because it can't find that commit, it will use master

On running a make prepare today, if applying a patch fails we just error out.

Instead, running make prepare should be an interactive process where, if a three way patch between the upstream, scripts, and the user is observed, the user should have the ability to remediate the errors before continuing a the call, similar to the behavior exhibited by a git rebase on encountering conflicts.

make charts, which uses make prepare under the hood, should also specify a non-interactive call of make prepare, which will fail out if it encounters any patches.

Currently, for packages like gke-operator (https://github.com/rancher/charts/tree/dev-v2.6/packages/rancher-gke-operator) that just contain a package.yaml, we need to make a PR to upstream and then make a PR to our charts repository to bump the version.

Instead, we should add support for a GitHub Bot that periodically (daily?) looks at all package.yaml files for autoBumpCharts: true, checks if there is a new Release or GitHub commit to go to, and automatically opens up a PR that bumps the chart and assigns a user to it based on the specified maintainer: GITHUB_USERNAME value in the package.yaml to review and merge (at which point they can also take care of any conflicts that come up).

While this would require some amount of effort to set up (probably would require 2 weeks), this can move us towards a far more automated process for charts.

Add a field to the package.yaml to allow for different options for patches exported to generated-changes/patches/.

The default (as it is now) would be:

diffOptions:
  type: unified_diff

But we could potentially support other diff types.

Would predominantly just require us to make changes to diff/diff.go.

Currently, it is always a manual step to delete the old index.yaml, assets, and charts related to an unreleased chart that will no longer be released.

Instead, make charts should generate an index.yaml based on a new, sorted index.yaml that merges:

• All index.yamls from charts we validate against
• The current charts tracked in packages/

Once the index is generated, we should clean up any charts/ or assets/ whose URLs are untracked in the merged index.yaml.

Epic Issue: #60

Since the Chart.yaml and values.yaml files are prone to conflicts on rebases but are guaranteed to be real YAML files (e.g. they won't have any weird go templating stuff in them, unlike other Helm files), the diff we store for these two root files should be a JSON patch instead of a Unix diff.

Since it's far more unlikely for a JSON patch applied on the base chart to encounter a conflict than a Unix diff, this should make the process of applying rebases far simpler and remove the caveat the scripts have with working with these "Managed Files"

./bin/charts-build-scripts --version                             
charts-build-scripts version v0.0.4 (567c991)

When there are multiple-line comments in the Chart.yaml file of the dependencies chart, like the following

apiVersion: v2
name: sachet
description: Sachet is an SMS alerting tool for the Prometheus Alertmanager
type: application

# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.1.0

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
# follow Semantic Versioning. They should reflect the version the application is using.
# It is recommended to use it with quotes.
appVersion: 0.2.3

Running the make prepare on the main chart will fail with the following error

> PACKAGE=rancher-alerting-driver make prepare
...
FATA[0001] Encountered error while preparing main chart: Encountered error while trying to prepare dependencies in charts: error unpacking sachet in rancher-alerting-driver: cannot load Chart.yaml: error converting YAML to JSON: yaml: line 19: could not find expected ':' 
make: *** [prepare] Error 1

Currently, ./scripts/package-ci is used in order to manually verify whether an upgrade to rancher/charts-build-scripts will break a charts-build-scripts repository.

It does this by running make prepare, make patch, make clean, and asserting no changes are found.

However, when two different machines are used to run make patch today, it's possible that both can produce equally valid patches that are slightly different due to the way each machine chooses to construct the patch. For example, the following failures are noted in rancher/charts today:

Failed to pass ci for the following packages:
- epinio/epinio-ui generated additional changes: M packages/epinio/epinio-ui/generated-changes/patch/templates/_helpers.tpl.patch
- harvester/harvester-cloud-provider generated additional changes: M packages/harvester/harvester-cloud-provider/generated-changes/patch/Chart.yaml.patch
- harvester/harvester-csi-driver generated additional changes: M packages/harvester/harvester-csi-driver/generated-changes/patch/Chart.yaml.patch
- longhorn/longhorn-1.2 generated additional changes: M packages/longhorn/longhorn-1.2/generated-changes/patch/README.md.patch
- longhorn/longhorn-1.3 generated additional changes: M packages/longhorn/longhorn-1.3/generated-changes/patch/README.md.patch
- neuvector generated additional changes: M packages/neuvector/generated-changes/overlay/crds/_helpers.tpl
- rancher-alerting/rancher-prom2teams generated additional changes: M packages/rancher-alerting/rancher-prom2teams/generated-changes/patch/templates/_helpers.tpl.patch M packages/rancher-alerting/rancher-prom2teams/generated-changes/patch/templates/deployment.yaml.patch
- rancher-gatekeeper generated additional changes: M packages/rancher-gatekeeper/generated-changes/patch/templates/_helpers.tpl.patch
- rancher-istio/1.14/rancher-kiali-server generated additional changes: M packages/rancher-istio/1.14/rancher-kiali-server/generated-changes/patch/templates/_helpers.tpl.patch
- rancher-monitoring/rancher-grafana generated additional changes: M packages/rancher-monitoring/rancher-grafana/generated-changes/patch/templates/tests/test-podsecuritypolicy.yaml.patch
- rancher-monitoring/rancher-node-exporter generated additional changes: M packages/rancher-monitoring/rancher-node-exporter/generated-changes/patch/templates/daemonset.yaml.patch M packages/rancher-monitoring/rancher-node-exporter/generated-changes/patch/values.yaml.patch

But on inspecting the changes, both patches are valid; i.e. here is the diff on the diff.

diff --git a/packages/longhorn/longhorn-1.2/generated-changes/patch/README.md.patch b/packages/longhorn/longhorn-1.2/generated-changes/patch/README.md.patch
index b92a6e20c..ce9896912 100644
--- a/packages/longhorn/longhorn-1.2/generated-changes/patch/README.md.patch
+++ b/packages/longhorn/longhorn-1.2/generated-changes/patch/README.md.patch
@@ -34,14 +34,14 @@
 -helm delete longhorn --purge
 -```
 +To prevent damage to the Kubernetes cluster, we recommend deleting all Kubernetes workloads using Longhorn volumes (PersistentVolume, PersistentVolumeClaim, StorageClass, Deployment, StatefulSet, DaemonSet, etc).
-+
-+From Rancher Cluster Explorer UI, navigate to Apps page, delete app `longhorn` then app `longhorn-crd` in Installed Apps tab.
  
 -With Helm 3 to uninstall Longhorn.
 -```
 -helm uninstall longhorn -n longhorn-system
 -kubectl delete namespace longhorn-system
 -```
++From Rancher Cluster Explorer UI, navigate to Apps page, delete app `longhorn` then app `longhorn-crd` in Installed Apps tab.
++
  
  ---
  Please see [link](https://github.com/longhorn/longhorn) for more information.

It is equally valid here to add From Rancher Cluster Explorer UI... before or after removing With Helm 3 to uninstall Longhorn, so both patches are logically the same but are failing CI.

To fix this, one of three approaches should be taken:

1. We should ensure that the output of make patch is the same across all machines by avoiding using the system call to run diff -uN to generate these patches:

   charts-build-scripts/pkg/diff/diff.go

   Line 25 in 2901446

   cmd := exec.Command(pathToDiffCmd, "-uN", "-x *.tgz", "-x *.lock", srcPath, dstPath)

2. We should ensure that the output of make patch is the same across all machines by containerizing rancher/charts-build-scripts runs; to do this, we'd need to avoid using an osfs (

   charts-build-scripts/pkg/filesystem/filesystem.go

   Line 24 in 2901446

   return osfs.New(path)

   ) and remove all os calls in favor of a memfs https://github.com/go-git/go-billy/blob/205ba6585ca03bd60e304f7fa0b270232ba1d707/memfs/memory.go#L28-L31
3. We should refactor the logic of the package-ci script to avoid comparing make patch and instead compare the output of make charts to ensure the underlying chart that is generated has not been changed.

As @aiyengar2 has pointed out, it looks like the invalid checksum message is coming from the go-git library here. It seems to occur specifically when retrieving the repo's information to check if the working tree is clean, see here.

Deleting my fork, recreating and re-attempting a make validate (with or without making any actual changes) results in the same error.

Not sure if local versions of these matter but here are my git and go versions

• git version: 2.38.0
• go version: 1.18

on local dev-v2.7:

on upstream dev-v2.7:

make validate result:

current commit hash comparison:

In rancher/charts today, we have a somewhat complicated process for release that involves running a set of git commands in order to be able to checkout only specific directories.

Unlike that, in rancher/system-charts we simply need to make a PR from dev-v2.x -> release-v2.x, since both are based on the same Git tree. This is a far simpler process to follow for release captains and has the benefit of auto-updating release PRs the moment a dev merges something into the dev branch.

Therefore, we would like to switch the scripts to the older model by removing any differences between the staging and release branch so that we can easily merge PRs from one to the other.

This will also involve deprecating branch-specific make template commands, since both staging and release will maintain the same docs.

Epic Issue: #60

./bin/charts-build-scripts --version gives the current version, so if we can parse it as semver and the version on pull-scripts is newer we should replace automatically.

If the version is a commit hash, we should ignore and throw a warning on the console.

Seen in https://github.com/rancher/charts/pull/1150/checks?check_run_id=2432884734.

Can be reproduced by going back to rancher/charts@eb38d50 and trying to add os: windows to the Windows image.

Currently the pull-scripts mechanism manually runs a go build in order to create the binary in the bin/ directory.

We need to switch to a Docker-based approach, which was inhibited before due to Docker slowness associated with making numerous file changes on a mounted volume from the host.

One thing that can enable this is switching the billy.Filesystem to use a memFs instead of an osFs

Support a workflow that makes it easier to be able to rebase existing patches on a chart to a new package.yaml upstream.

Add a Bash script to help interactively perform rebases on upstreams using git subtree and bash shell

Edit: See below.

Currently, one of the manual checks that are performed to check if a migration is successful or if released charts don't introduce any changes involves running scripts currently located in https://github.com/aiyengar2/charts-diff/tree/release-v2.5.6 in order to generate a directory of diffs between one charts script branch and another.

As part of a pull-request workflow on the Live branch, it would be nice if that could automatically run these scripts to generate a diff and publish it in a branch on rancher/charts-diff that could allow reviewers to confirm that the changes they introduce match what is currently available in the source branch.

Create script for quick resolution of situations like rancher/charts#982.

Basic outline is in rancher/charts#982 (comment), but we should try to see if there's a way to do it without modifying time stamps

Need to add docs explaining when to increment a packageVersion vs. releaseCandidateVersion in packages/README.md

Charts that are already released need to be able to update rancher-min-version / rancher-max-version / kubeVersions in a way that is compatible with our current CI.

make validate currently does not allow for any changes to be made to already released charts. Therefore, we need some sort of workaround in order to be able to bypass CI for these changes so that it still passes when only those specific fields in the Chart.yaml are changed.

Epic Issue: #60

Instead of deprecating the packageVersion option, we will introduce one new option: version.

Here are some general rules we will follow:

• rule 1: version and packageVersion both are optional but can not exist at the same time
• rule 2: if the version exists, then use it as the final version ( developer needs to make sure ti is a valid semVer)
• rule 3: if the packageVersion exists, then add it to the patch version part after the chart's patch files are applied
• rule 4: if neither packageVersion nor version exists, we should default to omitting any packageVersion. A great example of where this would be used is fleet, which is a “do not touch me” policy

As @ibuildthecloud accurately pointed out, there's no real reason for us to distinguish releaseCandidateVersions from packageVersions except for the fact that it is nice to have 1 packageVersion bump occur per release.

And from a user perspective, whether a chart bumps from 9.4.20401 to 9.4.20415 does not make a difference as long as one number is greater than the other.

Instead of trying to resolve issues like #7 or adding logic for "Open" vs "Closed" packages as described in https://confluence.suse.com/display/EN/Multi-BranchSet+Release+Process+For+Feature+Charts, we should just deprecate releasedCandidateVersions in favor of always bumping the packageVersion in the pipeline.

As a result, a Package by definition will always be tied to its packageVersion and will be immutable (as opposed to temporarily mutable via multiple releasedCandidateVersions, as defined before).

We should also increase the size of the packageVersion to 4 digits to accommodate for up to 9999 changes to the same major.minor.patch of a chart. After that point, we might as well fork it 😄

package.yaml should be able to point to a tag under a new tag field for Github-based packages

An AdditionalChart that has CRDOptions should also be able to specify UpstreamOptions as long as templateDirectory is not set. This should trigger the charts to pull the chart base from a specified upstream and then move CRDs over to a given directory (conflicting filenames will be overridden).

Before releasing, we'll need to confirm generated changes for additional charts works as expected.

Running prepare should error out if a user specifies both upstream and templateDirectory. Eventually, we can completely deprecate the templateDirectory field by just storing the template directly in the workingDir of the CRD chart and using url: local.

Why?

This feature enhancement will allow us to use the packages/{packageName} mechanism to point to two new charts: rancher-crd-chart-base and rancher-crd-chart-assume-ownership.

Once this is done, we'll be able to maintain all of the CRD charts in two central packages instead of copying changes to templates across Backup, CIS, Monitoring, Kiali Server, etc.

Currently, rancher-monitoring required us to repackage packages/pushprox for every single pushprox dependency (e.g. compare the currently released version in main vs the one that will be released in dev-v2.5).

Instead, we should be able to specify a dependency.yaml that takes in aliases that are written into the requirements.yaml so that we only need to have rancher-pushprox pulled in once and referred to several times, as we did before.

On prepare:

• Get the Chart.Version (e.g. 2.3.1-<anything>)
• Remove the non-semver component appended to the end (e.g. anything outside the regex \d+\.\d+\.\d+)
• Parse the non-semver part; if it is in a defined format (e.g. rc<rcVersion><packageVersion> like rc100) then get the packageVersion / rcVersion from it and throw a warning if it doesn't match the contents on the package.yaml. Otherwise, just throw a catch all warning that the chart will be packaged with <packageVersion>-rc<rcVersion> appended to it.

On charts:

• On exporting the chart, remove anything from the Chart.Version that is not semver (e.g. 2.3.1-<anything> -> 2.3.1).

I'm using charts-build-scripts on the rke2-charts repo. Let's assume that we have a rke2-xxx package/chart with the following structure:

packages/rke2-xxx/
├── generated-changes
│   ├── exclude
│   └──  patch
└── package.yaml

I have a "clean" state of my repo and some files both in exclude/ and path/ directories.

Then I'm doing:

make prepare

The chart structure changes to:

packages/rke2-xxx/
├── charts
├── generated-changes
│   ├── exclude
│   └──  patch
└── package.yaml

My expectation would be that charts content:

1. Has the patches applied (which is true, that happens)
2. Doesn't have exluded files (which is not true, that doesn't happen and the files are still there)

I would expect the condition 2 to be also true. To make my patches correct again, I need to manually remove files I want to exclude.

Modifying 2.5 Rancher branches with the changes introduced in #60

Pending Rancher 2.5.12 release...

The following line messes with how Helm 3 handles dependencies:

https://github.com/rancher/charts-build-scripts/blob/v0.2.4/pkg/charts/dependencies.go#L257

more at rancher/partner-charts#265

Currently, the scripts support only two types of packages:

• local charts: charts that exist directly in the Source branch
• forked charts: charts that only exist as a set of patch / overlay / exclude files applied on top of a base

This proposal targets a new type of chart that is a hybrid between local charts and forked charts: branched charts.

What is a branched chart?

A branched chart is a forked chart whose workingDir is directly maintained in another repository / branch.

On merging a change to the target repository / branch, a Github Workflow will automatically pull the target in the workingDir of the target package (instead of running Chart.Prepare()), run make patch to create or update the generated-changes/ directory, and commit it to the Source branch.

Why is this useful?

Branched charts can directly point to an upstream chart, so you can use existing git rebase tools in order to pull in the latest upstream and resolve patch conflicts to cut a new release.

Pushing in a change to these charts also does not require developers to have to view "diff on diff" PRs since you are directly merging the new workingDir into the branch, but you can still call out the difference between your repository and an upstream by viewing the Source branch and do stuff like CRD charts directly on the Source branch instead.

PR run: https://github.com/rancher/rke2-charts/pull/73/checks?check_run_id=2443701093

./bin/charts-build-scripts validate
5
time="2021-04-27T00:44:08Z" level=info msg="Validating against released charts in main"
6
time="2021-04-27T00:44:08Z" level=info msg="Local chart does not need to be prepared"
7
time="2021-04-27T00:44:08Z" level=info msg="Loading dependencies for chart"
8
time="2021-04-27T00:44:08Z" level=info msg="Generated archive: assets/rke2-canal/rke2-canal-v3.13.300-build2021022302-rc00.tgz"
9
time="2021-04-27T00:44:08Z" level=info msg="Generated chart: charts/rke2-canal/rke2-canal/v3.13.300-build2021022302-rc00"
10
time="2021-04-27T00:44:08Z" level=info msg="Local chart does not need to be prepared"
11
time="2021-04-27T00:44:08Z" level=info msg="Loading dependencies for chart"
12
time="2021-04-27T00:44:08Z" level=info msg="Found chart options for cilium in generated-changes/dependencies/cilium/dependency.yaml"
13
time="2021-04-27T00:44:08Z" level=info msg="Found chart options for rke2-cilium-hard-defaults in generated-changes/dependencies/rke2-cilium-hard-defaults/dependency.yaml"
14
panic: runtime error: invalid memory address or nil pointer dereference
15
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x1643436]
16

17
goroutine 1 [running]:
18
github.com/rancher/charts-build-scripts/pkg/charts.LocalPackage.Pull(0xc00023b119, 0x6, 0x0, 0x1c03a40, 0xc000323760, 0x1c03a40, 0xc0002cdf60, 0x19ad813, 0x6, 0x0, ...)
19
	/home/runner/work/rke2-charts/rke2-charts/charts-build-scripts/pkg/charts/puller.go:30 +0xb6
20
github.com/rancher/charts-build-scripts/pkg/charts.PrepareDependencies(0x1c03a40, 0xc000323760, 0x1c03a40, 0xc000323a40, 0x19ad813, 0x6, 0x19c148b, 0x11, 0x26d3ee8, 0x0)
21
	/home/runner/work/rke2-charts/rke2-charts/charts-build-scripts/pkg/charts/dependencies.go:75 +0xa44
22
github.com/rancher/charts-build-scripts/pkg/charts.(*Chart).Prepare(0xc000a56e00, 0x1c03a40, 0xc000323760, 0x1c03a40, 0xc000323a40, 0xf, 0x17630e0)
23
	/home/runner/work/rke2-charts/rke2-charts/charts-build-scripts/pkg/charts/chart.go:27 +0xf9
24
github.com/rancher/charts-build-scripts/pkg/charts.(*Package).Prepare(0xc000a56e00, 0x0, 0x0)
25
	/home/runner/work/rke2-charts/rke2-charts/charts-build-scripts/pkg/charts/package.go:39 +0x67
26
github.com/rancher/charts-build-scripts/pkg/charts.(*Package).GenerateCharts(0xc000a56e00, 0x0, 0x0)
27
	/home/runner/work/rke2-charts/rke2-charts/charts-build-scripts/pkg/charts/package.go:95 +0x4f
28
github.com/rancher/charts-build-scripts/pkg/sync.ValidateRepository(0x1c03a40, 0xc0002fde20, 0xc0008b3dd0, 0x2a, 0x0, 0x0, 0xc00081e690, 0x4, 0x1, 0x0, ...)
29
	/home/runner/work/rke2-charts/rke2-charts/charts-build-scripts/pkg/sync/validate.go:52 +0x765
30
main.validateRepo(0xc00032e2c0)
31
	/home/runner/work/rke2-charts/rke2-charts/charts-build-scripts/main.go:248 +0x425
32
github.com/urfave/cli.HandleAction(0x174e660, 0x1a84900, 0xc00032e2c0, 0xc00032e2c0, 0x0)
33
	/root/go/pkg/mod/github.com/urfave/[email protected]/app.go:526 +0x58
34
github.com/urfave/cli.Command.Run(0x19b2f4d, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1a2c1f8, 0x77, 0x0, ...)
35
	/root/go/pkg/mod/github.com/urfave/[email protected]/command.go:173 +0x58e
36
github.com/urfave/cli.(*App).Run(0xc000526540, 0xc00000c080, 0x2, 0x2, 0x0, 0x0)
37
	/root/go/pkg/mod/github.com/urfave/[email protected]/app.go:277 +0x7e8
38
main.main()
39
	/home/runner/work/rke2-charts/rke2-charts/charts-build-scripts/main.go:121 +0x85a
40
make: *** [Makefile:8: validate] Error 2

Probably because we return nil, nil here

In

However, if the only change that we see is a modified tgz file and the charts contained within those tgz files are the same (e.g. only timestamp differences), we should not fail make validate.

This is the root cause of why we saw a failure on make validate in https://github.com/rancher/charts/runs/3204473035?check_suite_focus=true, where release-v2.5 contained an asset that was generated from dev-v2.5.9 but it was compared against the asset in dev-v2.5. The contents in dev-v2.5.9's version of the chart and dev-v2.5's version of the chart were identical, but validate resulted in a failure since we did not do a deeper check of equality for the tgzs.

Need more documentation to allow users to read through how to add dependencies and modify them

Instead of starting with an empty ReleaseOptions{} here:

We should be loading the release.yaml from the existing file (if it exists), since all of our walk functions will not track existing items in the release.yaml that will be dumped anyways:

Currently, charts-build-scripts uses a osfs:

However, ideally we should be making loading the current directory into a memfs, making all calls in memory, and then only modifying the underlying filesystem after executing all operations.

https://github.com/go-git/go-billy/blob/205ba6585ca03bd60e304f7fa0b270232ba1d707/memfs/memory.go#L28-L31

Bug.

Reproduce:
delete a file that exists only in the package/generated-changes/overlay directory. ie. delete the file that we appended to the chart before.

Result:
The file still exists in the final chart in the assets and charts directories.

Fix:
if the chart already exists in the assets and charts directories, delete them before exporting the newly generated charts.

Followup to #1.

On a make validate after running the equivalent of make charts, we should check whether every CRD chart that was generated by the preparation of charts has a corresponding main chart version.