carvel-dev / vendir Goto Github PK
View Code? Open in Web Editor NEWEasy way to vendor portions of git repos, github releases, helm charts, docker image contents, etc. declaratively
Home Page: https://carvel.dev/vendir
License: Apache License 2.0
Easy way to vendor portions of git repos, github releases, helm charts, docker image contents, etc. declaratively
Home Page: https://carvel.dev/vendir
License: Apache License 2.0
Describe the problem/challenge you have
want to download imgpkg bundles.
Describe the solution you'd like
imgpkgBundle:
# image URL; could be plain, tagged or digest reference (required)
image: gcr.io/repo/image:v1.0.0
# specifies name of a secret with registry auth details;
# secret may include 'username', 'password' and/or 'token' keys (optional)
secretRef:
# (required)
name: my-image-auth
would use imgpkg pull -b ...
TBD is is too confusing that we are using image
for referring to bundle? mostly interested in doing that because kbld picks up image
key automatically for image ref.
Anything else you would like to add:
related: should we update image source to use image key instead of url.
What steps did you take:
[A clear and concise description steps that can be used to reproduce the problem.]
A typical GitHub project generally includes its license at the root of the project. When specifying the newRootPath, this will adjust the root path of the project after the download and since the legalPath likely exists outside of newRootPath, the legalPath ends up missing.
Consider this config:
---
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
# vertical pod autoscaler
- path: source/cnpo.tanzulabs.vmware.com/tenancy/vertical-pod-autoscaler
contents:
- path: vendor
git:
url: https://github.com/kubernetes/autoscaler.git
ref: vertical-pod-autoscaler/v0.9.2
includePaths:
- vertical-pod-autoscaler/deploy/*.yaml
excludePaths:
- vertical-pod-autoscaler/deploy/kustomization.yaml
- vertical-pod-autoscaler/deploy/vpa-beta-crd.yaml
- vertical-pod-autoscaler/deploy/vpa-beta2-crd.yaml
- vertical-pod-autoscaler/deploy/vpa-crd.yaml
newRootPath: vertical-pod-autoscaler/deploy
legalPaths:
- LICENSE
What happened:
[A small description of the issue]
Directory was appropriately downloaded, however the legalPaths were missing because (assumption) it existed outside of the newRootPath
What did you expect:
[A description of what was expected]
The legalPaths would be placed within the newRootPath specified.
Anything else you would like to add:
[Additional information that will assist in solving the issue.]
Environment:
vendir --version
):vendir --version
vendir version 0.19.0
Succeeded
/etc/os-release
): MacOSVote on this request
This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.
π "I would like to see this addressed as soon as possible"
π "There are other more important things to focus on right now"
We are also happy to receive and review Pull Requests if you want to help working on this issue.
Hi, could you please clarify what exactly should be specified, and where?
to reproduce:
git clone https://github.com/k14s/vendir
cd vendir/examples/image
vendir sync
output:
Error: Validating config: Validating directory 'vendor' (0): Validating directory contents 'docker.io/dkalinin/consul-helm-naked' (0): Expected directory contents type to be specified (one of git, manual, etc.)
Describe the problem/challenge you have
When working with packages and where the vendir.yml
file is located in a sub directory, it is extra work to ensure that you first cd
into the directory before running vendir sync
. If running command manually or scripting actions working across multiple packages, means have to use something like the following to ensure left in original directory when command done.
(cd package/foo/bundle; vendir sync)
Describe the solution you'd like
Would like to see a command line option where can tell vendir sync
the directory to run in. Eg.,
vendir sync -c package/foo/bundle
The c
for option indicative of "change directory".
Thus vendir sync
would then itself change the working directory of the process to the specified directory before taking any action.
Anything else you would like to add:
N/A
Vote on this request
This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.
π "I would like to see this addressed as soon as possible"
π "There are other more important things to focus on right now"
We are also happy to receive and review Pull Requests if you want to help working on this issue.
Describe the problem/challenge you have
in some cases it's useful to retain certain temp files among fetched directories. (e.g. temp files generated by tools that were operating on top of fetched files.)
Describe the solution you'd like
add ignorePaths key on directory contents. ignorePaths would specify a set of globs that would be used to match in existing directories and copy them over on top of newly fetched sources.
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
- path: components/terraform
contents:
- path: account-map
git:
url: https://github.com/cloudposse/terraform-aws-components
ref: 0.137.0
newRootPath: modules/account-map
includePaths:
- modules/account-map/**/*
ignorePaths: &ignore
- backend.tf.json
- catalog/*
- path: account-settings
git:
url: https://github.com/cloudposse/terraform-aws-components
ref: 0.137.0
newRootPath: modules/account-settings
includePaths:
- modules/account-settings/**/*
ignorePaths: *ignore
Details:
We're using vendir
in a number of different ways to vendor configuration files in from a centralized source.
vendir.yml
.The problem is when we disable a configuration, then vendir sync
wants to delete all the files. And if we add files into a vendir'd configuration, and then run vendir sync
, it will delete those extraneous files. The current behavior makes sense, however, we're hoping we could introduce something that will address the use-cases above. We'll help implement it as well.
Support an enabled
setting. In the example below, enabled
is set to false
, so the configuration is skipped. Running vendir sync
will not attempt to modify or delete the local copy (it's as though the stanza were deleted from the file). This is useful for when we deliberately want to diverge, but want to keep a record in vendir.yml
the point at which we diverged.
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
# Terraform components
- path: components/terraform
contents:
- path: account-map
# Disable this from running
enabled: false
git:
url: https://github.com/cloudposse/terraform-aws-components
ref: 0.137.0
newRootPath: modules/account-map
includePaths:
- modules/account-map/**/*
- path: account-settings
git:
url: https://github.com/cloudposse/terraform-aws-components
ref: 0.137.0
newRootPath: modules/account-settings
includePaths:
- modules/account-settings/**/*
Support an ignorePaths
block which specifies paths that should not be deleted (or synchronized), even if they do not exist at the source. This is useful when we layer in configurations. For example, we add additional configuration files that do not exist at the source.
In the example below, the ignorePaths
block says to ignore the fact that backend.tf.json
does not exist at the source and also, ignore any files in the catalog/*
folder.
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
# Terraform components
- path: components/terraform
contents:
- path: account-map
# Ignore files which are generated locally and should not be overridden or deleted.
ignorePaths: &ignore
- backend.tf.json
- catalog/*
git:
url: https://github.com/cloudposse/terraform-aws-components
ref: 0.137.0
newRootPath: modules/account-map
includePaths:
- modules/account-map/**/*
- path: account-settings
# Use a YAML anchor to ignore common files
<<*ignore
git:
url: https://github.com/cloudposse/terraform-aws-components
ref: 0.137.0
newRootPath: modules/account-settings
includePaths:
- modules/account-settings/**/*
Support a --skip-delete
flag. on vendir sync
. This is useful during development or as a precaution to avoid accidentally deleting any files.
Running vendir sync --skip-delete
will run as usual, but not remove any files (but possibly overwrite).
Describe the problem/challenge you have
When it comes to using overlapping sync options, it's usually that you would mention the same repo to pull from several times in order to substitute one of the folders with a custom one from the private repo, for instance:
directories:
- path: components
contents:
- path: terraform/tfstate-backend
git:
url: [email protected]:cloudpossy/terraform-aws-components.git
ref: all-new-components
newRootPath: modules/tfstate-backend
- path: terraform/account
git:
url: [email protected]:cloudpossy/terraform-aws-components.git
ref: all-new-components
newRootPath: modules/account
- path: terraform/account-map
git:
url: [email protected]:cloudpossy/terraform-aws-components.git
ref: all-new-components
newRootPath: modules/account-map
- path: terraform/nexus
git:
url: [email protected]:privateorg/terraform-aws-components.git
ref: atmos-novel
newRootPath: privateorg/modules/nexus
Here where it gets too long to wait when vendir sync
is used because it basically tries to react on that planly by downloading [email protected]:cloudpossy/terraform-aws-components.git
every time it's mentioned in the spec without attempting to re-use the cached sample.
I'd guess, that it's something of utmost importance to implement this simple functionality and make the tool much more efficient.
Vote on this request
This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.
π "I would like to see this addressed as soon as possible"
π "There are other more important things to focus on right now"
We are also happy to receive and review Pull Requests if you want to help working on this issue.
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
minimumRequiredVersion: 0.8.0
directories:
- path: vendor
and
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
minimumRequiredVersion: 0.8.0
directories:
- path: vendor/
should be equivalent. in v0.10.0 trailing slash causes a fs rename error.
We use the following configuration in vendir.yml
:
- path: components/terraform
contents:
- path: account-map
git:
url: https://github.com/cloudposse/terraform-aws-components
ref: 0.135.0
newRootPath: modules/account-map
includePaths:
- modules/account-map/**/*
- path: account
git:
url: https://github.com/cloudposse/terraform-aws-components
ref: 0.135.0
newRootPath: modules/account
includePaths:
- modules/account/**/*
and getting the following error:
Error: Parsing resource config 'vendir.yml': Unmarshaling config: Validating config: Expected to not manage overlapping paths: 'components/terraform/account-map' and 'components/terraform/account'
Note that account_map
does not work either, throwing the same validation error.
Moving the folders to the top-level throws the same error as well:
- path: components/terraform/account-map
contents:
- path: .
git:
url: https://github.com/cloudposse/terraform-aws-components
ref: 0.135.0
newRootPath: modules/account-map
includePaths:
- modules/account-map/**/*
- path: components/terraform/account
contents:
- path: .
git:
url: https://github.com/cloudposse/terraform-aws-components
ref: 0.135.0
newRootPath: modules/account
includePaths:
- modules/account/**/*
This does not work either:
- path: accountmap
git:
url: https://github.com/cloudposse/terraform-aws-components
ref: 0.135.0
newRootPath: modules/account-map
includePaths:
- modules/account-map/**/*
- path: account
git:
url: https://github.com/cloudposse/terraform-aws-components
ref: 0.135.0
newRootPath: modules/account
includePaths:
- modules/account/**/*
throwing the error:
Error: Parsing resource config 'vendir.yml': Unmarshaling config: Validating config: Expected to not manage overlapping paths: 'components/terraform/accountmap' and 'components/terraform/account'
But this works ok:
- path: a-ccountmap
git:
url: https://github.com/cloudposse/terraform-aws-components
ref: 0.135.0
newRootPath: modules/account-map
includePaths:
- modules/account-map/**/*
- path: account
git:
url: https://github.com/cloudposse/terraform-aws-components
ref: 0.135.0
newRootPath: modules/account
includePaths:
- modules/account/**/*
So the issue is that the code checks not for the full path, but if a path starts with a string.
@cppforlife can you please take a look at the issue?
We like vendir
and want to use it in our architectures.
Thank you
originally filed here: carvel-dev/kapp-controller#6. since vendir will be used by kapp-controller (carvel-dev/kapp-controller#12), this feature belongs here.
vendir
in our https://github.com/cloudposse/packagesFwiw, here's how we do it:
fetching bundles using imgpkg was added, so we should add an e2e test for that behavior
useful when fetching github releases.
Hi, I wonder it would not be useful to be able to pick jar java artifacts (aka. maven/gradle dependencies).
We could target:
Error: Syncing directory 'vendor/k8s': Moving staging directory '.vendir-tmp/staging' to final location 'vendor/k8s': rename .vendir-tmp/staging vendor/k8s: no such file or directory
That vendir/k8s
directory does not exist, and vendir should probably create it as long as it's relative to the current path.
Any chance we can get a new release cut? =) We anxiously await use of #64?
We use the release to distribute the packages we depend on.
vendir currently supports the --directory
flag:
$ vendir sync -d my_dependency
But it doesn't save the lock file. I would like vendir to be able to isolate sync/bumps to a specific dependency.
I would like vendir
to strip/rename paths
- path: vendor/k8s
contents:
- path: eduk8s
git:
url: https://github.com/eduk8s/eduk8s
ref: 20.08.04.1
includePaths:
- resources/crds-v1/**/*
- resources/operator/**/*
and this ends up creating:
vendor/k8s/eduk8s/resources/crds-v1
vendor/k8s/eduk8s/resources/operator
but I would want:
vendor/k8s/eduk8s/resources/
everything just in this dir.
The use case is:
Iβm collecting the k8s resource definitions from multiple sources to then process them with ytt and apply them with kapp, and have a consistent directory structure. When I get stuff from github releases I just get the files I need in the root of the path, when I get it this other way it ends in subdirectories, and hence dir layout is not really consistent
What steps did you take:
vendir sync -l -d somepath
What happened:
I invoked vendir with -l
and -d some_path
and it downloaded all the artifacts in the lockfile, instead of just the locked dependency I was specifying.
What did you expect:
I was expecting to fetch only the directory specified by -d some_path
Anything else you would like to add:
Not really.
Environment:
Vote on this request
This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.
π "I would like to see this addressed as soon as possible"
π "There are other more important things to focus on right now"
We are also happy to receive and review Pull Requests if you want to help working on this issue.
In cf-for-k8s, we'd like to remove disableAutoChecksumValidation: true
from our entry for kpack, for example.
That block looks like this:
- path: github.com/pivotal/kpack
githubRelease:
slug: pivotal/kpack
tag: v0.0.8
disableAutoChecksumValidation: true
includePaths:
- release-0.0.8.yaml
And when we remove the disableAutoChecksumValidation
line, we get the error:
Error: Syncing directory 'config/_ytt_lib': Syncing directory 'github.com/pivotal/kpack' with github release contents: Finding checksums in release notes: Expected to find sha256 checksum for file 'logs-v0.0.8-linux.tgz'
...even though the only file in our include path is release-0.0.8.yaml
.
Here's the kpack release page in case that's helpful: https://github.com/pivotal/kpack/releases/tag/v0.0.8
I'm seeing an issue where vendir throws an error when checking the release asset file size: Checking asset 'file.txt' size: Expected file size to be 74, but was 9
. I checked the api response for release assets, it shows the asset size is 74 bytes. When I manually download the file, it is 74 bytes. Any idea what would cause this?
Some details from the API response (some parts are omitted or renamed):
[
{
"name": "file.txt",
"label": "",
"content_type": "application/text",
"state": "uploaded",
"size": 74,
"download_count": 5,
"created_at": "2020-08-04T19:58:28Z",
"updated_at": "2020-08-04T19:58:29Z",
"browser_download_url": "https://github.com/org/repo/releases/download/v3/file.txt"
}
]
What steps did you take:
vendir sync
with this config:
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
minimumRequiredVersion: 0.12.0
directories:
- path: config/upstream
contents:
- path: serving
git:
url: https://github.com/knative/serving
ref: v0.21.0
includePaths:
- config/core/**/*
What happened:
config/upstream/serving/config/core/300-imagecache.yaml
is an invalid symlink, pointing to a file that doesn't exist.
What did you expect:
I'm not 100% sure, probably it should collapse the symlink into a regular file with the contents of the file it pointed to. Maybe this needs to be configurable.
Anything else you would like to add:
I looked into this a little and found it was broken in the library that vendir depends on: otiai10/copy#32
Environment:
vendir --version
): 0.16.0/etc/os-release
): ubuntu focalWhat steps did you take:
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
- path: base
contents:
- path: "."
git:
url: https://github.com/keycloak/keycloak-operator
ref: "12.0.2"
includePaths:
- deploy/**/*
newRootPath: deploy/
vendir sync
:vendir sync
Fetching: base + . (git from https://github.com/keycloak/[email protected])
--> git init
Initialized empty Git repository in /Users/tester/.vendir-tmp/incoming/git/.git/
--> git config credential.helper store --file /Users/tester/.vendir-tmp/incoming/git-auth/.git-credentials
--> git remote add origin https://github.com/keycloak/keycloak-operator
--> git fetch origin
From https://github.com/keycloak/keycloak-operator
* [new branch] 10.0.x -> origin/10.0.x
* [new branch] latest -> origin/latest
* [new branch] master -> origin/master
* [new tag] 10.0.1 -> 10.0.1
* [new tag] 7.0.1 -> 7.0.1
* [new tag] 10.0.0 -> 10.0.0
* [new tag] 8.0.2 -> 8.0.2
--> git -c advice.detachedHead=false checkout 12.0.2
error: pathspec '12.0.2' did not match any file(s) known to git
Error: Syncing directory 'base': Syncing directory '.' with git contents: Fetching git repository: Git [-c advice.detachedHead=false checkout 12.0.2]: exit status 1 (stderr: error: pathspec '12.0.2' did not match any file(s) known to git
)
make: *** [base] Error 1
What happened:
The command git fetch
doesn't seem to fetch all tags for the keycloak-operator repo.
What did you expect:
I would expect that all existing tags on upstream could be resolved.
Anything else you would like to add:
I've tried to execute the git commands manually like so:
mkdir keycloak-operator
cd keycloak-operator
git init
git remote add origin https://github.com/keycloak/keycloak-operator
git fetch origin
remote: Enumerating objects: 127, done.
remote: Counting objects: 100% (127/127), done.
remote: Compressing objects: 100% (91/91), done.
remote: Total 10348 (delta 64), reused 65 (delta 32), pack-reused 10221
Receiving objects: 100% (10348/10348), 9.74 MiB | 4.19 MiB/s, done.
Resolving deltas: 100% (5474/5474), done.
From https://github.com/keycloak/keycloak-operator
* [new branch] 10.0.x -> origin/10.0.x
* [new branch] latest -> origin/latest
* [new branch] master -> origin/master
* [new tag] 10.0.1 -> 10.0.1
* [new tag] 7.0.1 -> 7.0.1
* [new tag] 10.0.0 -> 10.0.0
* [new tag] 8.0.2 -> 8.0.2
I get the same behavior as if I use vendir. The 12.0.2 tag doesn't get fetched. I was able to get all tags by using the --tags
flag:
git fetch origin --tags
remote: Enumerating objects: 1954, done.
remote: Counting objects: 100% (1330/1330), done.
remote: Compressing objects: 100% (251/251), done.
remote: Total 771 (delta 583), reused 669 (delta 483), pack-reused 0
Receiving objects: 100% (771/771), 211.08 KiB | 1.09 MiB/s, done.
Resolving deltas: 100% (583/583), completed with 158 local objects.
From https://github.com/keycloak/keycloak-operator
* [new tag] 11.0.0 -> 11.0.0
* [new tag] 11.0.1 -> 11.0.1
* [new tag] 11.0.2 -> 11.0.2
* [new tag] 11.0.3 -> 11.0.3
* [new tag] 12.0.0 -> 12.0.0
* [new tag] 12.0.1 -> 12.0.1
* [new tag] 12.0.2 -> 12.0.2
* [new tag] 8.0.1 -> 8.0.1
* [new tag] 9.0.0 -> 9.0.0
* [new tag] 9.0.2 -> 9.0.2
Environment:
vendir --version
): 0.13.0/etc/os-release
): Darwin (MacOS BigSur)vendir currently supports tracking specific refs and versions. I'd like to have support for specific version ranges:
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
- path: vendor
contents:
- path: github.com/cloudfoundry/cf-k8s-networking
git:
url: https://github.com/cloudfoundry/cf-k8s-networking
- ref: master
+ version: ^1.0.0
includePaths:
- cfroutesync/crds/**/*
- install/ytt/networking/**/*
- path: github.com/cloudfoundry/cc
manual: {}
- path: github.com/GoogleCloudPlatform/metacontroller
manual: {}
- path: local-dir
directory:
path: local-dir
So I can specify valid version ranges for each one of my dependencies.
What steps did you take:
I attempted to run vendir sync
with a githubRelease
for open-policy-agent/gatekeeper
. The entry under contents:
looks like:
- path: gatekeeper
githubRelease:
slug: open-policy-agent/gatekeeper
latest: true
disableAutoChecksumValidation: true
What happened:
I get an error that reads Error: Syncing directory 'config/upstream': Filtering paths in directory 'gatekeeper': Expected to find at least one file within directory.
This error is very cryptic. I spent a few troubleshooting cycles creating empty files in config/upstream/gatekeeper
or deleting the directory, thinking that the error message was complaining about the local filesystem.
Reading the vendir
code, the problem seems to be that thereβs no particular asset tied to the release - this project doesnβt provide binaries with their releases. You can download the source code, but thatβs not listed in the github API json response as part of the assets array.
the release in question: https://api.github.com/repos/open-policy-agent/gatekeeper/releases/37006810
...
"assets": [
],
"tarball_url": "https://api.github.com/repos/open-policy-agent/gatekeeper/tarball/v3.3.0",
"zipball_url": "https://api.github.com/repos/open-policy-agent/gatekeeper/zipball/v3.3.0",
...
https://github.com/vmware-tanzu/carvel-vendir/blob/1da965479755f7b3583e275d337e66f6d7a25950/pkg/vendir/fetch/githubrelease/sync.go#L76 refers specifically to the assets
array, and loops over that array, looking for assets to copy. Finding none, it errors out, apparently at https://github.com/vmware-tanzu/carvel-vendir/blob/bf0685a51eae291131fe5d1475ac0a6eceeb3c1e/pkg/vendir/directory/directory.go#L160
What did you expect:
I expected to be able to download the source code, since the Github UI shows it as an available download, and only after looking at the implementation did I understand why the vendir
githubResource
was not able to download it. If my project chooses to use vendir
to track upstream OSS dependencies via githubResources
, we probably want to track them uniformly, even if they don't ship any specific assets.
I would expect to be able to choose the tarball_url
or zipball_url
as options to download, possibly by specifying their paths as they appear when you download them via the github UI: https://github.com/open-policy-agent/gatekeeper/archive/v3.3.0.zip
and https://github.com/open-policy-agent/gatekeeper/archive/v3.3.0.tar.gz
Paths within the sourcecode for the release would also be nice, since in that case we'd probably only need metadata from a small number of files within the release.
Environment:
vendir --version
): 0.16.0/etc/os-release
): MacOSToday on cf-for-k8s, when I vendir sync --use-directory
to replace some remote code with local dev copies, it takes 30s to redownload all the dependencies. I really only need to move the files from the one directory. Could this be made faster?
What steps did you take:
I am using the carvel suite with helm on windows. Any helm chart should be able to reproduce the issue.
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
- path: vendor
contents:
- path: kube-prometheus
helmChart:
name: kube-prometheus
version: 5.0.0
repository:
url: https://charts.bitnami.com/bitnami
What happened:
Error: Syncing directory 'vendor': Syncing directory 'kube-prometheus' with helm chart contents: Fetching helm chart: exit status 1 (stderr: Error: failed to untar: mkdir C:\WINDOWS\helm-741980851: Access is denied.)
What did you expect:
It should use the .vendir-tmp
dir relative to the current dir or use ~/.cache/vendir
or event /tmp
or C:\Users\{user}\AppData\Local\Temp
Environment:
vendir --version
): v0.19.0/etc/os-release
): Windows 10Vote on this request
This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.
π "I would like to see this addressed as soon as possible"
π "There are other more important things to focus on right now"
We are also happy to receive and review Pull Requests if you want to help working on this issue.
When having:
- path: k8s/config
contents:
- path: A
git:
url: https://github.com/config/A
ref: master
- path: B
git:
url: https://github.com/config/B
ref: master
- path: k8s/examples
contents:
- path: A
git:
url: https://github.com/example/A
ref: master
- path: B
git:
url: https://github.com/example/B
ref: master
- path: C
git:
url: https://github.com/example/C
ref: master
If one wants to update only examples, it would need to do it one by one with vendir sync -d k8s/examples/A
then B and then C. It would be ideal to support the whole tree under examples when done via vendir sync -d k8s/examples
.
And linked to #11 It would be ideal to support this use case for -d that doesn't save vendir.lock file as well as with an option to save the vendir.lock file.
vendir
's features allow a lot of things, not only around kubernertes. It could be interesting to show how vendir
can make things easier, and to make package software installation even smoother.
vendir
can make APOC plugin installation easier in Neo4J (shox how to move away manual tasks)What steps did you take:
- path: terraform.d/plugins
contents:
- path: carvel.dev/carvel/k14s/0.6.0
githubRelease:
slug: vmware-tanzu/terraform-provider-carvel
tag: v0.6.0
assetNames: ["terraform-provider-k14s-binaries.tgz"]
unpackArchive:
path: terraform-provider-k14s-binaries.tgz
vendir sync
What happened:
The extracted package binaries will be set as non-executable even though in the archive they are executable.
What did you expect:
The file permissions should be left alone and they should be executable.
Anything else you would like to add:
This is very important for downloading of CLI tools
Environment:
vendir --version
): 0.13.0/etc/os-release
): Ubuntu 20.04Describe the problem/challenge you have
I would like to relocate a Helm chart that is stored in an experimental OCI image
Additional troubleshooting: https://kubernetes.slack.com/archives/CH8KCCKA5/p1618514640321100
Describe the solution you'd like
The helm chart is inside the container image, but it has an invalid rootfs so is failing on image pull, no preference on whether that means it's helmChart
imgpkgBundle
image
or even http
Anything else you would like to add:
What steps did you take:
When you add more than 1 path to directories using the http plugin you get these errors on windows.
---
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
- path: vendor/one
contents:
- path: one
http:
url: https://raw.githubusercontent.com/vmware-tanzu/carvel-vendir/develop/examples/http/vendir.yml
- path: vendor/two
contents:
- path: two
http:
url: https://raw.githubusercontent.com/vmware-tanzu/carvel-vendir/develop/examples/http/vendir.yml
I suspect that the tmp dir is not unique and/or is shared with both downloads that run in some goroutines in parallel but then they try to delete the dir (or parent dir).
What happened:
Error: Syncing directory 'vendor/two': Deleting tmp dir '.vendir-tmp': remove .vendir-tmp\incoming\vendir-http521295291: The process cannot access the file because it is being used by another process.
Environment:
vendir --version
): v0.19.0/etc/os-release
): Windows 10Vote on this request
This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.
π "I would like to see this addressed as soon as possible"
π "There are other more important things to focus on right now"
We are also happy to receive and review Pull Requests if you want to help working on this issue.
Foo
Looks like there's a bug getting the checksum of carvel-vendir due to a file name aux.go
.
β― go version
go version go1.13.15 linux/amd64
β― go get github.com/vmware-tanzu/carvel-vendir
go: downloading github.com/vmware-tanzu/carvel-vendir v0.14.0
verifying github.com/vmware-tanzu/[email protected]: github.com/vmware-tanzu/[email protected]: reading https://sum.golang.org/lookup/github.com/vmware-tanzu/[email protected]: 410 Gone
β― curl https://sum.golang.org/lookup/github.com/vmware-tanzu/[email protected]
not found: create zip: malformed file path "pkg/vendir/config/aux.go": "aux" disallowed as path element component on Windows
Describe the problem/challenge you have
Currently, semvers are not being sorted by their build metadata, which leaves some room for non determinism of selected versions.
Describe the solution you'd like
Sort the semvers on build metadata. Although the spec says not to rely on it, we should sort build metadata the same way as pre-releases. Spec
Vote on this request
This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.
π "I would like to see this addressed as soon as possible"
π "There are other more important things to focus on right now"
We are also happy to receive and review Pull Requests if you want to help working on this issue.
I'm really looking forward to testing this out for use with retrieving manifests, code, and github releases from sources and syncing them. I think there's tons of potential for declarative configuration from this. Will test it out more. Thanks a lot for all you do π not an actual issue just wanted to say thanks
for example:
Describe the problem/challenge you have
Assets in Github releases often include the release version as part of the asset's filename, but vendir
's unpackArchive
option does not currently support globbing, although the githubRelease.assetNames
array does support globbing. Lack of globbing support means that unpackArchive
cannot be used as intended for many Github Releases.
Currently, attempting to pass a glob results in a cryptic "Expected known archive type (zip, tgz, tar)" error message.
For example,
this works, but you have to know the filename in advance:
- path: prometheus
githubRelease:
slug: prometheus/prometheus
latest: true
disableAutoChecksumValidation: true
assetNames: ["prometheus-*.linux-amd64.tar.gz"]
unpackArchive:
path: 'prometheus-2.25.0.linux-amd64.tar.gz'
this fails:
- path: prometheus
githubRelease:
slug: prometheus/prometheus
latest: true
disableAutoChecksumValidation: true
assetNames: ["prometheus-*.linux-amd64.tar.gz"]
unpackArchive:
path: 'prometheus-*.linux-amd64.tar.gz'
βError: Syncing directory βconfig/upstreamβ: Syncing directory βprometheusβ with github release contents: Expected known archive type (zip, tgz, tar)β
Describe the solution you'd like
Globbing should be supported. If the user-provided glob matches multiple files, an error message would be acceptable, as would be unpacking all of the specified archives.
Hi, I use the release tweet to communicate on the choco package availabiliy, but this time I just can't find the related tweet : can you do one as usual please ?
Thanks in advance for your help.
Friendly yours,
Adrien
What steps did you take:
vendir-windows-amd64.exe
binary from https://github.com/vmware-tanzu/carvel-vendir/releases/tag/v0.19.0vendir-windows-amd64.exe version
0.18.0
(instead of 0.19.0
)What happened:
0.18.0
instead of 0.19.0
What did you expect:
0.19.0
Environment:
/etc/os-release
): Windows
Vote on this request
This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.
π "I would like to see this addressed as soon as possible"
π "There are other more important things to focus on right now"
We are also happy to receive and review Pull Requests if you want to help working on this issue.
similar to manual but points to a local directory. copies over files from there and filters out content per usual includePaths/excludePaths rules.
Describe the problem/challenge you have
If i have a recursive bundle, I would like to be able to pull it recursively
Describe the solution you'd like
Expose a recursive: true/false
option to the imgpkgBundle fetch config
Anything else you would like to add:
Eventually, we may want to also support a filter here using proposed filtering behavior in imgpkg
Vote on this request
This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.
π "I would like to see this addressed as soon as possible"
π "There are other more important things to focus on right now"
We are also happy to receive and review Pull Requests if you want to help working on this issue.
Tests are failing due to the removal of https://kubernetes-charts.storage.googleapis.com/.
The url is hardcoded in vendir here.
Many people create yaml files with either yml
or yaml
extensions. Vendir should look for both extensions (prioritizing maybe yml), so that it won't fail when the user creates a vendir.yaml
file.
If vendir sync
fails, manually managed directories might be deleted.
There are two paths in the config: 1) manually managed lib/manifests
, 2) Non-existing git
lib/bin
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
- path: lib
contents:
- path: manifests
manual: {}
- path: bin
git:
url: https://github.com/k14s/vendir
ref: origin/master
includePaths:
- bin/non-existing
newRootPath: bin
Dir structure before sync
:
.
βββ lib
β βββ manifests
β βββ tpl.yaml
βββ vendir.yml
After sync
:
.
βββ lib
βββ vendir.yml
As far as I understand, vendir
processes paths in the order they're defined in the configuration. For manually managed paths it moves them into temporary space. If one of the next paths can't be ensured due some error, the path previously moved to the temporary space is not moved back.
If I change the order of the paths, lib/manifests
directory stay intact.
It looks like a bug and might lead to data loss (files and staged changes can be restored from git or from IDE history, but still).
Describe the problem/challenge you have
When the target is a zip file, vendir automatically unzip it.
It would be nice to introduce a flag to disable this behavior.
Describe the solution you'd like
Example:
- path: artifacts/plugins/myplugin.zip
http:
url: https://artifacts.example.com/downloads/plugins/myplugin.zip
unpackArchive: false
Anything else you would like to add:
[Additional information that will assist in solving the issue.]
When using this config:
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
minimumRequiredVersion: 0.8.0
directories:
- path: k8s/eduk8s
contents:
- path: upstream/eduk8s
git:
url: https://github.com/eduk8s/eduk8s
ref: 20.08.04.1
includePaths:
- resources/crds-v1/**/*
- resources/operator/**/*
- path: downstream/hub
manual: {}
I would expect to end up with this folder structure:
k8s/eduk8s/upstream/eduk8s
k8s/eduk8s/downstream/hub
I understand the second needs to exist, since it's a manual sync, but the former is not and the resources specified in that rule end up in k8s/eduk8s/content/...
rather than k8s/eduk8s/upstream/eduk8s/content/...
.
I have seen the test https://github.com/k14s/vendir/blob/develop/examples/overlapping-dir/vendir.yml which confuses me as it seems to be the expected behavior. What am I missing?
eg show that it's working on particular directory (and may be stream output from git)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.