Comments (4)
This is a rather critical feature for our users on Windows. Pulling in for this upcoming release.
from imgpkg.
This was released in v0.4.0.
from imgpkg.
Consolidating #71 into this issue.
Story
As a operator wishing to collect a bundle that includes software that includes non-distributable bits (e.g. Windows-based products) for use in an air-gapped scenario
I want to be able to override the default behavior of skipping transferring so-called non-distributable layers ("foreign layers" in the docker manifests, "non-distributable" in OCI manifests)
and I want the bits of those layers to be pulled and included in the tarball of the bundle
and I want those bits to be loaded into the registry I copy the tarballed bundle.
Context
Original GitHub Issue: imgpkg#8
Requested by the .NET Build team (Malini Valliath, PM) multiple times since this summer.
PCS/MAPBU leadership has pointed to the importance of servicing our Windows-using customers — Ajay pointed out the importance of Windows support during an all-hands meeting, noting that a significant percentage of our customers use Windows.
Acceptance Criteria
Note: we say, "foreign layers", here. We wish for all this to be true for any non-distributable layer.
🟢 Copying foreign layers to a tarball
Given a bundle which includes a reference to an image containing a foreign layer (e.g. a bundle that references Microsoft's nanoserver — mcr.microsoft.com/windows/nanoserver:1809-amd64
)
When I copy that bundle to a tarball
And I indicate that non-distributable layers (including foreign layers) should be included
$ imgpkg copy -b index.docker.io/k8slt/bundle-with-foreign-layers --to-tar the-football.tar --include-non-distributable
Then the tarball includes the contents of the layer
🟢 Copying a tarball containing one or more foreign layers to a registry
Given a tarball containing a bundle which includes one or more foreign layers
And a registry enabled to receive foreign layers (
When I copy the bundle to an image reference (i.e. a location in a registry)
And I indicate that non-distributable layers (including foreign layers) should be included
$ imgpkg copy --from-tar the-football.tar --to-repo gcr.io/cf-k8s-lifecycle-tooling-klt/bundle-with-foreign-layers --include-non-distributable
Then I can find that bundle in the registry AND its digest has not changed
And I can find those foreign layers in the registry
🟡 Warn when foreign layer from registry is skipped
Given a bundle which includes a reference to an image containing a non-distributable layer (e.g. a foreign layer)
When I copy the bundle with no additional instruction to a tarball
$ imgpkg copy -b <image ref to bundle> --to-tar the-football.tar
Then I see a warning next to each non-distributable layer that indicates "skipped: foreign layer" or "skipped: non-distributable" (depending on the exact media type), similar to how Docker does it
And I see a warning in the summary that notes that one or more layers are not in the tarball because they are "non-distributable/foreign layers" and a link to documentation on how to include them, if desired
And I see that the overall operation is "Successful (with warnings)."
And the tarball does NOT include those layers (neither in the tarball itself, nor in the manifest.json
)
🟡 Warn when foreign layer from tarball is skipped
Given a tarball containing a bundle which includes one or more foreign layers
When I copy the bundle to an image reference (i.e. a location in a registry), with no additional instruction
$ imgpkg copy --from-tar the-football.tar --to-repo gcr.io/cf-k8s-lifecycle-tooling-klt/bundle-with-foreign-layers
Then I see a warning next to each non-distributable layer that indicates "skipped: foreign layer" or "skipped: non-distributable" (depending on the exact media type), similar to how Docker does it
And I see a warning in the summary that notes that one or more layers are not in the tarball because they are "non-distributable/foreign layers" and a link to documentation on how to include them, if desired
And I see that the overall operation is "Successful (with caveats)."
And I can NOT find those foreign layers in the registry
📚 Documentation
Given I want to know more about how imgpkg
handles non-distributed/foreign layers
When I read the documentation around the copy
command
Then I discover/understand:
- what are non-distributable layers:
- how they are identified in both OCI manifests and Docker manifests) and
- what purpose they serve
- (all this with links to authoritative sources, balancing clarity with minimizing duplicating information)
- that by default
imgpkg
respects the non-distributable nature of foreign layers - that — as instructed by Microsoft — such layers can be copied
- known limitations. 'foreign layers' don't appear to work on dockerhub. validate this is true and add any other registries that
Implementation Notes
- on Docker Hub, "side-loading" non-distributable layers and later referencing them in a manifest does not seem to work. See the comments in this story for details.
Helpful Resources/Notes
- It appears that the GGCR code base does not support writing an image that contains 'non-distributable' image layers (please verify this for yourself). Micah has provided a proof of concept that shows a 'workaround' https://github.com/micahyoung/registry-foreign-layers-testing.git
- Docker Registry API: Pushing an Image — nicely details the step-by-step of how to push an image.
- Windows Containers FAQ: How Do I Make my Container Images Available on air-gapped Machines?
- There are two image manifest specs, (oci and docker). Both have a way of marking a layer as non-distributable.
- oci marks an image layer as non-distributable by using the media type:
application/vnd.oci.image.layer.nondistributable.v1.tar
orapplication/vnd.oci.image.layer.nondistributable.v1.tar+gzip
- docker marks an image layer as non-distributable by using the media type:
application/vnd.docker.image.rootfs.foreign.diff.tar.gzip
- oci marks an image layer as non-distributable by using the media type:
- this will likely need to work with artifactory, could reach out to Micah to see what they used
from imgpkg.
Task list:
- Improve error message when the following occurs: copying a repo to a tar without the
--include-non-distr
flag, and then copying from the tar to a repo with the--include-non-distr
flag should fail with a meaningful error message. currently it is failing with a cryptic error message - recursively iterate through image and imageindexes when printing a warning message to the user. https://github.com/vmware-tanzu/carvel-imgpkg/blob/caaacb7495bc4a9c518ac71aa42f3252db04b7b1/pkg/imgpkg/cmd/copy_repo_src.go#L171
from imgpkg.
Related Issues (20)
- Having the log implementation all internal causes friction when consuming imgpkg as a library
- Case when the registry link for pull is a image but the flags are -I along with —image-is-bundle-check flag=true which results in the image being pulled. HOT 5
- Are you using imgpkg?
- add ability to specify the select directories/files to be pulled/outputted. HOT 6
- Relocating bundle to Google Artifact Registry using `--repo-based-tags` fails HOT 1
- Sign `imgpkg` binaries while releasing them HOT 1
- imgpkg should emit plain text to be used in pipelines by default
- Add a limiter for pushing images HOT 3
- imgpkg miscalculates apache/nifi copy, produces corrupt image layer when copying apache/nifi in a bundle HOT 1
- Consider using OCI compliant media-types HOT 2
- imgpkg with flag --json swallows error
- Command `describe` does not include `metadata`. HOT 1
- imgpkg v0.40.0 breaks with relocated image HOT 2
- Imgpkg describe fails to print the output for the larger images HOT 1
- Update copyright headers
- Imgpkg describe fails to describe a bundle pushed to the repo
- Upgrade cosign version to latest version in GitHub workflows
- Cannot build for `GOARCH=386`
- Is there any way to disable progress bar shown during pushing of images ? HOT 5
- Bring tests against harbor back
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from imgpkg.