containers / image Goto Github PK
View Code? Open in Web Editor NEWWork with containers' images
License: Apache License 2.0
Work with containers' images
License: Apache License 2.0
Use case: I have built a local docker image. I need a way to reference this image to push it to a registry and sign it. We need to be able to reference local docker images to copy them to a registry and generate a manifest to sign without changing the name of the image.
localdocker:// is just a suggestion. Don't care what we call it.
After policy evaluation is merged (but filing now as a reminder to myself), signature verification in policy evaluation, with its temporary directory creation, will take about a second per case. Both general performance and test suite interactivity would greatly benefit from speeding this up.
At the very least we can reuse the temporary directories across all verifications with a single policy; that’s why PolicyContext
exists in the first place. Also look into speeding up the verification as such, there might be a cheap optimization somewhere.
we need something to exercise the various source and destination somehow (except directory) here in containers/image
In
tagged, ok := r.(reference.NamedTagged)
if !ok {
return nil, fmt.Errorf("invalid image reference %s, %#v", ref, r)
}
and
r := strings.SplitN(dockerRef.RemoteName(), "/", 3)
if len(r) != 2 {
return nil, fmt.Errorf("invalid image reference %s", dockerRef.String())
}
we know fairly precisely what we want and what we aren’t getting, so we should tell the users what exactly they need to do to have the input accepted, instead of vaguely rejecting the input as “invalid”.
Currently master
fails with this error:
? github.com/containers/image [no test files]
ok github.com/containers/image/copy 0.052s coverage: 14.9% of statements
ok github.com/containers/image/directory 0.052s coverage: 78.1% of statements
ok github.com/containers/image/directory/explicitfilepath 0.011s coverage: 85.7% of statements
ok github.com/containers/image/docker 4.481s coverage: 47.1% of statements
ok github.com/containers/image/docker/daemon 0.008s coverage: 10.5% of statements
ok github.com/containers/image/docker/policyconfiguration 0.010s coverage: 94.7% of statements
--- FAIL: TestParseReferenceWithTagAndDigest (0.00s)
reference_test.go:248: Error parsing reference: "busybox:latest@sha256:86e0e091d0da6bde2456dbb48306f3956bbeb2eae1b5b9a43045843f69fe4aaa" is not a valid repository/tag
FAIL
coverage: 69.9% of statements
FAIL github.com/containers/image/docker/reference 0.005s
ok github.com/containers/image/image 0.039s coverage: 47.7% of statements
ok github.com/containers/image/manifest 0.065s coverage: 90.6% of statements
? github.com/containers/image/oci [no test files]
ok github.com/containers/image/oci/layout 0.011s coverage: 45.4% of statements
ok github.com/containers/image/openshift 0.005s coverage: 4.0% of statements
ok github.com/containers/image/signature 38.552s coverage: 94.2% of statements
ok github.com/containers/image/storage 0.011s coverage: 28.8% of statements
ok github.com/containers/image/transports 0.007s coverage: 92.9% of statements
? github.com/containers/image/types [no test files]
? github.com/containers/image/version [no test files]
make: *** [Makefile:18: test] Error 1
make test 11.77s user 2.23s system 33% cpu 41.214 total
However this error only happens if you update upstream to the latest version (go get -t
). So I think this is actually some upstream change that we aren't handling properly. Do we want to vendor this code, or otherwise integrate it here? I'm not sure, but something is definitely broken.
Right now if you import containers/image in docker you'll get tons of dependencies because of the openshift transport where instead docker only needs the docker transport.
There could be a way to setup transports by the caller instead of hardcoding them into https://github.com/containers/image/blob/master/transports/transports.go#L23
@mtrmac wdyt?
Right now we silently replace it with a digest, which is rather useless. This will need extra logic for v2s1 to either reject pushes with a different tag, or to regenerate the manifest.
The scenario is the following (requires interacting with Docker):
runcom/busybox:signed
runcom/busybox:signed
via containers/image
containers/image
provides the trusted digest for runcom/busybox:signed
docker pull runcom/busybox@sha256:%HEX%
trust-plugin
on the docker API side ...trust-plugin
check that a valid signature exists for runcom/busybox@sha256:%HEX%
trust-plugin
fails because we don't have signature for runcom/busybox@sha256:%HEX%
but for runcom/busybox:signed
(which is one of the tag for that digest, which should be already trusted)I'm proposing something to relax this since if you're pulling by digest at some time it's likely that you have already checked for signatures from a tag.
If you apply #169, you still get an error. This time it's because of the manifest:
% ../skopeo/skopeo --debug copy oci:opensuse:new docker-daemon:opensuse/amd64:latest
DEBU[0000] IsRunningImageAllowed for image oci:/home/cyphar/src/umoci/opensuse:new
DEBU[0000] Using default policy section
DEBU[0000] Requirement 0: allowed
DEBU[0000] Overall: allowed
Getting image source signatures
DEBU[0000] Will convert manifest from MIME type application/vnd.oci.image.manifest.v1+json to application/vnd.docker.distribution.manifest.v2+json
Copying blob sha256:467db25190688bc9dc1d5fd6dbced1ac56f55d93a942987f448e62ad2614e46e
DEBU[0000] Detected compression format gzip
0 B / 46.97 MB [--------------------------------------------------------------]DEBU[0000] Using original blob without modification
DEBU[0000] Sending as tar file sha256:467db25190688bc9dc1d5fd6dbced1ac56f55d93a942987f448e62ad2614e46e
27.81 MB / 46.97 MB [=================================>-----------------------]
Copying blob sha256:eb9108dbdabef43ae694b955990153159c1cbd069c3e4ca5ac9ccdbdda0e4b09
DEBU[0000] Detected compression format gzip
0 B / 23.48 MB [--------------------------------------------------------------]DEBU[0000] Using original blob without modification
DEBU[0000] Sending as tar file sha256:eb9108dbdabef43ae694b955990153159c1cbd069c3e4ca5ac9ccdbdda0e4b09
46.97 MB / 46.97 MB [=========================================================]
Copying config sha256:a04dcb512d1d9fdd58667c4b2aaec916418f51130f9d8a9dd5b2dbed066e8c62
DEBU[0000] No compression detected
0 B / 1003 B [----------------------------------------------------------------]DEBU[0000] Using original blob without modification
DEBU[0000] Sending as tar file sha256:a04dcb512d1d9fdd58667c4b2aaec916418f51130f9d8a9dd5b2dbed066e8c62
Writing manifest to image destination
the manifest: [[{"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json","config":{"mediaType":"application/vnd.oci.image.config.v1+json","size":1003,"digest":"sha256:a04dcb512d1d9fdd58667c4b2aaec916418f51130f9d8a9dd5b2dbed066e8c62"},"layers":[{"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip","size":49248775,"digest":"sha256:467db25190688bc9dc1d5fd6dbced1ac56f55d93a942987f448e62ad2614e46e"},{"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip","size":24623217,"digest":"sha256:eb9108dbdabef43ae694b955990153159c1cbd069c3e4ca5ac9ccdbdda0e4b09"}]}]]
DEBU[0000] docker-daemon: Closing tar stream to abort loading
FATA[0000] Error writing manifest: Unsupported manifest type, need a Docker schema 2 manifest
I added "the manifest: [[]]" for my own debugging purposes. It's clear that copy/copy.go
never does a conversion. And the reason for this is that the OCI
transport doesn't implement types.Image
(specifically it doesn't implement UpdatedImage
which is necessary).
Since the two specs' manifests (Docker schema 2 and OCI) have the same schema but just a different mediatype the implementation should be very simple (it just needs to update the mimetype in the manifest).
EDIT: This can be probably done in skopeo
If you try to use skopeo to push an OCI image to a Docker registry you get the weird error (this was done after copying the current master
of containers/image
to the vendor/
of `skopeo):
% skopeo copy oci:opensuse:new docker://cyphar/opensuse_playground:latest
Getting image source signatures
Copying blob sha256:467db25190688bc9dc1d5fd6dbced1ac56f55d93a942987f448e62ad2614e46e
46.97 MB / 46.97 MB [=========================================================]k
Copying blob sha256:eb9108dbdabef43ae694b955990153159c1cbd069c3e4ca5ac9ccdbdda0e4b09
23.48 MB / 23.48 MB [=========================================================]
FATA[1275] Error creating an updated image manifest: Conversion of image manifest from application/vnd.docker.distribution.manifest.v2+json to application/vnd.docker.distribution.manifest.v2+json is not implemented
So it's failing to convert from itself to itself?
This is a bit of a weird issue, but it's something that is exposed if you apply #148. Effectively because docker-daemon
(and thus docker/tarfile
) return the set of DiffIDs
as the LayerInfos
for a particular schema, you end up with GetBlob(diffid)
giving you the wrong hash.
I'm not really sure how we should handle this, because I don't think the docker save
-style archives don't contain enough nicely-formatted information about the compressed layer digests for us to be able to switch. Should we just un-gzip
layers?
See all the FIXMEs
in there, and in transports/transports_test.go
. And probably add some tests.
copy
semantic in current implementation, all data(include manifest and image layer) in destination will be overwrited by source without to see if it exist.
IMO we should implement sync
semantic between source and destination image repository, if layer in destination is exist, it won't be downloaded repeatedly. This will be more efficient.
We need to extend our custom client to sign images from inside a build pod:
/var/run/secrets/kubernetes.io/serviceaccount/token
/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
/var/run/secrets/kubernetes.io/serviceaccount/namespace
Our openshift client is a custom client. It would be much better to use the upstream k8s client. This shouldn't significantly increase the size of the project.
From a source perspective, this would be implemented as selecting what RepoTag
to extract layers from. And from a destination perspective, it would be implemented by changing what the resultant RepoTag
s already present at a destination are.
Essentially all of this would be controlled by choosing a full reference when creating references that use docker.tarfile
.
Following a chat between @rhatdan and @alexlarsson in #atomic on Freenode:
10:27 alexlarsson : skopeo copy oci:oci-dir oci:other-dir
10:27 alexlarsson : FATA[0000] Error initializing source oci:oci-dir:latest: Reading images not implemented for oci: image names
10:27 alexlarsson : Not much support for oci, is there?
10:28 dwalsh_ : runcom|afk, ^^
10:28 alexlarsson : I changed the flatpak oci code to export a directory following the oci image spec layout
10:28 alexlarsson : do we have anything that can read that?
10:29 alexlarsson : To like, test it or something :)
10:29 dwalsh_ : alexlarsson, I thought skopeo had grown support for it.
10:29 dwalsh_ : We need to wait for runcom|afk to come back.
What you need is the ability to have an OCI source, probably remote? well, since OCI hasn't "distribution" in scope that's simply not possible right now (there's work on distribution/distribution#2021)
… per #63 (comment) . That PR did add an imageCopier.copiedBlobs
but it has no uses.
Hello,
In GrootFS [1], we are caching the blobs and image config when downloading remote (Docker) images. The only thing we can't cache at the moment is the manifest, since ImageSource.GetManifest
is fetching the whole manifest regardless of whether or not is already cached.
The Docker registry v2 API supports Docker-Content-Digest
header [2], which contains the manifest digest. We would like to extend the ImageSource
interface by adding a GetManifestDigest
method which uses HTTP HEAD to just fetch the digest and MIME type. That way we will be able to use our cache if we have the most recent manifest.
diff --git a/types/types.go b/types/types.go
index b67e6b3..669f579 100644
--- a/types/types.go
+++ b/types/types.go
@@ -85,6 +85,9 @@ type ImageSource interface {
// Reference returns the reference used to set up this source, _as specified by the user_
// (not as the image itself, or its underlying storage, claims). This can be used e.g. to determine which public keys are trusted for this image.
Reference() ImageReference
+ // GetManifestDigest returns the image's manifest digest along with its MIME type. The empty string is returned if the MIME type is unknown. The slice parameter indicates the supported mime types the manifest should be when getting it.
+ // You may use this operation to check if you have the most recent manifest in cache. It may use a remote (= slow) service.
+ GetManifestDigest([]string) (string, string, error)
// GetManifest returns the image's manifest along with its MIME type. The empty string is returned if the MIME type is unknown. The slice parameter indicates the supported mime types the manifest should be when getting it.
// It may use a remote (= slow) service.
GetManifest([]string) ([]byte, string, error)
--
We would like to PR this work, but before we would like to know WDYT?
Thanks,
@albertoleal and @glestaris
[1] https://github.com/cloudfoundry/grootfs
[2] https://github.com/docker/distribution/blob/master/docs/spec/api.md#digest-header
dockerImageDestination.SupportedManifestMIMETypes
returns a hard-coded value; if the destination registry does not in fact support schema 2, we would still attempt to upload this.
This incorrect information will break any manifest conversion we might attempt.
I'm confused why the sigstore configuration is not part of policy.json data structure. These things naturally go together. Our registries.d scheme seems to unnecessarily complicate matters.
Example:
{
"transports": {
"docker": {
"docker.io/library/busybox": {
"type": "signedBy",
"keyType": "GPGKeys",
"keyPath": "/path/to/official-pubkey.gpg",
"sigstore": "https://sigs.example.com"
}
}
}
}
Right now they are only mentioned in policy.json.md
as far as necessary to write the policy. We should document the naming syntax, the protocols, the configuration options etc…
In the signature
package, the mechanism
source uses a cgo wrapper for GnuPG Made Easy library: github.com/mtrmac/gpgme
. Is there a reason why this was chosen over pure Go's native OpenPGP package?
Thanks.
Now that we have (AFAICT) abandoned Go 1.6 and require Go 1.7, we probably can use the stdlib-included one.
At least covering the success case for manifest conversion, like docker_schema2_test.go
Anything against having {Image,ImageSource,ImageDestination}.Close()
implement io.Closer
https://golang.org/pkg/io/#Closer? (which basically is adding an error return). I think we would have this for #63 also.
ref: https://github.com/containers/image/blob/master/types/types.go#L98
@mtrmac wdyt?
Namely, the docker load
-friendly format. This would make it possible to use skopeo
to have a full OCI build system that will then pipe directly into a Docker image without needing a Docker registry or Docker daemon on the build system. In particular this would be very handy for implementing the Open Build Service support that we're planning at SUSE (we can't easily run a Docker registry as part of our infrastructure).
Probably blocking projectatomic/docker#200 but not really since we never thought well about this scenario.
Right now in docker/docker you can pull an image which its manifest isn't a single manifest (with a digest) but it's a list of manifests.
@mtrmac is this a blocking issue? should we trust all sub-manifests in a manifest list if the manifest list it's signed?
For manifest list (and following the idea from #99) we could have another field in signedBy with: "manifestList": {prohibit, trustList, trustTarged} and by default is trustList (or is it too open?)
Per implementation (also doc #120) the manifest digest reference for the "docker" transport type is HASH_FUNCTION:HASH_VALUE, using a colon (:) delimiter. The "at" symbol (@) is also used the delimit image and manifest digest (again, see #120). Per URI: Generic Syntax these are reserved delimiting characters.
Impact
The colon in the URI is blocking uploading of signatures to JFrog's Artifactory repository management platform, returning 500 error:
"Invalid path. ':' is not a valid name character: atomic-sigstore/aweiteka/true@sha256:f292b8573b1679a512f575d7bc2441815b7528eb114217781199e9106e742e21/"
Percent-encoding the path (%3A
) does not help. Artifactory has not responded to this related issue
RepoTags
in directory transport miessed:
# skopeo inspect dir:busybox-dir
{
"Digest": "sha256:29f5d56d12684887bdfa50dcd29fc31eea4aaf4ad3bec43daf19026a7ce69912",
"RepoTags": [],
"Created": "2016-10-07T21:03:58.469866982Z",
"DockerVersion": "1.12.1",
"Labels": {},
"Architecture": "amd64",
"Os": "linux",
"Layers": [
"sha256:56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190"
]
}
When ~/.kube/config has multiple contexts configured sometimes the incorrect context is used. See debug output: https://paste.fedoraproject.org/429048/47404610/
Version: issue reproduced using skopeo version 0.1.14-dev commit: 02b9f8b479fee04c6aa0c5038ec07612853b5e67
cc @mtrmac
We have genericManifest
which has Config()
in which contains, among other things needed for ImageInspectInfo
, some runtime information about an image. An example of where this is needed is appc/docker2aci
project.
Question: should we be able to provide such information to users?
Most of the times this is what users expect:
User string `json:"User"`
Memory int `json:"Memory"`
MemorySwap int `json:"MemorySwap"`
CpuShares int `json:"CpuShares"`
ExposedPorts map[string]struct{} `json:"ExposedPorts"`
Env []string `json:"Env"`
Entrypoint []string `json:"Entrypoint"`
Cmd []string `json:"Cmd"`
Volumes map[string]struct{} `json:"Volumes"`
WorkingDir string `json:"WorkingDir"`
The other alternative is of course to not expose it and have the docker -> appc conversion here in containers/image which internally uses the Config()
from genericManifest
to build an ACI for instance.
OTOH we expose the whole manifest so probably this won't be needed and ppl will go around and get the config directly from the manifest perhaps.
… otherwise a MITM can prevent use of TLS by blocking connections, and then on unauthenticated HTTP they can send a custom WWW-Authenticate: Bearer realm=$attacked-controlled-URL
"` header which will cause the user name and password to be sent to the attacker-controlled location.
was contacted today because people could not figure out the right syntax for skopeo inspect
and they were trying skopeo inspect fedora
.
would be nice to provide better help messages in all commands (not just inspect
) and tell ppl about our image prefixies e.g. docker://
, atomic:
, oci:
etc
/cc @TomasTomecek @mtrmac
Hello,
containers/image imports docker/docker
for the reference
package which is used for types of for reference parsing [1]. docker/docker
is a quite big repository and would prefer to avoid having it as a dependency in GrootFS [2], which consumes containers/image
.
Would it be possible to use docker/distribution
instead for the reference types/parsing? Additionally, the Docker transport package seems to be doing all the requests to a Docker registry [3]. Could this behavior be delegated to docker/distribution
?
Thanks!
[1]
image/docker/docker_transport.go
Line 47 in 0c795ea
image/docker/docker_image_src.go
Line 71 in 0c795ea
Outstanding comments from #160:
ServerDefault
tls.Config
in a client?tls.Config
for our purposes?tlsclientconfig.NewTransport
, the proxyDialer
setup should happen before we create a http.Transport
, so that we don’t need to override tr.Dial
in a created object.Hi, does is make sense to add more detail about this project to the README. I want to know more about this project but hardly find a docs to begin with.
First question in my mind, what's the different with the OCI/image-tool?
Sorry I don't want to bother you guys @philips @runcom but I find you are the only members of github.com/container.
Cross posting from containers/skopeo#110
From kubernetes/kubernetes#26788 (comment)
Would be nice to support such a thing in skopeo itself as part of our types
If you do something like:
% skopeo copy docker-daemon:opensuse/amd64:tumbleweed oci:opensuse:latest
Getting image source signatures
Copying blob sha256:bed80e2cbdd7d22d83fb2b743713df956c59d5edbfded661eda52e346ceed553
150.50 MB / 150.50 MB [=======================================================]
Copying config sha256:0bd5203d80dd7795a2eaf8873f60f50b7f916ad05a538db489c4e60190e7121c
1.73 KB / 1.73 KB [===========================================================]
Writing manifest to image destination
Storing signatures
% file opensuse/blobs/sha256/*
opensuse/blobs/sha256/0bd5203d80dd7795a2eaf8873f60f50b7f916ad05a538db489c4e60190e7121c: ASCII text, with very long lines, with no line terminators
opensuse/blobs/sha256/527ad855882abe493134cc4ffb282c020e13bb94a08f708003781d11d6eb5130: ASCII text, with very long lines, with no line terminators
opensuse/blobs/sha256/bed80e2cbdd7d22d83fb2b743713df956c59d5edbfded661eda52e346ceed553: POSIX tar archive
Which is wrong.
Creating as a TODO (for myself mainly)
We need to fix OCI destination as per opencontainers/image-spec#208
I'm assigning this to myself and fix it.
Right now docker-daemon: destination just passes the input string as a tag in the stream. Docker seems to be doing some canonicalization, but it is unclear what exactly the rules are.
docker pull busybox
gets the image tagged as docker.io/busybox
(per docker inspect
)docker-daemon:busybox
gets it tagged as docker:busybox
docker-daemon:docker.io/library/busybox
gets it tagged as docker.io/busybox
docker inspect busybox:latest
finds docker.io/busybox:latest
docker inspect docker.io/busybox.latest
does not find busybox:latest
We need to figure out what is going on, document the relevant formats, and perhaps do some normalization in daemonReference
.
cc @baude
If we end up having another transport named docker-daemon:
(or whatever the naming will be) - we would move the implementation under docker/*
to be docker/registry/*
and have github.com/containers/image/docker/registry
package (and docker/daemon
ofc).
As discussed in more detail #195, the code in #191 is non-obvious, it neither documents why it is needed nor carries a test case; it is fairly risky that we will break it over time just because we don’t know what to maintain.
Basic
auth without checking the WWW-Authenticate
header contents at all is necessary; can’t we tell exactly that this is what the server needs? https://bugzilla.redhat.com/show_bug.cgi?id=1409873 (note, not the original bug motivating #191) mentions Unimplemented: WWW-Authenticate Bearer replaced by "basic"
, which suggests there may in fact be some data to base the decision on./v2/
ping results and assuming that the ping’s WWW-Authenticate
value is relevant for other URLs is unsound in general (as it seems to be, per #195), is there another cleaner approach we could take? E.g. pinging only to determine http/https (then we don’t need care about HTTP response codes at all), and reacting to WWW-Authenticate
for each individual URL separately? (The difficulty with this would be delaying a streamed blob upload until we know it would be accepted—we need the server to give us an “auth OK, go ahead” response.)So here is a completely horrible idea: As part of tests, perhaps in Makefile
and definitely in Travis, checkout skopeo
, replace the vendored copy with current working directory, and test that skopeo runs and its tests pass.
As I said, completely horrible. But perhaps worth it, at least until most of the transports in this repo grow some real tests. Merging containers/skopeo#102 and only a week later discovering that it breaks pushing to OpenShift would be pretty awkward…
WDY(all)T?
as rightly pointed our here moby/moby#26369 (comment) it would be nice to use docker/distribution/digest
type and functions instead of just string
. Anyone? 👼
Under package oci
there will be also an OCI image destination which will support fetching images (like Docker) from registries (one day...)
It makes sense to me to rename the functions now so we don't really break so many people (and since a lot is changing these days...)
@mtrmac WDYT?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.