Right now if you import containers/image in docker you'll get tons of dependencies because of the openshift transport where instead docker only needs the docker transport.
There could be a way to setup transports by the caller instead of hardcoding them into https://github.com/containers/image/blob/master/transports/transports.go#L23

@mtrmac wdyt?

we need something to exercise the various source and destination somehow (except directory) here in containers/image

If you apply #169, you still get an error. This time it's because of the manifest:

% ../skopeo/skopeo --debug copy oci:opensuse:new docker-daemon:opensuse/amd64:latest
DEBU[0000] IsRunningImageAllowed for image oci:/home/cyphar/src/umoci/opensuse:new
DEBU[0000]  Using default policy section
DEBU[0000]  Requirement 0: allowed
DEBU[0000] Overall: allowed
Getting image source signatures
DEBU[0000] Will convert manifest from MIME type application/vnd.oci.image.manifest.v1+json to application/vnd.docker.distribution.manifest.v2+json
Copying blob sha256:467db25190688bc9dc1d5fd6dbced1ac56f55d93a942987f448e62ad2614e46e
DEBU[0000] Detected compression format gzip
 0 B / 46.97 MB [--------------------------------------------------------------]DEBU[0000] Using original blob without modification
DEBU[0000] Sending as tar file sha256:467db25190688bc9dc1d5fd6dbced1ac56f55d93a942987f448e62ad2614e46e
 27.81 MB / 46.97 MB [=================================>-----------------------]
Copying blob sha256:eb9108dbdabef43ae694b955990153159c1cbd069c3e4ca5ac9ccdbdda0e4b09
DEBU[0000] Detected compression format gzip
 0 B / 23.48 MB [--------------------------------------------------------------]DEBU[0000] Using original blob without modification
DEBU[0000] Sending as tar file sha256:eb9108dbdabef43ae694b955990153159c1cbd069c3e4ca5ac9ccdbdda0e4b09
 46.97 MB / 46.97 MB [=========================================================]
Copying config sha256:a04dcb512d1d9fdd58667c4b2aaec916418f51130f9d8a9dd5b2dbed066e8c62
DEBU[0000] No compression detected
 0 B / 1003 B [----------------------------------------------------------------]DEBU[0000] Using original blob without modification
DEBU[0000] Sending as tar file sha256:a04dcb512d1d9fdd58667c4b2aaec916418f51130f9d8a9dd5b2dbed066e8c62

Writing manifest to image destination
the manifest: [[{"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json","config":{"mediaType":"application/vnd.oci.image.config.v1+json","size":1003,"digest":"sha256:a04dcb512d1d9fdd58667c4b2aaec916418f51130f9d8a9dd5b2dbed066e8c62"},"layers":[{"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip","size":49248775,"digest":"sha256:467db25190688bc9dc1d5fd6dbced1ac56f55d93a942987f448e62ad2614e46e"},{"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip","size":24623217,"digest":"sha256:eb9108dbdabef43ae694b955990153159c1cbd069c3e4ca5ac9ccdbdda0e4b09"}]}]]
DEBU[0000] docker-daemon: Closing tar stream to abort loading
FATA[0000] Error writing manifest: Unsupported manifest type, need a Docker schema 2 manifest

I added "the manifest: [[]]" for my own debugging purposes. It's clear that copy/copy.go never does a conversion. And the reason for this is that the OCI transport doesn't implement types.Image (specifically it doesn't implement UpdatedImage which is necessary).

Since the two specs' manifests (Docker schema 2 and OCI) have the same schema but just a different mediatype the implementation should be very simple (it just needs to update the mimetype in the manifest).

I'm confused why the sigstore configuration is not part of policy.json data structure. These things naturally go together. Our registries.d scheme seems to unnecessarily complicate matters.

Example:

{
    "transports": {
        "docker": {
            "docker.io/library/busybox": {
                "type": "signedBy",
                "keyType": "GPGKeys",
                "keyPath": "/path/to/official-pubkey.gpg",
                "sigstore": "https://sigs.example.com"
            }
        }
    }
}

As discussed in more detail #195, the code in #191 is non-obvious, it neither documents why it is needed nor carries a test case; it is fairly risky that we will break it over time just because we don’t know what to maintain.

• Most importantly, what is the way the original code failed?
• It is non-obvious that blindly sending a Basic auth without checking the WWW-Authenticate header contents at all is necessary; can’t we tell exactly that this is what the server needs? https://bugzilla.redhat.com/show_bug.cgi?id=1409873 (note, not the original bug motivating #191) mentions Unimplemented: WWW-Authenticate Bearer replaced by "basic", which suggests there may in fact be some data to base the decision on.
• Also, if using /v2/ ping results and assuming that the ping’s WWW-Authenticate value is relevant for other URLs is unsound in general (as it seems to be, per #195), is there another cleaner approach we could take? E.g. pinging only to determine http/https (then we don’t need care about HTTP response codes at all), and reacting to WWW-Authenticate for each individual URL separately? (The difficulty with this would be delaying a streamed blob upload until we know it would be accepted—we need the server to give us an “auth OK, go ahead” response.)

as per https://docs.docker.com/registry/spec/api/#cross-repository-blob-mount

Following a chat between @rhatdan and @alexlarsson in #atomic on Freenode:

10:27         alexlarsson : skopeo copy oci:oci-dir oci:other-dir
10:27         alexlarsson : FATA[0000] Error initializing source oci:oci-dir:latest: Reading images not implemented for oci: image names
10:27         alexlarsson : Not much support for oci, is there?
10:28             dwalsh_ : runcom|afk, ^^
10:28         alexlarsson : I changed the flatpak oci code to export a directory following the oci image spec layout
10:28         alexlarsson : do we have anything that can read that?
10:29         alexlarsson : To like, test it or something :)
10:29             dwalsh_ : alexlarsson, I thought skopeo had grown support for it.
10:29             dwalsh_ : We need to wait for runcom|afk to come back.

What you need is the ability to have an OCI source, probably remote? well, since OCI hasn't "distribution" in scope that's simply not possible right now (there's work on distribution/distribution#2021)

When ~/.kube/config has multiple contexts configured sometimes the incorrect context is used. See debug output: https://paste.fedoraproject.org/429048/47404610/

Version: issue reproduced using skopeo version 0.1.14-dev commit: 02b9f8b479fee04c6aa0c5038ec07612853b5e67

cc @mtrmac

Right now docker-daemon: destination just passes the input string as a tag in the stream. Docker seems to be doing some canonicalization, but it is unclear what exactly the rules are.

• docker pull busybox gets the image tagged as docker.io/busybox (per docker inspect)
• docker-daemon:busybox gets it tagged as docker:busybox
• docker-daemon:docker.io/library/busybox gets it tagged as docker.io/busybox
• docker inspect busybox:latest finds docker.io/busybox:latest
• docker inspect docker.io/busybox.latest does not find busybox:latest

We need to figure out what is going on, document the relevant formats, and perhaps do some normalization in daemonReference.

cc @baude

was contacted today because people could not figure out the right syntax for skopeo inspect and they were trying skopeo inspect fedora.

would be nice to provide better help messages in all commands (not just inspect) and tell ppl about our image prefixies e.g. docker://, atomic:, oci: etc

/cc @TomasTomecek @mtrmac

We have genericManifest which has Config() in which contains, among other things needed for ImageInspectInfo, some runtime information about an image. An example of where this is needed is appc/docker2aci project.

Question: should we be able to provide such information to users?
Most of the times this is what users expect:

User         string              `json:"User"`                              
Memory       int                 `json:"Memory"`                            
MemorySwap   int                 `json:"MemorySwap"`                        
CpuShares    int                 `json:"CpuShares"`                         
ExposedPorts map[string]struct{} `json:"ExposedPorts"`                      
Env          []string            `json:"Env"`                               
Entrypoint   []string            `json:"Entrypoint"`                        
Cmd          []string            `json:"Cmd"`                               
Volumes      map[string]struct{} `json:"Volumes"`                           
WorkingDir   string              `json:"WorkingDir"`

The other alternative is of course to not expose it and have the docker -> appc conversion here in containers/image which internally uses the Config() from genericManifest to build an ACI for instance.

OTOH we expose the whole manifest so probably this won't be needed and ppl will go around and get the config directly from the manifest perhaps.

@sgotti @mtrmac WDYT?

Right now they are only mentioned in policy.json.md as far as necessary to write the policy. We should document the naming syntax, the protocols, the configuration options etc…

If we end up having another transport named docker-daemon: (or whatever the naming will be) - we would move the implementation under docker/* to be docker/registry/* and have github.com/containers/image/docker/registry package (and docker/daemon ofc).

RepoTags in directory transport miessed:

# skopeo inspect dir:busybox-dir  
{
    "Digest": "sha256:29f5d56d12684887bdfa50dcd29fc31eea4aaf4ad3bec43daf19026a7ce69912",
    "RepoTags": [],
    "Created": "2016-10-07T21:03:58.469866982Z",
    "DockerVersion": "1.12.1",
    "Labels": {},
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190"
    ]
}

After policy evaluation is merged (but filing now as a reminder to myself), signature verification in policy evaluation, with its temporary directory creation, will take about a second per case. Both general performance and test suite interactivity would greatly benefit from speeding this up.

At the very least we can reuse the temporary directories across all verifications with a single policy; that’s why PolicyContext exists in the first place. Also look into speeding up the verification as such, there might be a cheap optimization somewhere.

EDIT: This can be probably done in skopeo

… otherwise a MITM can prevent use of TLS by blocking connections, and then on unauthenticated HTTP they can send a custom WWW-Authenticate: Bearer realm=$attacked-controlled-URL"` header which will cause the user name and password to be sent to the attacker-controlled location.

In

    tagged, ok := r.(reference.NamedTagged)
    if !ok {
        return nil, fmt.Errorf("invalid image reference %s, %#v", ref, r)
    }

and

    r := strings.SplitN(dockerRef.RemoteName(), "/", 3)
    if len(r) != 2 {
        return nil, fmt.Errorf("invalid image reference %s", dockerRef.String())
    }

we know fairly precisely what we want and what we aren’t getting, so we should tell the users what exactly they need to do to have the input accepted, instead of vaguely rejecting the input as “invalid”.

copy semantic in current implementation, all data(include manifest and image layer) in destination will be overwrited by source without to see if it exist.

IMO we should implement sync semantic between source and destination image repository, if layer in destination is exist, it won't be downloaded repeatedly. This will be more efficient.

From a source perspective, this would be implemented as selecting what RepoTag to extract layers from. And from a destination perspective, it would be implemented by changing what the resultant RepoTags already present at a destination are.

Essentially all of this would be controlled by choosing a full reference when creating references that use docker.tarfile.

If you do something like:

% skopeo copy docker-daemon:opensuse/amd64:tumbleweed oci:opensuse:latest
Getting image source signatures
Copying blob sha256:bed80e2cbdd7d22d83fb2b743713df956c59d5edbfded661eda52e346ceed553
 150.50 MB / 150.50 MB [=======================================================]
Copying config sha256:0bd5203d80dd7795a2eaf8873f60f50b7f916ad05a538db489c4e60190e7121c
 1.73 KB / 1.73 KB [===========================================================]
Writing manifest to image destination
Storing signatures
% file opensuse/blobs/sha256/*
opensuse/blobs/sha256/0bd5203d80dd7795a2eaf8873f60f50b7f916ad05a538db489c4e60190e7121c: ASCII text, with very long lines, with no line terminators
opensuse/blobs/sha256/527ad855882abe493134cc4ffb282c020e13bb94a08f708003781d11d6eb5130: ASCII text, with very long lines, with no line terminators
opensuse/blobs/sha256/bed80e2cbdd7d22d83fb2b743713df956c59d5edbfded661eda52e346ceed553: POSIX tar archive

Which is wrong.

Under package oci there will be also an OCI image destination which will support fetching images (like Docker) from registries (one day...)

It makes sense to me to rename the functions now so we don't really break so many people (and since a lot is changing these days...)

@mtrmac WDYT?

… per #63 (comment) . That PR did add an imageCopier.copiedBlobs but it has no uses.

So here is a completely horrible idea: As part of tests, perhaps in Makefile and definitely in Travis, checkout skopeo, replace the vendored copy with current working directory, and test that skopeo runs and its tests pass.

As I said, completely horrible. But perhaps worth it, at least until most of the transports in this repo grow some real tests. Merging containers/skopeo#102 and only a week later discovering that it breaks pushing to OpenShift would be pretty awkward…

WDY(all)T?

Anything against having {Image,ImageSource,ImageDestination}.Close() implement io.Closer https://golang.org/pkg/io/#Closer? (which basically is adding an error return). I think we would have this for #63 also.

ref: https://github.com/containers/image/blob/master/types/types.go#L98

@mtrmac wdyt?

Hi, does is make sense to add more detail about this project to the README. I want to know more about this project but hardly find a docs to begin with.

First question in my mind, what's the different with the OCI/image-tool?

Sorry I don't want to bother you guys @philips @runcom but I find you are the only members of github.com/container.

Now that we have (AFAICT) abandoned Go 1.6 and require Go 1.7, we probably can use the stdlib-included one.

Creating as a TODO (for myself mainly)

We need to fix OCI destination as per opencontainers/image-spec#208

I'm assigning this to myself and fix it.

as rightly pointed our here moby/moby#26369 (comment) it would be nice to use docker/distribution/digest type and functions instead of just string. Anyone? 👼

Per implementation (also doc #120) the manifest digest reference for the "docker" transport type is HASH_FUNCTION:HASH_VALUE, using a colon (:) delimiter. The "at" symbol (@) is also used the delimit image and manifest digest (again, see #120). Per URI: Generic Syntax these are reserved delimiting characters.

Impact
The colon in the URI is blocking uploading of signatures to JFrog's Artifactory repository management platform, returning 500 error:

"Invalid path. ':' is not a valid name character: atomic-sigstore/aweiteka/true@sha256:f292b8573b1679a512f575d7bc2441815b7528eb114217781199e9106e742e21/"

Percent-encoding the path (%3A) does not help. Artifactory has not responded to this related issue

Right now we silently replace it with a digest, which is rather useless. This will need extra logic for v2s1 to either reject pushes with a different tag, or to regenerate the manifest.

See all the FIXMEs in there, and in transports/transports_test.go. And probably add some tests.

In the signature package, the mechanism source uses a cgo wrapper for GnuPG Made Easy library: github.com/mtrmac/gpgme. Is there a reason why this was chosen over pure Go's native OpenPGP package?

Thanks.

This is a bit of a weird issue, but it's something that is exposed if you apply #148. Effectively because docker-daemon (and thus docker/tarfile) return the set of DiffIDs as the LayerInfos for a particular schema, you end up with GetBlob(diffid) giving you the wrong hash.

I'm not really sure how we should handle this, because I don't think the docker save-style archives don't contain enough nicely-formatted information about the compressed layer digests for us to be able to switch. Should we just un-gzip layers?

We need to extend our custom client to sign images from inside a build pod:

• Use env vars KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT
• Use token from /var/run/secrets/kubernetes.io/serviceaccount/token
• Use CA from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
• Use namespace /var/run/secrets/kubernetes.io/serviceaccount/namespace

Our openshift client is a custom client. It would be much better to use the upstream k8s client. This shouldn't significantly increase the size of the project.

Hello,

In GrootFS [1], we are caching the blobs and image config when downloading remote (Docker) images. The only thing we can't cache at the moment is the manifest, since ImageSource.GetManifest is fetching the whole manifest regardless of whether or not is already cached.

The Docker registry v2 API supports Docker-Content-Digest header [2], which contains the manifest digest. We would like to extend the ImageSource interface by adding a GetManifestDigest method which uses HTTP HEAD to just fetch the digest and MIME type. That way we will be able to use our cache if we have the most recent manifest.

diff --git a/types/types.go b/types/types.go
index b67e6b3..669f579 100644
--- a/types/types.go
+++ b/types/types.go
@@ -85,6 +85,9 @@ type ImageSource interface {
  // Reference returns the reference used to set up this source, _as specified by the user_
  // (not as the image itself, or its underlying storage, claims).  This can be used e.g. to determine which public keys are trusted for this image.
  Reference() ImageReference
+ // GetManifestDigest returns the image's manifest digest along with its MIME type. The empty string is returned if the MIME type is unknown. The slice parameter indicates the supported mime types the manifest should be when getting it.
+ // You may use this operation to check if you have the most recent manifest in cache. It may use a remote (= slow) service.
+ GetManifestDigest([]string) (string, string, error)
  // GetManifest returns the image's manifest along with its MIME type. The empty string is returned if the MIME type is unknown. The slice parameter indicates the supported mime types the manifest should be when getting it.
  // It may use a remote (= slow) service.
  GetManifest([]string) ([]byte, string, error)
--

We would like to PR this work, but before we would like to know WDYT?

Thanks,
@albertoleal and @glestaris

[1] https://github.com/cloudfoundry/grootfs
[2] https://github.com/docker/distribution/blob/master/docs/spec/api.md#digest-header

Cross posting from containers/skopeo#110

From kubernetes/kubernetes#26788 (comment)

Would be nice to support such a thing in skopeo itself as part of our types

The scenario is the following (requires interacting with Docker):

• docker pull runcom/busybox:signed
• the docker CLI checks that a valid signature exists for runcom/busybox:signed via containers/image
• containers/image provides the trusted digest for runcom/busybox:signed
• the docker CLI now tries to docker pull runcom/busybox@sha256:%HEX%
• since there's the trust-plugin on the docker API side ...
• trust-plugin check that a valid signature exists for runcom/busybox@sha256:%HEX%
• trust-plugin fails because we don't have signature for runcom/busybox@sha256:%HEX% but for runcom/busybox:signed (which is one of the tag for that digest, which should be already trusted)

I'm proposing something to relax this since if you're pulling by digest at some time it's likely that you have already checked for signatures from a tag.

/cc @mtrmac @aweiteka @rhatdan

Currently master fails with this error:

?       github.com/containers/image     [no test files]
ok      github.com/containers/image/copy        0.052s  coverage: 14.9% of statements
ok      github.com/containers/image/directory   0.052s  coverage: 78.1% of statements
ok      github.com/containers/image/directory/explicitfilepath  0.011s  coverage: 85.7% of statements
ok      github.com/containers/image/docker      4.481s  coverage: 47.1% of statements
ok      github.com/containers/image/docker/daemon       0.008s  coverage: 10.5% of statements
ok      github.com/containers/image/docker/policyconfiguration  0.010s  coverage: 94.7% of statements
--- FAIL: TestParseReferenceWithTagAndDigest (0.00s)
        reference_test.go:248: Error parsing reference: "busybox:latest@sha256:86e0e091d0da6bde2456dbb48306f3956bbeb2eae1b5b9a43045843f69fe4aaa" is not a valid repository/tag
FAIL
coverage: 69.9% of statements
FAIL    github.com/containers/image/docker/reference    0.005s
ok      github.com/containers/image/image       0.039s  coverage: 47.7% of statements
ok      github.com/containers/image/manifest    0.065s  coverage: 90.6% of statements
?       github.com/containers/image/oci [no test files]
ok      github.com/containers/image/oci/layout  0.011s  coverage: 45.4% of statements
ok      github.com/containers/image/openshift   0.005s  coverage: 4.0% of statements
ok      github.com/containers/image/signature   38.552s coverage: 94.2% of statements
ok      github.com/containers/image/storage     0.011s  coverage: 28.8% of statements
ok      github.com/containers/image/transports  0.007s  coverage: 92.9% of statements
?       github.com/containers/image/types       [no test files]
?       github.com/containers/image/version     [no test files]
make: *** [Makefile:18: test] Error 1
make test  11.77s user 2.23s system 33% cpu 41.214 total

However this error only happens if you update upstream to the latest version (go get -t). So I think this is actually some upstream change that we aren't handling properly. Do we want to vendor this code, or otherwise integrate it here? I'm not sure, but something is definitely broken.

If you try to use skopeo to push an OCI image to a Docker registry you get the weird error (this was done after copying the current master of containers/image to the vendor/ of `skopeo):

% skopeo copy oci:opensuse:new docker://cyphar/opensuse_playground:latest
Getting image source signatures
Copying blob sha256:467db25190688bc9dc1d5fd6dbced1ac56f55d93a942987f448e62ad2614e46e
 46.97 MB / 46.97 MB [=========================================================]k
Copying blob sha256:eb9108dbdabef43ae694b955990153159c1cbd069c3e4ca5ac9ccdbdda0e4b09
 23.48 MB / 23.48 MB [=========================================================]
FATA[1275] Error creating an updated image manifest: Conversion of image manifest from application/vnd.docker.distribution.manifest.v2+json to application/vnd.docker.distribution.manifest.v2+json is not implemented

So it's failing to convert from itself to itself?

https://github.com/opencontainers/image-spec/blob/master/specs-go/v1/descriptor.go

@mtrmac wdyt?

Namely, the docker load-friendly format. This would make it possible to use skopeo to have a full OCI build system that will then pipe directly into a Docker image without needing a Docker registry or Docker daemon on the build system. In particular this would be very handy for implementing the Open Build Service support that we're planning at SUSE (we can't easily run a Docker registry as part of our infrastructure).

Outstanding comments from #160:

• Why are we using a ServerDefault tls.Config in a client?
• Why are we applying it only to a single path, not when we override tls.Config for our purposes?
• In tlsclientconfig.NewTransport, the proxyDialer setup should happen before we create a http.Transport, so that we don’t need to override tr.Dial in a created object. Irrelevant after #1748

At least covering the success case for manifest conversion, like docker_schema2_test.go

Use case: I have built a local docker image. I need a way to reference this image to push it to a registry and sign it. We need to be able to reference local docker images to copy them to a registry and generate a manifest to sign without changing the name of the image.

localdocker:// is just a suggestion. Don't care what we call it.

Probably blocking projectatomic/docker#200 but not really since we never thought well about this scenario.
Right now in docker/docker you can pull an image which its manifest isn't a single manifest (with a digest) but it's a list of manifests.

@mtrmac is this a blocking issue? should we trust all sub-manifests in a manifest list if the manifest list it's signed?

For manifest list (and following the idea from #99) we could have another field in signedBy with: "manifestList": {prohibit, trustList, trustTarged} and by default is trustList (or is it too open?)

dockerImageDestination.SupportedManifestMIMETypes returns a hard-coded value; if the destination registry does not in fact support schema 2, we would still attempt to upload this.

This incorrect information will break any manifest conversion we might attempt.

Hello,

containers/image imports docker/docker for the reference package which is used for types of for reference parsing [1]. docker/docker is a quite big repository and would prefer to avoid having it as a dependency in GrootFS [2], which consumes containers/image.

Would it be possible to use docker/distribution instead for the reference types/parsing? Additionally, the Docker transport package seems to be doing all the requests to a Docker registry [3]. Could this behavior be delegated to docker/distribution?

Thanks!

[1]