containers / skopeo Goto Github PK

Hi, I'm wondering if anyone here has looked at the signing discussion here:

opencontainers/image-spec#22

skopeo inspect
skopeo pull
...

it's ugly that a library is retrieving credentials from something on the host w/o users noticing. I think we should find a way to abstract credentials and put them directly on ImageSource initialization

#39 fixed build with current RHEL 7, but the updated code is rejected with Go 1.6 (see proglottis/gpgme#5 ).

A hopefully final and fully correct fix was sent upstream as proglottis/gpgme#7 . After we are sure this is the best fix, we should update mtrmac/gpgme (if necessary) and then update the vendored copy in skopeo.

(No action needed right now; filing this now so that it happens eventually.)

From kubernetes/kubernetes#26788 (comment)

Would be nice to support such a thing in skopeo itself as part of our types /cc @rhatdan

@mtrmac do you think it makes sense?

e.g. cmd/skopeo/utils.py uses c.GlobalBool("tls-verify") which defaults to false. Should audit everything, really (not that there is that much to audit, but the above may not be the only place).

it's been a while since the last release of this was put out. Can we cut a tag and so that a distro (like Fedora) can ship an updated version?

Building skopeo in RHEL7 with go1.4.2 gives:

$ /usr/local/go1.4.2/go/bin/go build -o skopeo ./cmd/skopeo
# github.com/projectatomic/skopeo/signature
src/github.com/projectatomic/skopeo/signature/json.go:68: dec.Token undefined (type *json.Decoder has no field or method Token)
src/github.com/projectatomic/skopeo/signature/json.go:72: undefined: json.Delim
src/github.com/projectatomic/skopeo/signature/json.go:76: dec.Token undefined (type *json.Decoder has no field or method Token)
src/github.com/projectatomic/skopeo/signature/json.go:80: undefined: json.Delim
src/github.com/projectatomic/skopeo/signature/json.go:103: dec.Token undefined (type *json.Decoder has no field or method Token)

The fastest thing we can do to deal with this is to create a file signature/json_go1.4.2.go (or the like) and put there a build tag to be compiled only with go1.4.2 and lower - and another file to build with go > 1.4.2 (which has the undefined methods above in the errors). We can just move there paranoidUnmarshalJSONObject so we have two version of this function for the different go versions
.
@mtrmac sounds good to you? is paranoidUnmarshalJSONObject fine to be reimplemented w/o using Token and Delim which aren't available in go1.4.2?

A recent demo of the signing functionality is fairly easy to reuse / explain. Within skopeo, we will replace it with Go code in #118, so there is no reason to keep it in the git repo right now, but it seems a pity to just throw it away.

The shell script still might be useful as is, perhaps as a part of an introduction/walkthrough documentation in the repository, or as a blog post.

So, I will attach the shell script to this issue. Note that it requires the full --sign-by and --policy functionality and the fixtures from #118 (~per my current integrate-all-the-things branch), and it is designed to run inside the skopeo-dev container built by Makefile.

In the worst case, this issue can be closed (perhaps even immediately) and the attached shell script will be preserved in the issue.
start-openshift.sh.txt

Would make handling the json easier.

@jwhonce @jberkus @rhatdan

skopeo has tests that run on Travis, could you please enable Travis for this repository please?

The outputs of skopeo --help and skopeo $command --help are not helpful enough; we need to review them and add the missing documentation.

skopeo-0.1.11-1.fc23.x86_64

No matter what I do I can only get an error to be output:

# skopeo --debug inspect https://docker.io/fedora 
FATA[0000] Please use a fully qualified repository name

Repository description isn't accurate anymore since skopeo is about to be able to download image's layers in addition to inspect images on a repository

This is more of a question: can I use skopeo to copy an OCI image (directory containing refs/ and blobs/) to a docker registry?

… to detect build breakage before we merge to master.

I got as far as #46 , but finishing it would require more time than I can spare right now.

I'm on a Mac trying to fool around with skopeo. Building the Docker image locally takes "for ever". A simple docker run atomicregistry/skopeo would be great.

it's empty right now

As of now, the Atomic CLI must use a subprocess to interact with skopeo. This is not entirely ideal. Among other things, you can see type differences between python2 and 3 when working with the returned information.

DBUS would allow us to "bind" with subprocess, define return types, etc.

Currently skopeo is under a freeze in CAHC:

Trying to unfreeze, I get:

iinstall -m 755 skopeo /builddir/build/BUILDROOT/skopeo-0.1.12.10-0d95328125be9e411b8b4b5f301f2dced5a04d31.5d9911952c34defcf625a69b2ca451345e52e459.el7.centos.x86_64/usr/bin
install: cannot stat 'skopeo': No such file or directory

I looked at this briefly but it seems like the actual binary is ending up in some strange place (WORK=/tmp/go-build43465149 ).

Hello, I know I can do this:

$ skopeo copy docker://busybox:latest oci:existingemptydirectory

However, can I specify the source as a docker image which is only present locally? Trying it gives me:

$ sudo docker images
REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE
amitsaha/rootfs-1               latest              bbb478419aa9        7 minutes ago       115.9 MB

$ sudo skopeo copy docker://amitsaha/rootfs-1:latest oci:existingemptydirectory1
FATA[0005] Error reading manifest: error fetching manifest: status code: 401, body: {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Name":"amitsaha/rootfs-1","Action":"pull"}]}]}

If this is not yet available, is there any alternative to directly converting a docker image which is not in any supported registry to an OCI bundle? Thanks for any hints!

When trying to download an old style docker v1 image format I get an error:

$ ./skopeo copy docker://dustymabe/fedora20-pdftk:latest oci:fedorapdftk 
FATA[0010] Error writing manifest: Unrecognized manifest media type

talking with @runcom I was able to find out that the image format was old and I needed to point to newer v2 images in order to get it to work. would be great if we could add a helpful message for our users on this point.

On a fresh machine install I just needed skopeo to pull some oci images needed for testing and I realized the tool is failing with:

$ skopeo copy docker://busybox oci:busybox
FATA[0000] Error loading verification policy: open /etc/containers/policy.json: no such file or directory

@mtrmac do you think we can relax this somehow? I know I can use default-policy.json tbh but.. wdyt?

As a registry admin I am able to configure the registry to serve on any port. When using a port other than 5000 I get this error:

    "Invalid format of docker reference: expecting port 5000"

See https://github.com/projectatomic/skopeo/blob/4421e7ea2face14ce4530ed6e6df66b6f94d544f/vendor/github.com/containers/image/openshift/openshift.go#L164

Could we discover the port using ~/.docker/config.json?

@mtrmac we need something like https://godoc.org/github.com/docker/distribution/manifest and https://godoc.org/github.com/docker/distribution#RegisterManifestSchema.

https://github.com/docker/distribution/blob/master/manifests.go#L86

This should play well with the manifest interface also. We really need to abstract manifests code in skopeo.

I think their structure is perfect in our case when dealing with different kind of manifests media types (and marshaling/unmarshaling)

WDYT I can work on this after #102 and #103

In many cases, signatures are served from a different server than the registry. We need a way to map this signature server (sigstore?) to a registry.

The OpenShift API ImageSignature object provides storage for useful metadata. Skopeo should write this data if possible.

• imageIdentity: <image_reference>
• created:
• issuedBy:
• issuedTo: # what's the difference?

To clarify to the end user that this information is not verified, populate the SignatureCondition object with the following fields:

• type: "Complete"
• status: "Unknown"
• lastProbeTime:
• reason: "Initial signing request"
• message: <???>

$ skopeo registry.access.redhat.com/rhel7 | jq '.Config.Labels.Name'
time="2016-03-08T16:31:20+01:00" level=error msg="Error trying v2 registry: Error parsing HTTP response: invalid character '<' looking for beginning of value: \"<!DOCTYPE HTML PUBLIC \\\"-//W3C//DTD HTML 3.2 Final//EN\\\">\\n<title>404 Not Found</title>\\n<h1>Not Found</h1>\\n<p>The requested URL was not found on the server.</p><p>If you entered the URL manually please check your spelling and try again.</p>\\n\"" 
"rhel7/rhel"

command is successful but that debug output is weird and docker.io registry doesn't suffer from this

The capability to use .kubeconfig files is great but we frequently don't have the oc client installed on the system to manage the .kubeconfig file.

This is a request for skopeo to support this so we can reduce the host dependencies.

@mtrmac just a tracking question

Wouldn't it be better if ImageDestination return Writers? I don't have anything in mind about how to unify all the image destinations to do so but as we return Readers from ImageSource it came to my mind why don't do the same and return Writers there.

I'm saying, why don't we try and further abstract methods in ImageDestination to support this? is it doable? wdyt?

I've been experimenting with CentOS Atomic Host and Fedora Atomic Host recently, and when looking at system containers and installation using the atomic command, I've hit a snag. These boxes don't get web access except via a proxy, and even though I've set HTTP_PROXY and HTTPS_PROXY environment variables, I'm still not able to pull images. Looking through the python code, I noticed that atomic calls skopeo, and on testing skopeo independently, it seems skopeo is not using the http(s) proxy environment variables.

The command I'm running is:
HTTPS_PROXY=http://wwwcache.keele.ac.uk:8080 HTTP_PROXY=http://wwwcache.keele.ac.uk:8080 skopeo --debug inspect docker://docker.io/fedora

The output I receive is:
DEBU[0254] Ping https://registry-1.docker.io/v2/ err &url.Error{Op:"Get", URL:"https://registry-1.docker.io/v2/", Err:(_net.OpError)(0xc820278b90)}
DEBU[0572] Ping http://registry-1.docker.io/v2/ err &url.Error{Op:"Get", URL:"http://registry-1.docker.io/v2/", Err:(_net.OpError)(0xc820278d70)}
FATA[0572] Get http://registry-1.docker.io/v2/: dial tcp 54.175.214.89:80: getsockopt: connection timed out

Additionally, running tcpdump shows no traffic heading to the web proxy, instead direct connection attempts are made to servers hosted in AWS, which fail.

Should this be working? If not, are there any plans to support operating over a web proxy?

it's ugly I know - let's move to something else such as glide and glide-vc https://github.com/Masterminds/glide

After parseDockerImageName, we treat digests and tags the same. This seems rather suspicious, when the general syntax is name:tag but name@digest; e.g. the CanonicalDockerReference and IntendedDockerReference implementations are probably incorrect.

1. More test cases a) push & pull images using local docker registry b) see @runcom todo notes
2. OCI test cases need to be added (OCI enabled local registry) a) using modified registry fork that enables OCI push pull.

I'm doing some work on the above. Comments & questions?

DEBU[0019] Uploading mitr/busybox/blobs/uploads/?digest=sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 
DEBU[0019] POST https://atomic-registry.usersys.redhat.com:5000/v2/mitr/busybox/blobs/uploads/?digest=sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 
DEBU[0019] Error uploading, status 202                  

or after s/POST/PUT/:

DEBU[0019] Uploading mitr/busybox/blobs/uploads/?digest=sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 
DEBU[0019] PUT https://atomic-registry.usersys.redhat.com:5000/v2/mitr/busybox/blobs/uploads/?digest=sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 
DEBU[0019] Error uploading, status 404                  

This is explained by distribution/distribution@2a4deee , the API never existed.

I have no idea how this got through all my testing 😞

Currently RH registry doesn't support V2 APIs and skopeo does not support V1 APIs either:

$ ./skopeo inspect docker://registry.access.redhat.com/rhel7 
FATA[0001] error pinging repository, response code 404

In order for skopeo to work correctly the following APIs should be supported in the RH registry:

GET /v2/
GET /v2/<name>/tags/list
GET /v2/<name>/manifests/<reference>
GET /v2/<name>/blobs/<digest>

More on each of the above https://docs.docker.com/registry/spec/api/

Per #149 (comment) :

But I would prefer if we rewrote these all in markdown, so they could easily be displayed on web sites and it would make it easier for people to contribute.

I would like to get the tarsum of docker save'd images so that I can use the same sha256 used in the v2 registry. Is Skopeo the right place to add this functionality? I would like something like `skopeo tarsum /path/to/foo.tar' and that internally uses https://github.com/docker/docker/tree/master/pkg/tarsum

What do you think?

With all the new stuff we merged in master the README.md got a bit outdated now. We would need to updated it in order to reflect the current master code with examples. Also, with the upcoming move to libraries it will be far far better to have something easy shown in the main github page.

@mtrmac :)

The word "trust" seems to resonate with docker users. See Docker Content Trust. The skopeo policy file describes trust quite well but its use must be translated for the uninitiated.

I'm not suggesting we do a global search and replace, but in cases where users interact with files or output, "trust" is a better term to use IMO. Specifically, these files:

cmd/skopeo/copy.go
cmd/skopeo/main.go
docs/skopeo.1.md
Makefile

cc @rhatdan

@mtrmac #24 broke master CI, we should look into it

The atomic type is awkward when using with "copy" workflow. The primary object the user knows about is a docker image, locally or in the registry. What we do with the atomic type is infer the docker image reference. This means we don't have the registry URL or port (see #189).

What seems to make more sense is to drop atomic transport type and use a flag to enable writing to openshift API.

Copy:

skopeo copy --sign-by [email protected] --atomic docker-daemon:registry.example.com/my/image docker://registry.example.com/my/image

Standalone:

skopeo standalone-sign --atomic MANIFEST registry.example.com/my/image KEY-FINGERPRINT

We want to be able to build an image then use skopeo copy to sign and push to a registry. For example:

docker build foo/bar
skopeo copy --sign-by [email protected] localdocker://foo/bar docker://reg.example.com/foo/bar

cc @baude

Various registry configuration items are stored in tool-specific configuration locations:

• “this registry can be accessed over HTTP” via docker-daemon’s --insecure-registry (which can be in /etc/sysconfig/docker’s INSECURE_REGISTRY, or OPTIONS, or overridden in docker.service); altogether quite difficult to parse; skopeo does not read any of this, adds its own command-line option.
• The TLS certificate & key supported in skopeo (but not docker?!) also can be only specified on the command line.
• The sigstore* options considered in #170 , configured in atomic.conf (which makes it awkward to include the parsing and semantics in containers/image).

Would it make sense to introduce a system-wide configuration file, perhaps /etc/containers/docker-registries.yaml, which allows per-registry and per-repository configuration, to be used by all of the tools if possible?

• The configuration would be shared across various tools (we could even generate --insecure-registry automatically, no need to patch Docker)
• Well-defined format instead of plain-text OPTIONS would make it easier to manipulate with tools.
• Consolidates similar information into one place
• Vendor/project-neutral (but of course supporting vendor extensions, e.g. via x-reverse.domain.name or whatever)

@runcom , @rhatdan ?