We have several examples of well-formed JSON objects that pass schema validation, but no examples that fail. We need several for the following reason:

1. Ensure that schema validation is actually working.
2. Clarify aspects of the specification.

Ideally, these could be part of the specification and marked as bad, in congruence with the approach of having the examples validated.

Please see #81 (comment).

cc @s-urbaniak

https://github.com/opencontainers/image-spec/blob/master/media-types.md#relations is missing the diagram. How do we have the diagram on GitHub and in the final build?

• The serialization.md shows a config json per layer, where the manifest.md only has one config json. What happens during translation here?
• The serialization.md shows file names for the objects, where the manifest.md just has the objects in a base-layer-first order.

with https://github.com/opencontainers/image-spec/releases/tag/v0.1.0
The media-types.png caused issues in the make docs target. See #66 (comment)

As a make-shift effort, I switched it to inline html. While that caused pandoc to not fail, it did not render the image in the resulting document.
Here's the diff that let pandoc run successfully.

diff --git a/media-types.md b/media-types.md
index 3bf7221..de6cc3b 100644
--- a/media-types.md
+++ b/media-types.md
@@ -46,6 +46,6 @@ This section shows where the OCI Image Specification is compatible with formats

 The following figure shows how the above media types reference each other:

-![Media Types](media-types.png)
+<img src="media-types.png" alt="Media Types"/>

 A reference is defined as the target content digest, as defined by the [Registry V2 HTTP API Specificiation](https://docs.docker.com/registry/spec/api/#digest-parameter). The manifest list being a "fat manifest" references one or more image manifests per target platform. An image manifest references exactly one target configuration and possibly many layers.

Other option would be a full base64 inline of the image, in an html tag. 😛

@philips requested an issue to require Image IDs to be crypto hashes and not be randomly generated.

Image ID

Each layer is given an ID upon its creation. It is represented as a hexadecimal encoding of 256 bits, e.g., a9561eb1b190625c9adb5a9513e72c4dedafc1cb2d4c5236c9a6957ec7dfd5a9. Image IDs should be sufficiently random so as to be globally unique. 32 bytes read from /dev/urandom is sufficient for all practical purposes. Alternatively, an image ID may be derived as a cryptographic hash of image contents as the result is considered indistinguishable from random. The choice is left up to implementors.

However, wouldn't that make

Image Checksum

The checksum is a cryptographic digest of the artifact.

redundant?

I'm not sure why you shouldn't be able to use a 'random' (or other) ID, but I'm also not sure why you wouldn't want to use a hash of the content.

The descriptor spec has a section on reserved fields which we really don't have great precedent for. This issue is to discuss the formalization or removal of the "reserved" concept across the spec.

Previous discussions:
#111 (diff)
#111 (diff)

I'm confused why layers are refered to using random numbers rather than with cryptographic hashes. We already know that using random numbers as IDs lead to security problems at Docker, so why continue using this approach in a new standard?

The only thing that occures to me, is that one might think it possible that two layers of the same image would have the same hash, for example, if a file was repetedly whited out. But in truth this would NEVER happen, as the JSON file, which specifies the layers parent, would be different.

We have two definitions of layers:

1. https://github.com/opencontainers/image-spec/blob/master/manifest.md#image-manifest-property-descriptions
2. Layer DiffIDs specified in https://github.com/opencontainers/image-spec/blob/master/serialization.md#terminology

Which list should we consider for unpacking/oci-runtime-bundling?

We need to factor the spec for top to bottom reading with three audiences in mind:

• Build system creators
• Image registry creators
• Container engine creators

I will try and solicit feedback from folks on what they would like to see from the spec. And it would great if other @opencontainers/image-spec-maintainers could help out here now that we have a v0.1.0 out the door.

The "Image JSON", media type oci-image-tool: Implement config validation (media type application/vnd.oci.image.serialization.config.v1+json validation needs to be implemented in the oci-image-tool

If a client has a URL/URI/file for an image spec. manifest, how does the client then know how to download/obtain the relevant layers?

This seems like its an important part of the image spec. that is currently not defined anywhere. Given the image manifest as its currently defined, I don't think that anyone will actually be able to distribute image manifests in a way where multiple different client implementations can figure out how to get the layers.

If we enter a world where each client does it differently, then the whole point of having a standardize specification will be lost, so I think we need to have some sort of guidance for this in the image spec.

I don't know that we have to require every implementation of specification to implement this part of the spec, but we should at least make it optional so that we don't end up with N different implementations.

It could be something as simple as:

https://my-server.com/layers/

Or we could actually add fields to the image spec that give a URL, but regardless we need something in order to begin implementation.

For the "image layout" that can be tar'd, zip'd, etc we need to create the spec and media-type for the "descriptor"

https://github.com/opencontainers/image-spec/blob/master/schema/defs-image.json#L23

Relevant IRC convo:

4:08 PM <philips> stevvooe: this response? https://github.com/opencontainers/image-spec/pull/82#issuecomment-221980346
4:09 PM <stevvooe> philips: yes
4:09 PM <stevvooe> philips: it just means that we deserialize a descriptor, rather than a few fields of the manifest
4:09 PM <philips> stevvooe: I don't know what you mean by half-deserialized still
4:09 PM <philips> stevvooe: what is a descriptor?
4:09 PM <stevvooe> philips: "partially-serialized" would probably be more accurate
4:10 PM <stevvooe> stevvooe: this structure: {"size": 4096, "digest": "sha256:afff3928", "mediatype": "application/vnd.oci.image.manifest.v1+json"}
4:10 PM <philips> stevvooe: ah, ok. well, did you see my concern about mime types?
4:10 PM <stevvooe> philips: yea, and now i think i understand your example
4:11 PM <philips> stevvooe: like my structure is identical but allows for N media types: https://github.com/opencontainers/image-spec/pull/82#issuecomment-221760143
4:11 PM <stevvooe> philips: i think if we push that dispatch to a manifest list, we can avoid a large amount of unverified content
4:12 PM <philips> stevvooe: so you suggest always force a manifest list?
4:12 PM <philips> stevvooe: like the manifest list is always the thingie we point to in this "container"
4:12 PM <stevvooe> philips: so, the options are <descriptor> -> <manifest list> or <new format: [descriptor]> -> <arbitrary>
4:13 PM <stevvooe> philips: that might be the right approach
4:13 PM <philips> stevvooe: yea, not opposed to the descriptor thing for "file transport containers"
4:13 PM <philips> stevvooe: and always pointing to a manifest list makes sense, I think.
4:13 PM <stevvooe> philips: btw, here is the descriptor def : https://github.com/opencontainers/image-spec/blob/master/schema/defs-image.json#L14
4:15 PM <philips> stevvooe: got it, we only ever refer to it that way in code instead of in the spec so I wasn't familiar with the term 
4:15 PM <philips> stevvooe: thanks, makes sense
4:17 PM <stevvooe> philips: great! we may want to pull out the descriptors into another specification, maybe post-.10
4:17 PM <wking> philips: re: "always pointing to a manifest list makes sense".  If you don't assume that, it's easy to check and have tooling error out if it hits an unsupported MIME type
4:18 PM <wking> If you do assume that, you're stuck with it for as long as you want to keep backwards compatability with old tooling
4:18 PM <philips> stevvooe: like into their own mime-type?
4:18 PM <wking> (assuming the old tooling also assumed that and didn't check the descriptor MIME type https://github.com/opencontainers/image-spec/blob/master/schema/defs-image.json#L23
4:19 PM <stevvooe> philips: yes. file an issue and i'll take that on
4:20 PM <philips> wking: ack, right, I think the list is sufficiently dumb/flat enough to not require breaking changes for a long time.
4:20 PM <philips> stevvooe: sounds good, thanks.
4:20 PM <wking> sure, but what's the benefit to taking that risk in the spec?
4:20 PM <philips> wking: what are you suggesting as an alternative?
4:21 PM <wking> that the spec not require a particular MIME type for the reference descriptors
4:21 PM <philips> wking: so a descriptor sans mediatype? {"size": 4096, "digest": "sha256:afff3928"}
4:21 PM <wking> I am +1 on clients only supporting the manifest list for as long as they make sense, but if the spec *requires* a given MIME type it makes the eventual upgrade awkward
4:22 PM <wking> philips: no, include the linked MIME type in the descriptor, just don't say "reference descriptors MUST set ...manifest.list... as their mediaType"
4:22 PM <philips> oh, right, yes, +1
4:22 PM <philips> that was my intention/plan

As per #82 (comment): Currently, the oci-image-tool supports a simple unpacking mechanism to inspect deflated images and validation.

More concretely the config/image JSON https://github.com/opencontainers/image-spec/blob/master/serialization.md#image-json-description has to be converted to a https://github.com/opencontainers/runtime-spec/blob/master/config.md#container-configuration-file.

https://travis-ci.org/opencontainers/image-spec/builds/132403826

0aafcef

Once #94 is finalized we can make the oci-image-tool create to create an OCI runtime bundle:

$ oci-image-tool create-runtime-bundle path/to/image path/to/bundle
$ oci-runtime path/to/bundle

Similar to #103, the config object is not indicated as REQUIRED or OPTIONAL.

It might be good to have some additional guidance around schemaVersion

schemaVersion int
This REQUIRED property specifies the image manifest schema version. This schema uses the version 2 .

For example, is this intended to align with major and minor versions? i.e. v0.1 ==1, v0.2 ==2, etc.?

The Image Manifest specifies a "config" object in https://github.com/s-urbaniak/image-spec/blob/master/manifest.md#image-manifest-field-descriptions and references a application/vnd.oci.image.serialization.v1+json media type but it is not specified.

Can I assume this refers to application/vnd.oci.image.serialization.config.v1+json?

Will it be possible with this spec to share tar blobs across multiple images? Requiring a parent field in a layer's metadata could make that impossible, depending on what's hashed and how, but I've had trouble finding out if a parent is required.

My ideal is to be able to specify an image as a collection of tar files that contain disjoint collections of files. Multiple images might share data or libraries, and I'd like to maximize that sharing and minimize the number of distinct tar files that need to be transferred and stored.

I've tried to figure this out from the spec. The schema itself makes no mention of parents. The surrounding markdown files mention parents, as does #25.

https://github.com/s-urbaniak/image-spec/blob/master/serialization.md#combined-image-json--filesystem-changeset-format specifies a couple of files, none of which have any media types defined:

• VERSION (plain text file) schema version of the following json file
• json (json file), pointing to a "version 1" metadata format of the, which show 404.
• repositories (json file)
• layer.tar (layer tar file)
• manifest.json (json file) (is this the "combined Image JSON"?)

It is not clear to me how this relates to the Image Manifest file (https://github.com/s-urbaniak/image-spec/blob/master/manifest.md#image-manifest) and how we can reference a combined image from the "Image Manifest".

Also it is unclear to which of the above files the specified application/vnd.oci.image.serialization.combined.v1+json media type points to, the +json indicates a JSON format, is it manifest.json?

In the spec:

digest string
The digest of the content, as defined by the Registry V2 HTTP API Specificiation.

Other similar items such as schemaVersion and mediaType indicate if they are REQUIRED or OPTIONAL but item does not.

For:

• DCO
• commit length
• etc

As seen in https://github.com/opencontainers/image-spec/blob/master/schema/fs.go#L227 *.go files are encoded in schema/fs.go but shouldn't.

From a single tar archive of tar archives and JSON documents, or a manifest JSON document that points to other objects needing to be fetched.

UserStory:

• As an opensource project, I want to produce an OCI image for the world to consume.

See also https://groups.google.com/a/opencontainers.org/d/msg/dev/VKpZNs-qYoI/RzPhj71ODgAJ

We need a tool to make it easy for people to validate and play with an OCI Image.

Basic validation tool that can parse and inspect all of the objects inside of the OCI Image spec

$ oci-image-tool validate serialized-image.tar
$ oci-image-tool validate manifest.json
$ oci-image-tool validate config.json
json: cannot unmarshal object into Go value of type types.Env

Unpack an image on disk (dependency on #23)
to a OCI runtime bundle:

$ oci-image-tool create-runtime-bundle path/to/image path/to/bundle
$ oci-runtime path/to/bundle

Details:

• We should probably use the bog standard cobra CLI library: https://github.com/spf13/cobra
• We can steal some of the code form the actool: https://github.com/appc/spec/tree/master/actool

The oci-image-tool image unpacker will rely on a in-flux discussion starting from #23 (comment). The proposed layout should be spec'ed out in markdown format and be part of this specification.

xref #75

I have gathered from various discussions that the intention is to have v1.0 be compatible with Docker's spec. If that's right, the goal should be detailed in the README or elsewhere, along with some justification.

$ make docs
pandoc: /input/README.md: openFile: permission denied (Permission denied)
Makefile:30: recipe for target 'output/docs.pdf' failed
make: *** [output/docs.pdf] Error 1

@vbatts something simple?

We need a few "golden standards" to test against the json schema specs to catch unintentional changes. It will also help in having them as part of PRs that show the schema change and how it affects the "golden standard".

@vbatts Thoughts?

I think we should tag and release v0.1.0 of the image spec. We have imported the relevant docs, added information about the media types, and created the new media types for the OCI. It isn't perfect but I want to start having discussions with people implementing registries and build systems and I think it is in good enough shape to start having those discussions.

Any objections @opencontainers/image-spec-maintainers ?

Major themes for the v0.2.0 spec

• Finish discussion on naming (#11) and add a solution to the spec
• Improve spec based on feedback from teams implementing registries/build systems
• Introduce the oci-image-tool #43
• Cleanup the serialization spec #36
• Improved introductory text

See the full milestone for v0.2.0 . If you can take up one of these issues please do! I want to get the v0.2.0 release out in about 2-3 weeks.

Carrying over the approach from https://github.com/opencontainers/runtime-spec/tree/v0.4.0/schema
We might ought to have JSON schema for the structures defined in this project and easy validation of them.

The Backwards Compatibility section has a number of references to "old format."

It may be helpful to clarify what "old format" is referring to.

Per #26 (comment) and
https://github.com/opencontainers/image-spec/issues?q=is%3Aissue+is%3Aopen+label%3A%22component%2Fserialization+spec%22

There is much cruft carried forward with the version that was cherry-picked for this image-spec.

size int
...
If the length of the retrieved content does not match the specified length, the content should not be trusted.

If intended as a recommendation, use "SHOULD" (ALL CAPS.)
If not intended as a recommendation invoke RFC meaning, consider alternate wording.

Carrying over from #7 but specifically for the serialization formats.

• image JSON
• repositories JSON
• manifest.json
• <sha256 id>.json

It would be nice to be able to add metadata to an image (or a layer) without affecting the CAS id for the image. This allows for images to be built reproducibly but stills allows for keeping useful information with the image like the build date, build software versioning, build machine, etc.

This could go into a specific directory (.meta/) inside the image that can be ignored when generating the content id, or it could be shipped in a separate filesystem structure that is outside the "data" filesystem for the image.

With docker save, we typically have a complete image that can be transported independent of a registry. It includes configuration, layers and pointers to the actual content.

For OCI, we should define a similar tree-structured format focused on distributing OCI manifests and constituent content addressable content. The resulting archive can be used to transport complete OCI images based on manifests.

Currently we don't have any canonical independent implementations of the specification. Docker's image format is different to ours (even though the base came from them), and rkt doesn't have support for ours.

As a result, this spec is entirely untested as we don't have any code implementing the spec word-for-word. As several people have discussed on the mailing list, this should be considered a blocking issue for 1.0.

Personally, I would want to have two implementations of the image-spec (like we do with the runtime-spec) before we can consider this ready. Preferably these implementations would be part of Docker and rkt (and have a reasonable cook time).

It is likely that the OCI Image Spec won't have an optional transport layer for a few more months while the TOB discusses and comes to a conclusion. But, as people start to use/implement the spec they will want advice on how to use the format with existing transport mechanisms.

The first topic to cover is how to use the OCI Image Format spec with the Docker v2 Registry API. This document will need to explain the entire process from a name to extracted on disk. @wking drew this diagram of the basic process:

We should probably create a helper oci-image-docker-registry-fetch tool to make it all concrete as well.

Carrying over #35 (comment)

Like appc/spec ACI dependencies https://github.com/appc/spec/blob/master/spec/aci.md#dependency-matching

Ensure that OCI image spec can support an image (config plus descriptors) that includes references to dependent layers that may themselves include its own config.

To some extent, this would satisfy #39

The "image JSON" also described as "Container config JSON" https://github.com/opencontainers/image-spec/blob/master/serialization.md#image-json-description does have a media-type application/vnd.oci.image.serialization.config.v1+json but no schema defined.

Also the example image/config JSON does not have the schemaVersion and mediaType headers defined. I am aware this was taken over, but it seems not to be consistent with the manifest/manifest list schemas.

Finally I suggest to have a unique wording for this JSON file, either "Image JSON" or "Container config JSON" and use this wording consistently across the spec.

It seems unfortunate that a new standard should use a method which unecesarilly limits the standard. With the .wh file approach, base images can no longer contain arbitrary data. This means, for example, that you cannot have an image with example OCI image-spec data stored in it. Is there any possibility of changing this to use, for example, a white out list instead. So that the files that made a layer would be:

VERSION
layer.tar
whiteouts
json

That would mean that truely arbitrary data could be stored in the images, which would be really nice :)

The oci-image-tool should be able to validate and unpack an oci-image from 1. a directory and 2. a tar file.

Proposed spec: #23 (comment)

DNS delegation is the idea that I can own the domain example.com but delegate hosting of my container images to container.hosting. Think MX records, where I have a DNS name that tells me where to send email for a domain without the dedicated special DNS record.

Today, AppC supports a method for doing this type of delegation that uses HTTPS, and HTML meta tags. This type of delegation is inspired by the method used by the go get package management system as well.

One critique of this method is that there is an RFC for this sort of meta information called 'well-known URIs'. I believe that using this type of well-known scheme is a better choice today and will stay out of people's way better; in fact we prototyped this in a project called abd.

FAQ
Why not DNS: If we used DNS it would make it hard to bootstrap trust of this source based on the existing well understood TLS infrastructure on the internet today. With systems like Let's Encrypt in place today I think it would be smart to rely on this existing trust root. We do this in rkt today and it works nicely.

But I have this way better way of doing delegation: We can add additional methods, but having some method to start for delegation is super helpful.

I don't want to allow outbound traffic to the internet: Container engines can allow for this feature to be turned off or configure that certain domains are statically delegated to internal mirrors. This is all about having a default UX.

Use a vendoring tool, i.e. glide to vendor all external dependencies

In ReadMe.md, there are references to "base layer" but seems to be an overloaded use of the term:

The high level components of the spec include:
•An image manifest and filesystem serialization (base layer)
•A process of hashing the image format for integrity and content-addressing (base layer)
•Signatures that are based on signing image content address (optional layer)
•Naming that is federated based on DNS and can be delegated (optional layer)

Maybe change:
"base layer" --> REQUIRED
"optional layer" --> OPTIONAL

This project should "specify a way to attach signatures". There are a few different approaches that could be taken based on prior art:

1 Detached signature sitting alongside each CAS object (this is the approach appc takes)
2. Inline signatures in the manifest list, putting things into the JSON objects themselves won't work, obviously.
3. Completely external signing with a signing server (this is the approach notary takes).

One thing I hope we can do with the image-spec is to remain compatible with the initial docker layer.tar format so that hosting providers don't need to duplicate filesystems to support both formats.

To that end we will need to create a compatibility.md file or something that says something like:

Compatible/Interchangeable: 
application/vnd.docker.image.rootfs.diff.tar.gzip, application/vnd.oci.image.rootfs.tar.gzip

Forward compatible:
application/vnd.oci.image.rootfs.tar.gzip, application/vnd.oci.image+1.rootfs.tar.gzip

This is coming from the introduction of the new mime types in this PR: https://github.com/opencontainers/image-spec/pull/12/files

Given the discussion on #37 and on the mailing list, non-chain topologies of "layers" are possible and parents are in fact optional. Given this, I think the term "layer" is not ideal. I think it comes directly from Docker's implementation and particularly from the Dockerfile build mechanism. For this spec, I think these should be considered historical, and not have a strong bearing on how images are defined, and what terminology is used.

As a use case, it should be possible to define an image as a collection of disjoint archives. Say I have a statically linked server binary, and a tree of files for it to serve. I would like to specify an image as combining these two, with neither the parent of the other. The ideal here would be to allow

• changing the binary without needing to redistribute the data
• changing the data without needing to redistribute the binary

In this setup, I'd find the term "layer" confusing.