We need to define how rkt knows how all of the processes running in a given stage1 have been destroyed and that the root filesystem can be cleaned up.

App Container Filesystem = afs
App Container (fileset) Image = aci

Do %s%afs%aci%g

Let's ensure that we don't have a nasty case of deja vu

https://github.com/docker/docker/blob/master/pkg/archive/archive.go#L311

How does the metadata service work with EC2? EC2 offers a metadata service on the same URI. I'd probably still want to access the EC2 metadata service from a container.

Currently ACI's can be compressed with xz or bzip2 but actool build can only do gzip compression because Go libraries don't exist for xz or bzip2.

This feature would require implementing these specs in pure-go. Shelling out or linking to a C library won't fix this bug.

Currently we simply use a fixed ID for every container.

Implement rkt gc to garbage collect containers based on a lifecycle policy.

We should be able to run applications such as:

• fleetd - #399
• etcd
• dockerd - #638
• flanneld - #389

Use the store to find all of the apps that are needed for the run and setup the stage2 filesystems automatically.

Move this repo to coreos/rocket. I will do this sometime Saturday.

rkt $ ./build 
Building actool...
Building ACE validator...
Building init (stage1)...
Packaging init (stage1)...
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix 

On Linux systems mktemp does not require a template, but OS X does. Should we even bother to fix this?

https://github.com/coreos/rocket/blob/master/app-container/SPEC.md#container-runtime-manifest

States that all apps in a container share the same mount namespace. But then goes on to talk about how volumes are mounted into specific apps.

Apps sharing the mount namespace seems really hard to get right, so I'm assuming that different apps in the same container have different mount namespaces?

Will it be possible to pipe to the standard input of a container? Docker supports this right now.

We need a datastore that can hold on to the app name secondary indexes for an image and deploy a given image to a new directory on request.

To get something going, rkt should setup a container on a private network plugged into a linux-bridge. rkt should allocate a bridge if one is not already present (rkt0), assign it a well known IP network (e.g. 10.111.0.0/16) and give out random IP addresses to individual containers.

This requires putting in systemd-networkd into the stage1 rootfs and generating a .network file.

For example if the user has example.com/project/subproject and we first try example.com/project/subproject but don't find a meta tag then try example.com/project then try example.com.

AC images can have dependent images however we don't have a library for finding, fetching and assembling these dependent images.

• Add a library to fetch dependent sets
• Add a library that given all of the images are in the store renders the correct filesystem to disk

sha1 shouldn't be used for new applications. Move everything to sha256. I will update the spec.

• Make the metadata service socket activate on some socket
• Teach rkt how to register a UUID with the service (perhaps not attempting if the /var/run/rktmetaadmin.sock unix socket doesn't exist so we can fall back)
• ????

Build and host a stage1 on a public HTTP server so we can share it and not build it everytime. We can use the storage at developer.storage.core-os.net for now.

Take an already mounted and setup stage1 and stage2 filesystem and run it. The basic steps involve:

• Generating unit files from the container manifest
• Placing unit files into the stage1 filesystem
• Exec'ing systemd nspawn

This should all work as a service file and be cleaned up.

It would be nice to run rkt fetch as a non-privileged user and then launch the container with zero outbound connections. We need to implement something like git's core.sharedRepository/setgid thing.

We will want to be able to socket activate applications in a rkt container. Investigate how both tcp and unix socket activation works with systemd-nspawn.

http://0pointer.de/blog/projects/socket-activated-containers.html

One key requirement for a container to publish or register itself with another service that is outside or independent of the container architecture is to provide your public IP address(es) and port number(s).

I don't see any provision for those mappings to be available via the metadata service.

For instance, Amazon provides /network/interfaces/macs/mac/public-ipv4s as a metadata resource (aws docs) for their public facing elastic-ip addresses.

To validate directory layouts.

Prototype and document how to setup a container with systemd-nspawn that can talk to and uniquely identify a container by its incoming IP address.

We already have an index on the remote URL. Having an index on the name would be great too.

It checks if the /run/systemd/system directory exists (lstat) and fails, simply mkdir -p this path on a non-sysd host and things work fine.

Maybe we can just LD_PRELOAD a shim in stage1 which spoofs success on lstat of that path, rather than vendoring systemd to hack on systemd-nspawn.

In specific, xz is fantastic using lzma/lzma2. The Linux kernel uses xz, bzip2, and gzip. Notice how much smaller the xz produced artifacts are?

The GNOME Project, of which I'm a Sysadmin Alum, stopped using gzip and only use bzip2 + xz for all of their most recent releases due to the space it saved.

The following command works:

docker run -v $src:/opt/rocket -i -t google/golang /bin/bash -c "apt-get install -y cpio squashfs-tools realpath && cd /opt/rocket && go get github.com/jteeuwen/go-bindata/... && ./build"

We need to figure out the production solution for rkt networking:

• What networking options we will support. E.g. linux bridge, veth, ovs, macvlan, ipvlan, host?
• How do we assign IP addresses (ourselves, host systemd-dhcpd, network DHCP server)?
• How do we integrate with flannel?
• How pluggable is the solution (Docker has no shortage of requests for pluggable networking solution)?

For example try:

actool build --help

@eyakubovich Can you help write some docs on running the system wide metadata service. Ideally it would talk about how to run it, etc.

rkt run docker://localhost:5000/mything should work. This depends on #142.

actool currently doesn't gzip the aci's after build, this would be a great feature to add.

Collected notes as I read through. Sorry for the length. If any of these become worth discussing, we can fork to different topics.

This changes the established naming from what Docker calls a container to what Kubernetes calls a pod. I think that changing the meaning of "container" at this point might be detrimental to the overall comprehension of the system. I would propose to keep container to mean what you call app-container and define a different word for a set of containers.

Example use case talks about "puts them into its local on-disk cache" and "extracts two copies". This sets off immediate alarm bells for me - disk IO is the single most contended resource, in our experience. To be successful, this spec really must be implementable with a minimum of disk IO. For example, I should be able to mount a pre-built cache of images and satisfy container run requests. That's not to say that disk IO can not satisfy the spec, but it must not be a requirement.

SIGTERM should be just one kind of termination signal

Files in an images "must maintain all of their original properties" - can this include capabilities?

One thing Docker does well is differentiate WHAT to run from HOW to run it. Does this address that idea? There's some overlap in lifecycle stuff, but some other things like resources are clearly dependent on how a container will be used. How about command line flags? Being able to take a pre-built container, such as ubuntu, and run things in it without creating and pushing a new container is powerful.

That rkt is not a daemon is very similar to what we were pushing with lmctfy. However, there are things that (currently) are hard to do without any daemon - an example we run up against is prioritized OOM handling. This spec should carefully consider the strata of the overall system and how some things cross between them.

Volumes are under-specified. Can I mount them at different places in each app? Can I not mount them in some apps?

Network: Does each APP get a network or each container? I'm a bit unclear on the naming and distinction between container and app. Does rkt provide "out of the box" network at all? Docker's got this part pretty well, even if it is terribly slow.

AC_APP_NAME - what does "the entrypoint that this process was defined from" mean?

Isolators: Who do I have to beg to NOT reinvent this, and instead use something derived from LMCTFY? We have captured YEARS of development in LMCTFY's concepts. For example, exposing CPU shares is a mess. If there are things about LMCTFY's structures that don't work, let's iron those out instead..

You say "if the ACE is looking for example.com/reduce-worker-1.0.0 it will request: https://example.com/reduce-worker?ac-discovery=1". What principle lead to some piece of software knowing where to split that string?

You make some remark about /etc/hosts being assumed present and parsed by libc. Does this mean you won't provide /etc/hosts, /etc/resolv.conf, etc? That seems like a bad idea.

meanYou say config-drives "make assumptions about filesystems which are not appropriate for all environments" - can you explain? Why is it not sufficient to say that config can also be found in a volume, if the user so prefers? The host environment should be able to provide that.

"name": "example.com/reduce-worker-1.0.0" - why is version just jammed in there? Or are you trying to say "version is opaque" and if you want it, you have to embed it in the name?

Can we define user and group as name strings, not as numerics (and why are they string numbers?)

Why do you need a private network flag? This hasn't really answered how networking will work. This has been a huge PITA with Docker because EVERYONE has a different idea of what they want in networks. You can NOT support flags for all of them, so I'd argue to support none of them and instead define that as a plugin, and ship some examples of network plugins but leave it open.

It would be interesting to make the app-container spec inter-op with docker registries. After implementing fileset dependencies (#140) we can write a tool that converts layers into filesets. The pkg should have an entry point where the user can start with a docker registry URL and tag (e.g. quay.io/coreos/etcd:v0.4.6) then:

• Talk to the registry API and fetch the layers
• Download the layers and convert them to filesets (How do we label these filesets? name={url},layer={layerid})
• For the root layer use the version as the final tag

I hate to be "that guy" but I would like to know what the plan is here in terms of ownership, governance, and the trademark, although I'm skeptical you could actually trademark "rocket" :P

The post announcing this project cited some divergences in philosophy from Docker (the project and the company). It would be good to know if this project intends to be a pure community effort or if this is just CoreOS disagreeing with Docker.

It would be great to see a contribution policy that laid out how people get involved in the project and contribute to decision making and how contentious issues like the ones that have coalesced in Docker that lead to this project would be resolved. If not it would seem like people are just trading Docker (the company) for CoreOS (the company).

We have two annoying options to get around this:

1. Chroot into the stage1 which makes doing bind mounts more annoying

2. Statically compile the stage1 so it doesn't depend on the host. Also patch out the stupid logic to check /run/systemd/system and talk to dbus to get the machine-id.

/cc @eyakubovich Where is that machine-id thing you were talking about? Can you add some more info on this?

This should be a straight forward conversion from the ports list to loading in socket units.

Built and uploaded on some public http server and runnable under rkt.

This is annoying, get rid of it:

Timezone UTC does not exist in container, not updating container timezone.

Hello,

I got to

Validate the application manifest

$ actool validate manifest.json
manifest.json: valid AppManifest

in the getting started guide... and I got:

No command 'actool' found, did you mean:
 Command 'atool' from package 'atool' (universe)
 Command 'autool' from package 'nas-bin' (universe)
actool: command not found

Support doing rkt run ./mylocal.aci

Use this to append a stage1 to the rkt binary and take over the /var/lib/rkt folder. https://github.com/jteeuwen/go-bindata

I've read the spec. And I like Rocket! Thank you! I already see how it would solve a couple of problems I had with Docker.

E.g. I wasted quite some time trying to setup a private Docker Registry when it was still new. It was nearly impossible to do so a year ago. In contrast, your idea of registry is so simple and transparent.

Another issue I had with Docker is all-ports-are-closed-by-default and always-private-network philosophy. It made it much more difficult to experiment. And made some things nearly impossible to do. E.g. SIP/RTP. In contrast, you specified firewall to be optional.

(I don't know it these problem are solved in current Docker. I've stopped following it.)

By the way, if you are interested, here's what I used Docker for:

• My "GSM on the table" post, about my open source GSM experiments. It's in Russian, but there are pictures and shell commands: http://habrahabr.ru/post/213845/
• Accompanying repo (in English): https://github.com/shamrin/osmonitb-docker

Notice how clumsy it was to configure RTP port range.

(I'll experiment with Rocket in a few days.)

By default launch a "local only" metadata service. We can use the AC_METADATA_URL environment variable to make this work on localhost if needed.