Spec first read feedback about rkt HOT 21 CLOSED

Hi Tim, can you link a reference about what lmctfy isolators architecture is like?

from rkt.

https://github.com/google/lmctfy

https://github.com/google/lmctfy/blob/master/include/lmctfy.proto

On Mon, Dec 1, 2014 at 8:43 PM, Timothy Chen [email protected]
wrote:

Hi Tim, can you link a reference about what lmctfy isolators architecture
is like?

Reply to this email directly or view it on GitHub
#151 (comment).

from rkt.

I'm seriously for keeping current terminology: Keep "container" being a single set of namespaces and context in which to run processes. Another term for a set of these.

from rkt.

+1 for keeping container as a set of namespaces.

from rkt.

The naming problem as I see it is this: if you define container as a set of
namespaces, then app containers are not really containers, or did I miss
that they can set up recursive namespaces? And if containers can hold
other containers, is that properly recursive (within some set of bounds)?
If so, what makes the 2-level definition here special?

Within Google (for full disclosure, of course) we make heavy use of
containers in a recursive way - apps can define arbitrary (again, within
limits) sub-containers. I think any spec should include this capability.
At the same time, we define 2 levels of "special" in Kubernetes (as this
spec does) because our experience shows 2 levels to be the most commonly
useful nesting.

The difference is that we have semantic differences between our two-level
"adminstrative containers" and user subcontainers. There are things that
users can not do. Because of this, we don't describe the two-level
construct as two levels of container.

I guess I am arguing for some principled definition of container and what
it means for a container to hold another container. Questions that pop up.

• Can I run a process in the outer-most container, or is it special? Does
  it have a chroot?
• Can I spec manifests inside app containers that spawn other sub-app
  containers?

It feels to me like the outer container is really ONLY a list of apps to
run and a spec of how to run them, whereas inner containers are an actual
spec of what to run AND how to run it. They share the HOW, but not the
WHAT. Are they really the same?

On Tue, Dec 2, 2014 at 6:41 AM, Ian Babrou [email protected] wrote:

+1 for keeping container as set of namespaces.

Reply to this email directly or view it on GitHub
#151 (comment).

from rkt.

Do you call directories something different because they can contain subdirectories?

from rkt.

so glad to see the k8s team here 👍

from rkt.

@thockin

define a different word for a set of containers

• +1 Container Set, Pod, my personal favorite "wolf pack" ;-)

Files in an images must maintain all of their original properties, can this include capabilities?

• I find this to be a double edged sword, users are bad at this, but it would be nice to provide some default.

However, there are things that (currently) are hard to do without any daemon - an example we run up against is prioritized OOM handling.

• IMHO this is a cluster/resource managers job.

+1 re: extraction.

from rkt.

No, I would not call a directory something different because a directory IS
a recursive definition.

But you don't call a tarfile a directory, even though it can hold
directories, and sometimes you can pretend it is a directory - because it
is different.
On Dec 2, 2014 10:08 AM, "Mark Lamourine" [email protected] wrote:

Do you call directories something different because they can contain
subdirectories?

Reply to this email directly or view it on GitHub
#151 (comment).

from rkt.

Re resource manager's job: I agree, but there is a lot of overlap between
strata, if you want to provide very strong guarantees of isolation.
On Dec 2, 2014 11:53 AM, "Timothy St. Clair" [email protected]
wrote:

@thockin https://github.com/thockin

define a different word for a set of containers

• +1 Container Set, Pod, my personal favorite "wolf pack" ;-)

Files in an images must maintain all of their original properties, can
this include capabilities?

• I find this to be a double edged sword, users are bad at this, but
  it would be nice to provide some default.

However, there are things that (currently) are hard to do without any
daemon - an example we run up against is prioritized OOM handling.

• IMHO this is a cluster/resource managers job.

Reply to this email directly or view it on GitHub
#151 (comment).

from rkt.

Example use case talks about "puts them into its local on-disk cache" and "extracts two copies". This sets off immediate alarm bells for me - disk IO is the single most contended resource, in our experience.

Disk usage is more important to me, but we're both going to be hugely impacted by this.

I'd specifically call out Revsiting How We Put Together Linux Systems as the ideal I'd like to see well supported in Rocket: each system points-to/shares a read only /usr system image, a read-only /etc configuration state, and perhaps some read-only /opt/foo apps, and has it's own r/w /var mounted up to make a system.

The repository of "images" becomes nothing more than a bunch of different /usr trees in this scenario, which is ideal for disk usage, and disk-io/fs-caching.

All the above goes to +1:

Volumes are under-specified. Can I mount them at different places in each app? Can I not mount them in some apps

from rkt.

Directory is not inherently recursive, it's defined as recursive. There's no reason a container couldn't contain a sub-container (so long as the nesting is proper in the mathematical sense) either metaphorically or technically. There may be other reasons to use a different term, but that isn't one.

from rkt.

Sorry, I don't follow.

A directory is recursive in that every operation supported on dir A/ is
also supported on A/B/ and A/B/C/ ... (assuming no change of filesystem,
which is a notoriously awkward thing). Unless Container is defined in a
similar way, it's not recursive and you have two different concepts.

Will the runtime environment support arbitrarily nested containers? Can I
run processes in the outermost container the same as the innermost
container? I'll argue it should not (as evidenced from kubernetes design)
but given that, it should not share a name.

On Tue, Dec 2, 2014 at 1:12 PM, Mark Lamourine [email protected]
wrote:

Directory is not inherently recursive, it's defined as recursive. There's
no reason a container couldn't contain a sub-container (so long as the
nesting is proper in the mathematical sense) either metaphorically or
technically. There may be other reasons to use a different term, but that
isn't one.

Reply to this email directly or view it on GitHub
#151 (comment).

from rkt.

Ok we're mixing metaphors and technical definitions.

The metaphor of a directory (phone book) is not recursive. (folder is, but that's not the term we use)
The technical definition of a file system directory is recursive

The metaphor of a container is recursive. Containers can contain other containers

The technical definition of a container within the software space is, as yet, undefined, that's what we're discussing. You commented that google uses sub-containers. So great. Let a container be the definition of a view which a process will see as defined by the kernel namespaces and other contextual settings. So there's still no reason that a container could be defined in terms of another container (a sub-container). It's not the current convention but it makes more sense to me than "app contaiiner" as opposed to some other kind of container. "app container" has it's own problems in that it implies a definition of an "app" which is to be contained, but is not yet defined.

from rkt.

just to touch on a couple of @thockin's original points:

You say "if the ACE is looking for example.com/reduce-worker-1.0.0 it will request: https://example.com/reduce-worker?ac-discovery=1". What principle lead to some piece of software knowing where to split that string?

"name": "example.com/reduce-worker-1.0.0" - why is version just jammed in there? Or are you trying to say "version is opaque" and if you want it, you have to embed it in the name?

These are bugs/inconsistencies in the text of the spec; "version" is a fully separate entity from the name, and essentially just another label like arch and os, so it should no longer be embedded in the string. I'm cleaning up the spec now to reflect this.

from rkt.

Have done what I mentioned in #151 (comment) and split out version as a first-class citizen in the app manifest along with {name, os, arch}.

AC_APP_NAME - what does "the entrypoint that this process was defined from" mean?

Cleared this up in the spec.

from rkt.

Hey Tim-

On Mon, Dec 1, 2014 at 6:25 PM, Tim Hockin [email protected] wrote:

Collected notes as I read through. Sorry for the length. If any of these become worth discussing, we can fork to different topics.

Thanks for the detailed feedback. I am going to reply-inline and also
cc rocket-dev. Perhaps we could continue discussion there since there
is so much to cover?
https://groups.google.com/forum/#!forum/rocket-dev

This changes the established naming from what Docker calls a container to what Kubernetes calls a pod. I think that changing the meaning of "container" at this point might be detrimental to the overall comprehension of the system. I would propose to keep container to mean what you call app-container and define a different word for a set of containers.

We talked this over a ton with different folks while working on the
naming. Unfortunately, no one had a better alternative. Things that
were suggested:

• Open Container - Open-container is a law to prevent drinking and
  driving in the US
• Distributable Container - It is a real mouthful and borderline meaningless
• Service Container - it might be a map-reduce job and not a service
• Package Container - eh, makes you think of rpm/deb

Essentially we want a name for a container that isn't a lightweight VM
but still runs multiple processes. Having the word container is
important since alot of people think in that term. I would love a noun
that works for everyone but I am short on ideas and to some degree
people understand the words "Application Container" already.

Example use case talks about "puts them into its local on-disk cache" and "extracts two copies". This sets off immediate alarm bells for me - disk IO is the single most contended resource, in our experience. To be successful, this spec really must be implementable with a minimum of disk IO. For example, I should be able to mount a pre-built cache of images and satisfy container run requests. That's not to say that disk IO can not satisfy the spec, but it must not be a requirement.

The "extracts copy" behavior is only for people with systems that
don't have an overlay filesystem option and is left up to the
implementation to make more efficient. Once Linux Kernel 3.18 lands
everyone will have access to a reasonable overlayfs.

SIGTERM should be just one kind of termination signal

What other signals should be sent to a UNIX process as it shuts down?
Do you want a lever on this?

Files in an images "must maintain all of their original properties" - can this include capabilities?

Hrm, am I about to be disappointed that tar doesn't encode capabilities?

One thing Docker does well is differentiate WHAT to run from HOW to run it. Does this address that idea? There's some overlap in lifecycle stuff, but some other things like resources are clearly dependent on how a container will be used. How about command line flags? Being able to take a pre-built container, such as ubuntu, and run things in it without creating and pushing a new container is powerful.

I am confused on what you are getting at here. Are you thinking about
appending arguments to the exec at runtime? That is certainly
something we should add but forgot to specify.

That rkt is not a daemon is very similar to what we were pushing with lmctfy. However, there are things that (currently) are hard to do without any daemon - an example we run up against is prioritized OOM handling. This spec should carefully consider the strata of the overall system and how some things cross between them.

I agree we need to consider these things.

Are you talking about re-prioritizing the OOM based on runtime
priority? In that case I can see two options:

1. Say that the container isolators can optionally be updated at runtime.
2. Punt on the issues and say that the init system can poke at the
   container after it has started however it wishes.

Volumes are under-specified. Can I mount them at different places in each app? Can I not mount them in some apps?

Acknowledged, I have some outstanding things that didn't make it into
the markdown'd version on Monday. Uploaded here:
#182

Network: Does each APP get a network or each container? I'm a bit unclear on the naming and distinction between container and app. Does rkt provide "out of the box" network at all? Docker's got this part pretty well, even if it is terribly slow.

AC_APP_NAME - what does "the entrypoint that this process was defined from" mean?

Sorry, this is a leftover from an early name bike-shedding. I will fix it up.

Isolators: Who do I have to beg to NOT reinvent this, and instead use something derived from LMCTFY? We have captured YEARS of development in LMCTFY's concepts. For example, exposing CPU shares is a mess. If there are things about LMCTFY's structures that don't work, let's iron those out instead..

ABSOLUTELY! I would love to have help on the schemas and names here.
Can you link directly.

You say "if the ACE is looking for example.com/reduce-worker-1.0.0 it will request: https://example.com/reduce-worker?ac-discovery=1". What principle lead to some piece of software knowing where to split that string?

Again, a stupid thing we didn't fix before release. Fixed here:
https://github.com/coreos/rocket/pull/177/files

You make some remark about /etc/hosts being assumed present and parsed by libc. Does this mean you won't provide /etc/hosts, /etc/resolv.conf, etc? That seems like a bad idea.

I would like to not provide them by default but have a way for an app
to opt-in to them being generated.

meanYou say config-drives "make assumptions about filesystems which are not appropriate for all environments" - can you explain? Why is it not sufficient to say that config can also be found in a volume, if the user so prefers? The host environment should be able to provide that.

This is just a weak argument that should be deleted. The primary
motivation for the metadata service is the identity.

"name": "example.com/reduce-worker-1.0.0" - why is version just jammed in there? Or are you trying to say "version is opaque" and if you want it, you have to embed it in the name?

Fixed as I mentioned above. We will make os, arch and version labels
on the image and users can add their own too.

Can we define user and group as name strings, not as numerics (and why are they string numbers?)

How would we support string user and group names?

Why do you need a private network flag? This hasn't really answered how networking will work. This has been a huge PITA with Docker because EVERYONE has a different idea of what they want in networks. You can NOT support flags for all of them, so I'd argue to support none of them and instead define that as a plugin, and ship some examples of network plugins but leave it open.

What are you referring to here in particular? I completely agree about
doing network namespace plugins and leaving it open. Eugene is working
on the spec for rkt on that one. I don't think we need to define the
network plugins for the app container spec beyond saying an L3 network
interface with a correctly configured IP.

Thanks again!

Brandon

from rkt.

I don't know where phone book came from - "directory" for this conversation
has always meant a filesystem (only GUIs call them folders :)

If a container can contain containers, then this spec should not
distinguish between types of containers. It should instead spec what a
container is and what it means to run an image in a container. Any
container.

Now to speak out the other side of my mouth. Allowing arbitrary nesting in
a config adds a lot of complexity, so it had better be very valuable. In
our experience, The vast majority of use cases were satisfied by 2 levels,
which is what we captured in the design of kubernetes pods. We could not
justify arbitrary recursion's complexity in config (though it is certainly
needed at runtime).

Tim

On Tue, Dec 2, 2014 at 3:41 PM, Mark Lamourine [email protected]
wrote:

Ok we're mixing metaphors and technical definitions.

The metaphor of a directory (phone book) is not recursive. (folder is, but
that's not the term we use)
The technical definition of a file system directory is recursive

The metaphor of a container is recursive. Containers can contain other
containers

The technical definition of a container within the software space is, as
yet, undefined, that's what we're discussing. You commented that google
uses sub-containers. So great. Let a container be the definition of a view
which a process will see as defined by the kernel namespaces and other
contextual settings. So there's still no reason that a container could be
defined in terms of another container (a sub-container). It's not the
current convention but it makes more sense to me than "app contaiiner" as
opposed to some other kind of container. "app container" has it's own
problems in that it implies a definition of an "app" which is to be
contained, but is not yet defined.

Reply to this email directly or view it on GitHub
#151 (comment).

from rkt.

On Tue, Dec 2, 2014 at 7:05 PM, Brandon Philips
[email protected] wrote:

Hey Tim-

On Mon, Dec 1, 2014 at 6:25 PM, Tim Hockin [email protected] wrote:

Collected notes as I read through. Sorry for the length. If any of these become worth discussing, we can fork to different topics.

Thanks for the detailed feedback. I am going to reply-inline and also
cc rocket-dev. Perhaps we could continue discussion there since there
is so much to cover?
https://groups.google.com/forum/#!forum/rocket-dev

This changes the established naming from what Docker calls a container to what Kubernetes calls a pod. I think that changing the meaning of "container" at this point might be detrimental to the overall comprehension of the system. I would propose to keep container to mean what you call app-container and define a different word for a set of containers.

We talked this over a ton with different folks while working on the
naming. Unfortunately, no one had a better alternative. Things that
were suggested:

• Open Container - Open-container is a law to prevent drinking and
  driving in the US

LOL - this is a great name.

• Distributable Container - It is a real mouthful and borderline meaningless
• Service Container - it might be a map-reduce job and not a service
• Package Container - eh, makes you think of rpm/deb

Essentially we want a name for a container that isn't a lightweight VM
but still runs multiple processes. Having the word container is
important since alot of people think in that term. I would love a noun
that works for everyone but I am short on ideas and to some degree
people understand the words "Application Container" already.

What's wrong with simply qualifying them both, if they are really
defined as the same thing. Group container vs App container. Stem
container and leaf container. Outer container and inner container.

Or adopt kubernetes names and run with pod & container?

Example use case talks about "puts them into its local on-disk cache" and "extracts two copies". This sets off immediate alarm bells for me - disk IO is the single most contended resource, in our experience. To be successful, this spec really must be implementable with a minimum of disk IO. For example, I should be able to mount a pre-built cache of images and satisfy container run requests. That's not to say that disk IO can not satisfy the spec, but it must not be a requirement.

The "extracts copy" behavior is only for people with systems that
don't have an overlay filesystem option and is left up to the
implementation to make more efficient. Once Linux Kernel 3.18 lands
everyone will have access to a reasonable overlayfs.

I'd like to see the spec describe it more generically, then.

SIGTERM should be just one kind of termination signal

What other signals should be sent to a UNIX process as it shuts down?
Do you want a lever on this?

HTTP get. Exec. Just to name a few.

Files in an images "must maintain all of their original properties" - can this include capabilities?

Hrm, am I about to be disappointed that tar doesn't encode capabilities?

probably :(

One thing Docker does well is differentiate WHAT to run from HOW to run it. Does this address that idea? There's some overlap in lifecycle stuff, but some other things like resources are clearly dependent on how a container will be used. How about command line flags? Being able to take a pre-built container, such as ubuntu, and run things in it without creating and pushing a new container is powerful.

I am confused on what you are getting at here. Are you thinking about
appending arguments to the exec at runtime? That is certainly
something we should add but forgot to specify.

Adding args (or redefining them entirely) is one facet. But also
things like what UID to run as and how much memory or CPU to request
are situational. If Dockerfiles included a "CPU" keyword, they would
be much less useful. By way of example: if there is a mysql
container, the person who builds and packages the container image has
no idea whether I want to run a tiny test or a production webservice.

That rkt is not a daemon is very similar to what we were pushing with lmctfy. However, there are things that (currently) are hard to do without any daemon - an example we run up against is prioritized OOM handling. This spec should carefully consider the strata of the overall system and how some things cross between them.

I agree we need to consider these things.

Are you talking about re-prioritizing the OOM based on runtime
priority? In that case I can see two options:

1. Say that the container isolators can optionally be updated at runtime.
2. Punt on the issues and say that the init system can poke at the
   container after it has started however it wishes.

What I meant is this. Consider two containers. Container P is a prod
server which needs high QoS. Container B is a batch app that wants
low cost and accepts low QoS. Somehow the system becomes OOM. the
OOM killer has to kill something. We want a deterministic result -
ALWAYS kill B before P.

But this is hard to do without some agent in userspace. We might be
able to keep pushing on the kernel to give us better APIs, but without
that we need to consider how to get correct behavior.

This goes a bit to the idea of layers and guarantees. The bottom-most
layer of the container stack, which I think rocket is targetting,
should guarantee correct, though maybe not optimal behavior.

Volumes are under-specified. Can I mount them at different places in each app? Can I not mount them in some apps?

Acknowledged, I have some outstanding things that didn't make it into
the markdown'd version on Monday. Uploaded here:
#182

I'll have to re-re-read this spec, won't I? :)

Network: Does each APP get a network or each container? I'm a bit unclear on the naming and distinction between container and app. Does rkt provide "out of the box" network at all? Docker's got this part pretty well, even if it is terribly slow.

AC_APP_NAME - what does "the entrypoint that this process was defined from" mean?

Sorry, this is a leftover from an early name bike-shedding. I will fix it up.

Isolators: Who do I have to beg to NOT reinvent this, and instead use something derived from LMCTFY? We have captured YEARS of development in LMCTFY's concepts. For example, exposing CPU shares is a mess. If there are things about LMCTFY's structures that don't work, let's iron those out instead..

ABSOLUTELY! I would love to have help on the schemas and names here.
Can you link directly.

You say "if the ACE is looking for example.com/reduce-worker-1.0.0 it will request: https://example.com/reduce-worker?ac-discovery=1". What principle lead to some piece of software knowing where to split that string?

Again, a stupid thing we didn't fix before release. Fixed here:
https://github.com/coreos/rocket/pull/177/files

You make some remark about /etc/hosts being assumed present and parsed by libc. Does this mean you won't provide /etc/hosts, /etc/resolv.conf, etc? That seems like a bad idea.

I would like to not provide them by default but have a way for an app
to opt-in to them being generated.

I think that you need to think hard about not being LSB-ish. Like it
or not a great many apps (approximately all of them) use some form of
libc-derived logic for things like name resolution. It might be cute
to opt OUT of standard resolution config files, but I would not
default to that mode. For example, in adding DNS to kubernetes, I
have to tell each container what its nameserver is and what DNS
domains to search. How will I do that for apps that don't have
standard configs.

Ditto for /proc and /sys anbd /dev/shm and /dev/pts and (.....) -
there are too many standard libs that expect these things to be
present - if you exclude them you will exclude the majority of
existing apps, which will be disastrous.

You say config-drives "make assumptions about filesystems which are not appropriate for all environments" - can you explain? Why is it not sufficient to say that config can also be found in a volume, if the user so prefers? The host environment should be able to provide that.

This is just a weak argument that should be deleted. The primary
motivation for the metadata service is the identity.

"name": "example.com/reduce-worker-1.0.0" - why is version just jammed in there? Or are you trying to say "version is opaque" and if you want it, you have to embed it in the name?

Fixed as I mentioned above. We will make os, arch and version labels
on the image and users can add their own too.

Can we define user and group as name strings, not as numerics (and why are they string numbers?)

How would we support string user and group names?

I'm not sure on this one. If you assume that the environment has a
passwd list that includes defined user names, then it can look up
UIDs. But maybe that's not a good assumption..

Why do you need a private network flag? This hasn't really answered how networking will work. This has been a huge PITA with Docker because EVERYONE has a different idea of what they want in networks. You can NOT support flags for all of them, so I'd argue to support none of them and instead define that as a plugin, and ship some examples of network plugins but leave it open.

What are you referring to here in particular? I completely agree about
doing network namespace plugins and leaving it open. Eugene is working
on the spec for rkt on that one. I don't think we need to define the
network plugins for the app container spec beyond saying an L3 network
interface with a correctly configured IP.

The first part was the isolator flag for "private network" - why? The
more you toss into the spec, the less likely you are to see valid
implementations.

Thanks again!

Brandon

from rkt.

One thing Docker does well is differentiate WHAT to run from HOW to run it. Does
this address that idea? There's some overlap in lifecycle stuff, but some other
things like resources are clearly dependent on how a container will be used.
How about command line flags? Being able to take a pre-built container, such as
ubuntu, and run things in it without creating and pushing a new container is
powerful.

I am confused on what you are getting at here. Are you thinking about
appending arguments to the exec at runtime? That is certainly
something we should add but forgot to specify.

Adding args (or redefining them entirely) is one facet. But also
things like what UID to run as and how much memory or CPU to request
are situational. If Dockerfiles included a "CPU" keyword, they would
be much less useful. By way of example: if there is a mysql
container, the person who builds and packages the container image has
no idea whether I want to run a tiny test or a production webservice.

Thats true, however what about defining WHAT a containerized application really needs
to run. Application expect some capabilities or resources to run. I.e. there is minimom
of memory a application needs to run otherwise it simply not start or even worst has
unexpected behavior. Or lets say a firewall application asks for at least two (virtual)
interfaces. In the past days Apps showed on the box the minimum requirements, or
apps ask you to start a JVM with args to increase the memory heap. etc etc.

In our experiences import in OVA Images into a Supervisor works like this, the OVA
defines the requirements and the import step maps this to capabilities while instantiating
the OVA.

just my toughs.

from rkt.

This issue was moved to appc/spec#12

from rkt.