Use data container instead of bind mount? about ceph-container HOT 13 CLOSED

For which component are you talking?

I believe I tried this in the past, with RBD mounts, and it didn't work... something to do with Docker not exporting mounts, I believe.

For configs, yes, I think the way forward is with etcd (or abstracted via confd to support consul, too). Also, though, I don't think there is anything to stop someone from using a data container with --volumes-from instead of a bind mount. There's nothing restricting that from the container side, anyway.

from ceph-container.

I was thinking about the config.

from ceph-container.

We can test it, but I don't believe there should be anything to preclude users from doing exactly this. If so, we can just add it to the documentation.

from ceph-container.

@leseb I was thinking of the data container for /var/lib/* files not /etc directories. IMO for config files, you want to pass those via the environment and configure the /etc files in the container via a start.sh script. If you don't do these things, you lose the idempotency, immutability, and declarative behavior of containers.

For example, someone could go and change a config value in /etc, restart the container, and the container may no longer start. Bam, you just nuked immutability. Bam, you just nuked declarativeness. An example of how bindmounds damage idempotency is more difficult to explain, but ideally you dont want to use bindmounts except for required things (like /sys/fs/cgroups, /lib/modules, etc) and you want to bring them into the container :ro as read only.

from ceph-container.

@sdake Do you mean the /var/lib/ceph/*, such as the storage filesystem for an OSD? Or other, non-ceph, arbitrary /var/lib/* directories?

from ceph-container.

@Ulexus /var/lib/ceph should be in a a data container IMO :) For an example of how mariadb works for OpenStack, see::

https://github.com/stackforge/kolla/blob/master/docker/mariadb-data/Dockerfile

And our compose file:

https://github.com/stackforge/kolla/blob/master/compose/mariadb.yml#L12

Note you can have more then one container share the same data container. Eg:

https://github.com/stackforge/kolla/blob/master/compose/nova-compute-network.yml#L17

You don't have to use compose if you want the raw docker command line. If that is so, there is a volumes_from command iirc that you specify during run IIRC.

Note docker 1.6.0-rc2 is busted with data containers, you will have to wait to 1.6.0-rc3 if you really want them to work well which is 1-2 days away.

Regards
-steve

from ceph-container.

I would heartily agree that for something like a database, the volume should be outboard, preferably abstracted and stored on some networked storage. What I'm failing to grasp here is that, provided that ceph is a component of that networked storage itself... in the case of the OSD, it is the storage component..., what is the use-case for further abstracting (and presumably, thereby, increasing the latency of) the storage for ceph daemons?

Now, don't get me wrong. I'm all for making as flexible a solution as possible. If you want to create a ceph cluster on top of some other network storage cluster, for whatever reason, I'm fine with that, and I don't want to stand in your way. If there is something that disallows that use-case and we can easily fix it, I'll be happy to include the fix. I'm not seeing, at this point, what would preclude you from using these ceph components in just that way right now. Is this not the case?

from ceph-container.

@sdake Take for example the OSD container. Now I use a bind mount to accommodate different OSD on different disks. What would be the point of using a separate container?

from ceph-container.

@Ulexus I don't understand enough of the ceph architecture to offer an opinionated response :) I do know that bindmounts are evil because they break the declarative, immutable, and idempotency properties of containers. I know a bunch of computer-sciency jibberish - but in practice I prefer the use of volume containers. This would mean ceph would need to back /var/lib/docker on the ceph hosts. I'm not even sure if overlayfs/aufs on top of ceph would work.

The main objection I have to the current containers is bindmounting the configuration directories. I have used docker extensively for 8 months and feel like I've broken through the "apprentice" stage. From that experience, I can tell you bindmounting /etc/ceph is going to cause you nothing but pain ;)

The idea of using etcd was kicked around in Kolla as well, but it again breaks the properties described above. That is why we went with docker-compose - to abstract the complexities of docker cli for normal deployment workflows.

regards
-steve

from ceph-container.

@dmitryint I have no technical answer to your question that doesn't involve adding a layer between ceph and /var/lib/docker (such as lvm) to map /var/lib/docker to multiple storage devices. Obviously this impacts performance.

Regards
-steve

from ceph-container.

@sdake I agree with you on the /etc/ceph part. However, that should work perfectly well right now with the --volumes-from option. I actually originally wrote the system with a ceph-config container, but I found that in practice, it was a fairly vacuous, so I didn't bother posting it.

As far as "bindmount are evil": in general, this is surely true. However, the architecture of ceph, as far as I can tell, pretty well breaks any kind of machine abstraction. OSDs in particular (and they are, by far, the most numerously-populated ceph container) are designed to run with very specific configurations and IP addresses, bound very specifically to a given set of disks and journals. You can't abstract the OSDs from the underlying storage. Moreover, it is dangerous to even allow that to happen.

In many ways, therefore, it is arguable whether ceph should run on docker. In practice, many of the features are necessarily circumvented. However, in my case (and I believe in the cases of at least a few others), the deployment consistency (speaking here mainly of CoreOS, for my part) and simplicity outweigh the loss of abstractions Docker, by its design affords.

I should say, however, that Docker is continually facing an uphill battle in this regard, which is part of the reason for such militancy, with respect to bind-mounts (and other machine-tied aspects). It is almost always necessary to force people to reexamine their deployment strategies to make sure they do not throw away the design benefits of Docker in their ignorance. As such, your objection is generally valid and beneficial. I just don't see a generalized way around this problem.

(I say generalized because, of course, given sufficient constraints, you could easily construct an abstractive system from ceph of docker. I just don't see a path which is sufficiently flexible for common distribution.... which is not to say there isn't one; just that I don't see it.)

With respect to /var/lib/docker being exported monolithically, that just won't work. OSDs are designed to have a one-to-one correspondence with their underlying hardware. Specifically, each OSD should have a decoupled an I/O path as possible. Otherwise, you catastrophically reduce performance through competition.

from ceph-container.

@Ulexus Thanks for the great information. I am a total ceph noob, so I don't know how any of it works :) I was going to ping Ric Wheeler this week to get some of these questions answered, but it looks like you answered the ones I had.

Thanks!
-steve

from ceph-container.

No problem; like I said, it's a good and valid objection. I just don't see a reasonable alternative for right now. I'm closing this issue, but feel free to open it, if there is new, related information.

from ceph-container.