I'm just starting to play with the KV setup, and I am finding it's pretty rough. I'm adding this issue as much for my own notes as for anything else. Also, these are just my opinions, so feel free to provide feedback/arguments.

Problems (update the following as necessary or discovered):

• The default schema for the ceph.conf construction doesn't get populated in a usable way
• The schema for ceph.conf isn't documented
• Some segments in the KV schema are directory-separated; others are _-separated
• Some schema changes I would prefer:
  • /auth/fsid should be /fsid (IMO, this is is only tertiarily auth: it's the cluster id)
  • Keyrings should be in /keyrings; examples: /keyrings/mon, /keyrings/bootstrapOsd (maybe even /keyrings/bootstrap/osd
  • /auth/cephx_* should be /auth/cephx/* (though this seems redundant with the /globals/ section)
• If mandatory directories in the KV store don't exist, the config construction dies -- need valid defaults
• Public and Cluster networks should be stored in the KV
• It doesn't appear the bootstrap keyrings actually get added to the auth database
• I also really like the idea of merging a user-supplied, whole ceph.conf with a confd-generated one.

NOTE: Many of these would be breaking changes, so I would also like to get an idea of how many people are actually using the KV backend so that we can see if we need to add backward compatibility shims.

I have a container with mounted RBD image as /mnt/rbd volume. Inside this container, I can see the contents of /mnt/rbd. But when I try to mount it to the host filesystem mountpoint (-v /host/path/mountpoint:/mnt/rbd) or use link to that volume from other containers with the --volumes-from Docker run option, I see only an empty directory.
Correct me if I'm wrong, but I think it's Dockers issue that cannot pass the mount point inside the container through the -v or --volumes-from options?

It might be possible to reduce the size of the docker images by using a different base image such as alpine.

At the moment, ceph-osd relies on the physical ceph osd drives to be passed in via mounts which should appear under /var/lib/ceph/osd/ceph-$id

Note: The rest of this relates to running multiple OSD's in a single docker container.

Unfortunately, you can't pass a single /var/lib/ceph/osd as a host mount for all the mounts below it. And I'm thinking that it should be possible to hotswap osd's in and out on the fly without restarting the other osd's. Unfortunately, I don't see a way you can add volume mounts to an already running container without restarting it with the commandline changed. And passing in a top level /var/lib/ceph/osd results in that being mounted as tmpfs which breaks things.

Maybe another approach is to change the bootstrap init script to search for block devices that have the right uuid (4fbd7e29-9d25-41b8-afd0-062c0ceff05d), mount them temporarily and read the whoami file, then mount them as the correct place internal to the image. Then add other helpers to stop osd's, or add new ones on the fly which you can call through exec on the running ceph-osd.

I have a bash script that is already doing the osd id discovery and mounts on the outside which does that now and is based on what ceph-disk does.

If this makes any sense I'm prepared to fork this and give it a try, I'll post the results back here.

Does this seem sensible? or are there other reasons why you shouldn't mount block devices inside the container?

Spawned by discussion from #113

CRUSH location types http://ceph.com/docs/master/rados/operations/crush-map/:

type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root

I think we should offer a free-form CRUSH_LOCATION for "experts," but we should also offer each of the types explicitly (ex. CRUSH_RACK), whose purpose and use should be more easily-grasped, though such may be more verbose.

No apache required, easy to setup.
Will push something soon.

I see some errors related to hdparm missing from ceph-osd. It's not installed in the image.

Proposed by @hunter in #28 .
Discussion can continue here.

https://registry.hub.docker.com/u/ceph/config/builds_history/151300/

Not sure why, but:
buh82pyfb6yjciwqaspwweb Error 2015-03-05 00:06:05 2015-03-05 00:17:02

Could someone retrigger the build and/or fix the issue ?

Thanks

Since wget was missing, the repo's key is not present.

I know running multiple processes defeats Docker's philosophy but I believe some use cases might require a simple standalone container running all the Ceph processes.
Actually it's easier to bootstrap.

@Ulexus thoughts?

Using ceph-config, the script fails on "Attempting to pull monitor map from existing cluster." because if the cluster is down, then there is no way to retreive the map from an existing monitor, and the script stops because of -e option, leaving the configuration in a locked state.

I'm really wanting that fully automated life cycle goal that the non-docker version as far as I can tell have.

So, toward that goal I just wanted somewhere to capture some ceph-disk notes and get peoples comments. Sorry if this is a bit long winded. I wish github had a discussion type of system. Anyway, onwards...

Right now we're able to use, ceph-disk prepare as a step toward getting a ceph osd up.
I use it, because I run alongside my OSD spinning disks, 2 SSD's for journalling.

I prepare my OSD's like this:

docker run --rm --net=host --privileged=true -v /etc/ceph:/etc/ceph -v /dev:/dev ceph/base ceph-disk prepare --cluster ceph --cluster-uuid 683f638b-e2d1-40c2-969f-907d961a7d24 --fs-type xfs /dev/sda /dev/sde

Assuming my cluster-uuid is that above (gotten from my ceph.conf) and /dev/sda is my main data disk, /dev/sde is my journal disk.

Very conveniently, if I run the command again with a different data disk, and then use the same journal disk, ceph-disk will create another journal partition. In actual fact, it creates a symbolic link to the journal disk on the main osd data disk.

Where everything falls over, is with ceph-disk activate. It's just not going to work actually. The reason it's not going to work is because it expects:

1. to be running natively (not docker).
2. it auto-detects what OS you're using and what init system it is. It's most likely going to get that totally wrong. And besides that, the init system used in our images is not on the list.
3. ceph-disk activate also writes the OSD ID to the whoami file so you could move the disk elsewhere in future.

So, either ceph-disk would need to be changed to be docker aware (which would likely be very tied to this implementation), or we override it. Or jump through hoops. Or use systemd inside the image... not sure I like that idea but I've heard it's possible.

ceph-disk code is here. It's really just a python script that uses sgdisk, blkid and a few other tools to get the job done.
https://github.com/ceph/ceph/blob/master/src/ceph-disk

I think the override idea is the easiest.
As ceph-disk activate doesn't work, what I do is currently:
Assuming my OSD is /dev/sda & is osd 10

sudo mkdir -p /osd/tmp
sudo mount /dev/sda1 /osd/tmp
sudo echo 10 | tee /osd/tmp/whoami
sudo umount /osd/tmp

Which just writes the whoami file. And combine that with my mount-osds script.
If the disks ever need to be moved to another machine, then knowing in advance what disk is becomes a bit tedious. This is where I think my mount-osds is really useful.. It'll mount them to /osd/ceph-$id. It's not really entirely cluster aware but could be made to be. i.e. currently semi-hardcoded to a cluster named ceph. It's actually doing a bit of what ceph-disk does but does it in shell script so it can be run natively from CoreOS without requiring python and can run outside a container.

What I do is mount the disks by calling mount-osds before the docker ceph-osd container is started.
Or just calling it as an ExecStartPre= from a systemd service file or similar.

What I am thinking is merging some of this functionality with my ceph-tools scripts. Override ceph-disk activate to actually do the above and the bits from the bootstrap-osd, then write a startup file to /etc/service/ceph-osd/run and start it with sv start.

I'm not sure if this makes a terrible lot of sense to people yet... later I'll whip up an implementation and you can have a peek. But I'd like to see @leseb daemon changes merged to make it work off that. Anyway, those are my thoughts currently.

Would love to know how to add into systemd to call some custom udev rules, e.g. like if a ceph osd disk were moved it could run through a setup or restart process automagically. Has anyone actually done that here?

Once ceph-demo is started, radosgw responds

curl localhost
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

In order to access radosgw using API I need to create a user. Following
http://ceph.com/docs/master/radosgw/config/
we can issue

docker exec cd30ab09156 radosgw-admin user create --uid="testuser" --display-name="First User"

and then use Python code like this one

import boto
import boto.s3.connection
conn = boto.connect_s3(
    aws_access_key_id='3HYE82O7DGS49LKIY4WJ',
    aws_secret_access_key='pARGxDCQ+D3fS+s6EQjeCGWLnEhMWdbncXeB\/hQu',
    host='localhost',    
    is_secure=False,
    calling_format = boto.s3.connection.OrdinaryCallingFormat(),)

    bucket = conn.create_bucket('my-new-bucket')

Unfortunately I get

    bucket = conn.create_bucket('my-new-bucket')
  File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 590, in create_bucket
    response.status, response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden

What else am I missing?
As far as I can tell ceph-demo is no different from ceph-rgw in terms rgw default behaviour.

ceph-rest-api is one of the components of ceph which is really useful for monitoring. I think we can add it in the all-in-one ceph/demo image. The command to run ceph-rest-api looks like this.

ceph-rest-api -n client.admin

It would be better to push these images in docker hub so that we don't need to build them again 😃

According to V1 api volume section should use "hostpath" key instead of "source"->"hostdir".

http://kubernetes.io/third_party/swagger-ui/#!/api%2Fv1/createReplicationController

As we know. Radosgw container is the gateway of Ceph on CoreOS Cluster,the Radosgw container
sometimes migrate to another server . How to solve this issues?

I am testing this docker container, thank you, but I have a problem, when I run multiple instances, suddenly the conntrack table gets filled.
e.g. I run one instance, sometimes 2, and nothing wrong, also:

cat /proc/sys/net/netfilter/nf_conntrack_count

gives me less than 100
but after one or two more instances, nf_conntrack_count increases very quickly, and after a few seconds it reaches 65k, I tried increasing it to 256k but it gets filled in seconds too.

Hey,

The way the OSDs are currently setup is not straightforward at all. It's really confusing how pieces interact with each other. Thus I'm proposing to rework a bit how we bootstrap the OSD.

In #78 I started integrating bootstrap-keys, so we don't need to have the ceph.admin key inside all the containers and more importantly on all the hosts running the containers.
So yes there is a prefetching operation that needs to collect the appropriate keys.
The operation will create the following entries:

/var/lib/ceph/
|-- bootstrap-mds
|   `-- ceph.keyring
|-- bootstrap-osd
|   `-- ceph.keyring
|-- bootstrap-rgw
|   `-- ceph.keyring

Later the idea is to use ceph-disk.
So basically this is how I set an OSD (manually):

$ sudo docker run -ti -v /etc/ceph:/etc/ceph -v /var/lib/ceph/:/var/lib/ceph/ -v /dev/:/dev/ --net=host --privileged=true ceph/base /bin/bash
root@coreos:/# ceph-disk zap /dev/vdc ; ceph-disk prepare /dev/vdc ; ceph-disk activate /dev/vdc1                                                                                                                                   [33/38422]
Creating new GPT entries.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Creating new GPT entries.
The operation has completed successfully.
The operation has completed successfully.
The operation has completed successfully.
meta-data=/dev/vdc1              isize=2048   agcount=4, agsize=1304255 blks
         =                       sectsz=512   attr=2, projid32bit=0
data     =                       bsize=4096   blocks=5217019, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
The operation has completed successfully.
got monmap epoch 1
sh: 1: /sbin/hdparm: not found
2015-06-12 13:01:20.358696 7f843ec13900 -1 journal _check_disk_write_cache: pclose failed: (0) Success
2015-06-12 13:01:20.359526 7f843ec13900 -1 journal check: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected 924fd2e5-c3ea-4d7f-b3c5-9bb81abefa07, invalid (someone else's?) journal
sh: 1: /sbin/hdparm: not found
2015-06-12 13:01:20.362257 7f843ec13900 -1 journal _check_disk_write_cache: pclose failed: (0) Success
sh: 1: /sbin/hdparm: not found
2015-06-12 13:01:20.533410 7f843ec13900 -1 journal _check_disk_write_cache: pclose failed: (0) Success
sh: 1: /sbin/hdparm: not found
2015-06-12 13:01:20.536231 7f843ec13900 -1 journal _check_disk_write_cache: pclose failed: (0) Success
2015-06-12 13:01:20.536667 7f843ec13900 -1 filestore(/var/lib/ceph/tmp/mnt.0_cGmZ) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2015-06-12 13:01:20.747410 7f843ec13900 -1 created object store /var/lib/ceph/tmp/mnt.0_cGmZ journal /var/lib/ceph/tmp/mnt.0_cGmZ/journal for osd.7 fsid fe49246f-9655-4034-ae32-7ff81a416e90
2015-06-12 13:01:20.748599 7f843ec13900 -1 auth: error reading file: /var/lib/ceph/tmp/mnt.0_cGmZ/keyring: can't open /var/lib/ceph/tmp/mnt.0_cGmZ/keyring: (2) No such file or directory
2015-06-12 13:01:20.759898 7f843ec13900 -1 created new key in keyring /var/lib/ceph/tmp/mnt.0_cGmZ/keyring
added key for osd.7
root@coreos:/var/log/ceph# ceph-osd -i 7
starting osd.7 at :/0 osd_data /var/lib/ceph/osd/ceph-7 /var/lib/ceph/osd/ceph-7/journal
root@coreos:/var/log/ceph# ps x
  PID TTY      STAT   TIME COMMAND
    1 pts/1    Ss     0:00 /bin/sh /entrypoint.sh /bin/bash
    7 pts/1    S      0:00 /bin/bash
  226 ?        Ssl    0:00 ceph-osd -i 6
  599 pts/1    R+     0:00 ps x

This is pretty straightforward and easy to bootstrap.
I can easily prototype this.

@Ulexus what do you think?

On a CoreOS host I run:

core@coreos ~ $ etcdctl ls
/coreos.com

docker run --net=host -e ETCDCTL_PEERS=http://192.168.0.20:4001 -e MON_IP=192.168.0.20 -e MON_NAME=etcdmon -e CLUSTER=ceph -v /etc/ceph:/etc/ceph ceph/config
Configuration found for cluster ceph. Writing to disk.
Error: 100: Key not found (/ceph-config/ceph/ceph.conf) [30]

Not sure why the script believes that /ceph-config/ceph/done exists...
Tried to dig a bit but without much luck.

@Ulexus did you get a chance to test that new container?

I'm interested in testing how ceph can be accessed through S3 API or, failing that, some other API. Any help would be much appreciated. It seems that the ceph demo starts RGW so using it could be an easy start. What are the steps to get the ceph-demo running? Specifically, am I right assuming that ceph/config has to be run (or config created) before running ceph/demo?

Unfortunately for me ceph/config hangs on "Configuration is locked by another host. Waiting." and ceph/demo immediately exits (no config?) although I have been able to run both with limited success in the past. This is what I'm trying to run now (etcd is running on the host itself)

docker run -e ETCDCTL_PEERS=http://172.16.X.Y:4001 -e MON_IP=172.16.X.Y \
-e MON_NAME=mymon -e CLUSTER=testing -v /etc/ceph:/etc/ceph ceph/config

docker run -d --net=host -v /etc/ceph:/etc/ceph -e MON_IP=172.16.X.Y \
-e CEPH_NETWORK=172.16.0.0/16 ceph/demo

I will add an image as soon as this lands: ceph/ceph#4191

for example, if run MDS:

sudo docker run -d --net=host \
-v /var/lib/ceph/:/var/lib/ceph/ \
-v /etc/ceph:/etc/ceph \
-e CEPHFS_CREATE=1 \
ceph/daemon mds

in entrypoint.sh kviator runs with --kvstore=none argument, what causes error: Backend storage not supported yet, please choose another one

I really love the ceph/demo docker image because it provides a quick and simple way to spin up ceph for a development environment.

The only problem is that it currently treats every launch as a new cluster. I want to be able to keep my data when the docker container is terminated and then restart it at a later stage.

This is the command I am using to run the container:

sudo docker run --net=host -v /work/data/ceph/config:/etc/ceph -v /work/data/ceph/data:/var/lib/ceph -e MON_IP=192.168.33.12 -e CEPH_NETWORK=192.168.33.0/24 ceph/demo

Unfortunately, if I terminate the container and then launch it again, I get this error:

2015-07-19 06:33:20.903601 7fd39a6a9900 -1  ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-0: (22) Invalid argument

I believe this is because the entrypoint.sh file treats each launch as a new cluster and will go and create the OSDs and MDS from scratch.

If would be awesome if the script has some checks in place so that it's able to "restore" if data exists rather than create a brand new cluster.

I was speaking with @TelanJ earlier about what happens in a dynamic environment (like K8s) with multiple Mons. Cases where a Mon being started with a different hostname and/or IP.

For a quick test, 7 mon containers were started, then 2 stopped and another 2 started. The monmap contained 9 mons of which 2 were down. This would have been my expectation but I don't think it's an ideal option for live environments.

It's not easy to know if a mon is temporarily offline of it's been migrated to a different hostname or IP. Ceph also makes it very hard as updating a monmap means all mons need to be restarted (http://ceph.com/docs/master/rados/operations/add-or-rm-mons).

To throw out a few incomplete ideas:

• Manual - Entry command/arg that cleans up missing mons (doesn't handle stop/injection/starting of all mons)
• Use the KV store for tracking active mons. Removing old mons when starting a new mon. Use confd to track changed keys and stop/start the mon inside the container.
• Give mons fixed IDs (similar to OSDs). If the ID is the same then check the IP (in monmap dump?) and remove/add as needed

None are really complete solutions since there needs to be ways of ensuring quorum, injecting the updated monmaps and regenerating the ceph.conf. Any thoughts on the best way to handle the monmap?

@leseb - for the impatient (that's me ;) are you able to update the readme for those wishing to switch from ceph/mon, ceph/osd to ceph/daemon?

I take it that ceph/osd, ceph/mon etc will be removed at some point in the near future.
Also, I think it might be worthwhile using ceph/config, how would you go about transitioning to this from a static config?

I was going to set up the Docker Registry automated builds for these Dockerfiles, but I cannot do so without access to the Github Ceph org. (I do have access to the Docker Registry Ceph org.)

rebuilding mds give me this error :

E: Can't find a source to download version '2.9.2-4ubuntu4' of 'fuse:amd64'

adding apt-get update to the Dockerfile fix it.

Because I didn't ever actually get around to writing a radosgw, I was looking around at what's out there. @bobrik's image looks good to me, but he uses nginx (which I prefer, but is not reflective of what is on ceph.com). I asked him, and he's happy to supply his code.

@leseb Does that work for you, or do you want to try and keep with an Apache installation, consistent with the docs on ceph.com?

As the pre-generated ceph.conf is very basic when bootstrapping a new cluster, it's often better to supply one.

However, the entrypoint.sh scripts on the mon and daemon skip the generation of admin & monitor keys if ceph.conf exists. It'd be better to prompt for them to be created, or just create them if ceph.conf is there, but those files are missing.

So in my case I had to run the authtool directly to complete the missing steps.

I'm trying to run ceph/radosgw:firefly instead of my perfectly running bobrik/radosgw, but errors appear in /var/log/apache2/error.log:

root@958cfdfeba97:~# cat /var/log/apache2/error.log
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.18. Set the 'ServerName' directive globally to suppress this message
[Thu Mar 12 19:52:38.268056 2015] [:notice] [pid 83:tid 139648254072704] FastCGI: process manager initialized (pid 83)
[Thu Mar 12 19:52:38.268450 2015] [mpm_event:notice] [pid 67:tid 139648254072704] AH00489: Apache/2.4.7 (Ubuntu) mod_fastcgi/mod_fastcgi-SNAP-0910052141 configured -- resuming normal operations
[Thu Mar 12 19:52:38.268487 2015] [core:notice] [pid 67:tid 139648254072704] AH00094: Command line: '/usr/sbin/apache2 -D FOREGROUND'
[Thu Mar 12 19:53:20.650887 2015] [fastcgi:error] [pid 84:tid 139648168228608] (2)No such file or directory: [client 192.168.2.89:37187] FastCGI: failed to connect to server "/var/www/s3gw.fcgi": connect() failed
[Thu Mar 12 19:53:20.650955 2015] [fastcgi:error] [pid 84:tid 139648168228608] [client 192.168.2.89:37187] FastCGI: incomplete headers (0 bytes) received from server "/var/www/s3gw.fcgi"
[Thu Mar 12 19:55:41.381134 2015] [fastcgi:error] [pid 84:tid 139648159835904] (2)No such file or directory: [client 192.168.2.89:37258] FastCGI: failed to connect to server "/var/www/s3gw.fcgi": connect() failed
[Thu Mar 12 19:55:41.381162 2015] [fastcgi:error] [pid 84:tid 139648159835904] [client 192.168.2.89:37258] FastCGI: incomplete headers (0 bytes) received from server "/var/www/s3gw.fcgi"
[Thu Mar 12 19:56:29.117816 2015] [fastcgi:error] [pid 86:tid 139648168228608] (2)No such file or directory: [client 192.168.2.89:37291] FastCGI: failed to connect to server "/var/www/s3gw.fcgi": connect() failed
[Thu Mar 12 19:56:29.117890 2015] [fastcgi:error] [pid 86:tid 139648168228608] [client 192.168.2.89:37291] FastCGI: incomplete headers (0 bytes) received from server "/var/www/s3gw.fcgi"
[Thu Mar 12 19:56:37.835153 2015] [fastcgi:error] [pid 86:tid 139648159835904] (2)No such file or directory: [client 192.168.2.89:37296] FastCGI: failed to connect to server "/var/www/s3gw.fcgi": connect() failed
[Thu Mar 12 19:56:37.835180 2015] [fastcgi:error] [pid 86:tid 139648159835904] [client 192.168.2.89:37296] FastCGI: incomplete headers (0 bytes) received from server "/var/www/s3gw.fcgi"
[Thu Mar 12 19:56:47.834311 2015] [fastcgi:error] [pid 84:tid 139648168228608] (2)No such file or directory: [client 192.168.2.89:37301] FastCGI: failed to connect to server "/var/www/s3gw.fcgi": connect() failed
[Thu Mar 12 19:56:47.834339 2015] [fastcgi:error] [pid 84:tid 139648168228608] [client 192.168.2.89:37301] FastCGI: incomplete headers (0 bytes) received from server "/var/www/s3gw.fcgi"
[Thu Mar 12 19:57:15.163202 2015] [fastcgi:error] [pid 84:tid 139648159835904] (2)No such file or directory: [client 192.168.2.89:37329] FastCGI: failed to connect to server "/var/www/s3gw.fcgi": connect() failed
[Thu Mar 12 19:57:15.163229 2015] [fastcgi:error] [pid 84:tid 139648159835904] [client 192.168.2.89:37329] FastCGI: incomplete headers (0 bytes) received from server "/var/www/s3gw.fcgi"

Apparently, something is missing in my config that should be noted in README.md:

[client.radosgw.gateway]
keyring = /etc/ceph/keyring.radosgw.gateway
rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
log file = /var/log/ceph/client.radosgw.gateway.log
rgw print continue = false
rgw_dns_name = s3.ceph.something.something

Hey @Ulexus
Just to let you know, I'm on freenode and oftc as leseb. So feel free to ping me if needed.
I'm based in Paris. UTC +1
Are you on IRC?

Thanks!

I am opening this issue to ask about the current status of rbd-volume and what is planned for it in the future. I am trying to figure out a nice pattern to deliver RBDs to my user's containers. Seems that moby/moby#8854 will be the long term solution but meanwhile I need another approach.

So, why is rbd-volume tagged as NON-FUNCTIONAL? Should I give it a try?

While it works if a configuration already exists, the script bombs out if the key doesn't exist

Now that we have the Ceph namespace on Docker, it seems to me that we should collapse the name for base (and all the other components).

TODO

I'd be good to get calamari working inside docker.

I tried to once get it working by following all the instructions but got stuck in the ruby frameworks and couldn't get it going.

etcdctl get --consistent ${CLUSTER_PATH}/done 2>/dev/null >/dev/null
if [ $? -ne 0 ]; then
  echo "Configuration found for cluster ${CLUSTER}. Writing to disk."

It probably should check if it's -eq 0.

I would like to try demo docker container but I'm confused about the parameters.

Here's my error message and I have no idea how to fix it. More instruction is really welsome 😃

➜  demo git:(master) docker run -d --net=host -v /etc/ceph:/etc/ceph -e MON_IP=172.17.42.1 -e CEPH_NETWORK=172.17.0.0/24 ceph/demo                                                                                  
ee3fa8858386595b7e66d7eeebf016694b0ee59a5a19a0d4b2a7f10d3fce5e87
➜  demo git:(master) docker logs -f ee3fa8858386595b7e66d7eeebf016694b0ee59a5a19a0d4b2a7f10d3fce5e87                              
creating /tmp/ceph.mon.keyring
importing contents of /etc/ceph/ceph.client.admin.keyring into /tmp/ceph.mon.keyring
importing contents of /etc/ceph/ceph.mon.keyring into /tmp/ceph.mon.keyring
2015-05-20 10:07:18.471878 7f54afc4b8c0 -1 unable to find any IP address in networks: 96.126.0.0/24

The way we currently generates the ceph.conf is really static.
Even if we use etcd or consul we still need a default raw implementation.
I believe Ansible could help on that matter.

After discussing with @SamYaple, it seems that Kolla has already a solution for this.
https://github.com/stackforge/kolla/blob/master/specs/ansible-multi.rst

This was suggested by @sdake on the openstack-dev ML.
This can potentially be done but I believe this adds (unnecessary?) complexity especially moving toward etcd or consul to store cluster configs. I think we should focus on etcd.

@Ulexus what do you think?

Just spotted this:

deprecation warning: MDS id 'mds.7cf73bf1fd74' is invalid and will be forbidden in a future version. MDS names may not start with a numeric digit.

An easy work around could be to do something like this on the MDS_NAME variable: : ${MDS_NAME:=a$(hostname -s)}

the demo container requires user_xattr

I was going to update the readme with a remount example but I'm not sure the best way to do it

something like this:

 sudo mount -o remount,user_xattr,rw /var

I'm running ceph/demo in localhost and try to test the rgw service.

Then I download the s3cmd. But what's the default .s3cfg file should I use? Can anyone help to give an sample file for me to test?

Hi,
I am trying to launch multiple ceph/osd in one host with macvlan( different IP for issue 19), each on a separate disk. This scenario is more like a production needs. But I found ceph/osd is not suitable at present:

1. It should accept OSD_ID. Disappointed that it was removed.
2. It should get it's hostname in container by "hostname -s" like what the ceph/mon do. Because it seems difficult for us to know the container hostname before it start.
   Can someone implement this? Thanks in advance.

We currently ship firefly as latest tag, we should move to giant and add a tag for firefly.

Next steps:

• Create firefly branch in Github
• Create firefly tag referencing that branch on Docker Hub
• Update each FROM statements in Dockerfiles for downstream components in the firefly branch to reference ceph/base:firefly
• Create firefly tags for each downstream component on Docker Hub
• Create giant tags for each component on Docker Hub

Just put this here as a reminder and potential blocker for Docker OSD deployments.
http://tracker.ceph.com/issues/10763

Would someone could nicely sharing how to build up a simple cluster with docker command and the IP specified method? Thx in advance.

I'm confusing about MON_IP, how can I get it before run it up to go Ceph-config?

I don't really the idea of having apache running on MON/OSD/MDS.
But having 2 base (even if one is named for rgw) is not really sophisticated.

I'm more in favour of new image for RGW base.

@Ulexus what do you think?