Comments (5)
Just reading from #81 (comment)
Were you proposing using ceph-disk inside the ceph-osd or ceph-daemon image?
i.e. to actually mount osds as well which requires --privileged
I was thinking you could maybe use a ceph-tools
container that is started only to run ceph-disk
privileged to do formatting. Outside of that, ceph-disk
is not used and instead we bootstrap the normal way. This gets around the need for privileged.
In fact you may be able to remove --privileged and use --cap-add SYS_ADMIN
and SYS_RAWIO
I hope I'm not confusing you all. I may have a completely different mindset on running this. It's good to bash ideas together. :-)
from ceph-container.
@hookenz You describe the primary reason I do not like ceph-disk
and ceph-deploy
. They are horrifically complicated python scripts for what they do; their complications are mainly due to their distribution detection and a whole bunch of really unnecessary "help-you" stuff (like the init script modification). That self-same detection also renders them useless in the realm of containers. All of that makes using them difficult and integrating with them a moving target. This is why I strongly prefer, for the containers, to build from the basic binary tools and manual procedures.
I do think lifecycle management is really important. There is the other discussion, #126 , which discusses the lifecycle management of the monitor. Here is a good place for discussion of OSD lifecycle management, I think.
Separation of duties
I've been irritated, recently, by our indistinction between the execution of the daemon and the bootstrapping and creation of the osd. It is easy to have them combined, and to have the container be "smart" about detecting and bootstrapping as necessary, but it is both dangerous and complicated. It leads to lower flexibility, overall.
I think, as such, we should take an example from ceph-disk
and decompose the process a bit. I don't think there is any problem with having, for instance, separate commands:
ceph/daemon osd create /dev/sdX [/dev/sdY]
ceph/daemon osd mount /dev/sdX
ceph/daemon osd [exec] /dev/sdX
ceph/daemon osd destroy /dev/sdX
This also allows us to separate out most --privileged
operations from the long-running exec
.
Argument could easily be made, here, that exec
should not, in fact, take a device argument, but should simply operate as it currently does, starting daemons for each OSD found in its /var/lib/ceph/osd/
directory.
Disk preparation
Another thing I don't like about ceph-disk
, but I think is probably a good compromise, is that it operates on raw disks, rather than filesystems. Personally, I greatly prefer the flexibility of mounting and preparing my own filesystems, but for use in a containerized environment, it might be best to simply have the partitioning and filesystem creation bundled into the process. I'm definitely of two minds in this area, and I would be interested to hear others' thoughts.
Ephemeralization
Full ephemeralization, like as in the #126 discussion for monitors, is not viable or safe for OSDs. However, since OSDs have their own internal state via their filesystems, I do think the containers themselves should be completely stateless. For the most part, they are already, but I want to keep this concept at the fore of design decisions. The more ephemeral we can be, the easier it will be to work with kubernetes and the like.
Miscellany
More could be done with specific backends (such as the key-value store), with creation, destruction, mounting, and execution, which could reduce external input and host-side/execution complexity. In particular, referencing the machine id and its attached OSDs in the key-value store could make the actual execution of OSDs a simply matter of starting, without any additional contextual data (beyond the machineid).
from ceph-container.
Great comment @Ulexus. I've had similar thoughts regarding the OSDs. Although it would be nice to have a single OSD task that can flexibly handle multiple devices or dirs (bootstrap and exec), it may be easier (and more secure) to split the creation/prep off as you suggest.
from ceph-container.
@Ulexus - I agree. Maybe we should write our own ceph-disk style utility.
I know you hate using raw devices, but I prefer it for journals. It bypass the filesystem layer for journalling. And in some ways, at least the way I was using it I think it is easier to set up. But then I could be swayed either way. :)
from ceph-container.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
from ceph-container.
Related Issues (20)
- /opt/ceph-container/bin/osd_disk_prepare.sh: line 46: ceph-disk: command not found HOT 7
- Need fix for CVE-2022-21797 HOT 4
- Bootstrap process hangs up for hours HOT 2
- not found /var/lib/ceph/osd/ceph-2//keyring HOT 2
- dnf update in ceph v18 container image is failing HOT 2
- RocksDBStore - cannot set permissions: Operation not permitted HOT 2
- /usr/bin/ceph: stderr Error EIO: Module 'cephadm' has experienced an error and cannot handle commands: ContainerInspectInfo HOT 2
- add ceph-mgr-callhome to IBM downstream container HOT 2
- cephadm has failed ContainerInspectInfo HOT 2
- populate_kvstore error HOT 1
- rename and repurpose this repository HOT 19
- reef builds don't work HOT 12
- Question about osd directory HOT 2
- docker-compose setup dose not run as expected mds and osd HOT 3
- With new quay.io/ceph/ceph:v16 image, ceph-csi meet segfault error HOT 2
- ceph/demo container does not expose mon port 3300 HOT 2
- Instructions for getting the zabbix template to work with rook-ceph HOT 2
- smartctl could not scrape metrics from HPE Smart Array in HBA mode HOT 2
- support VERSION=8 for contrib/compose-rhcs.sh
- Include cephfs-shell HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ceph-container.