Comments (8)
Can you elaborate on the UX? Doing so before breaking ground on code will be extremely helpful for others to collaborate during development, and have a shared vision of where this takes the project.
from deploykit.
Motivation
- We want to have a scalable way of managing multiple groups (each with its own group config JSON) as well as other resource types (e.g. load balancers) that are coming to InfraKit.
- We want a single command that will take care of all the group watching and updating, for all groups and resources. No separate invocations of
infrakit group watch <file>
orinfrakit group update <file>
.
Design
The proposal here is that we use directory hierarchy / filesystem to manage a large collection of configuration files (in JSON). There will be a well-defined hierarchy of files and directories and the "frontend" (aka demux) will traverse and invoke the plugins as watch or update. The reason for using a directory hierarchy:
- Small config JSONs. The config file in a directory stays small as it applies only to the plugin (e.g. a Group plugin) it configures.
- Easy to edit and make changes. Bad formatting is easily isolated and fixed
- Integrates well with third-party config files that are already stored in directories -- for example, Terraform config files (or ansible) already have their own folders. Everything can be in managed under one filesystem hierarchy.
- Easy / natural backup / storage -- tar ball, git, S3 or Keywhiz backed filesystems
There are also other nice properties such filesystem events / file hashes that provide enough signal to simplify interaction with plugins such as Group, which has distinct verbs like watch
and update
. These verbs can be deduced from the filesystem events and hashes or commit triggers.
Initially we will only support groups. So under a top level directory, we will have something like this, for managers
, workders-db
, workers-large
, and workers-small
groups.
infrakit/
└── groups
├── .trash
├── managers
│ └── config.json
├── workers-db
│ ├── config.json
│ └── terraform_files
│ ├── main.tf
│ └── variables.tf
├── workers-large
│ └── config.json
├── workers-small
│ └── config.json
└── zookeeper
└── config.json
8 directories, 7 files
/tmp$ cat infrakit/groups/zookeeper/config.json
{
"ID": "zk",
"Properties": {
"Instance" : {
"Plugin": "instance-vagrant",
"Properties": {
"Box": "bento/ubuntu-16.04"
}
},
"Flavor" : {
"Plugin": "flavor-zookeeper",
"Properties": {
"type": "member",
"IPs": ["192.168.1.200", "192.168.1.201", "192.168.1.202"]
}
}
}
}
/tmp$
Under the top level directory infrakit
, a groups
directory indicates that all members of this folder are configurations for plugins that conform to spi.Group
. In each subdirectory under groups
such as managers
, there is a file config.json
, which is the group config JSON as they are currently defined. Note it's possible to nest folders that are appropriate for the plugin -- e.g. the terraform_files
directory for config files used by the Terraform plugin.
At the moment, we assume there are no dependencies. Each group / folder is independent on one of another.
Removing (unwatch
) a group (see Group plugin api) is as simple as moving a directory into the .trash
folder under groups
. For example, unwatching the workers-small
group looks like
/tmp$ mv infrakit/groups/workers-small infrakit/groups/.trash/
/tmp$ tree -a infrakit/
infrakit/
└── groups
├── .trash
│ └── workers-small
│ └── config.json
├── managers
│ └── config.json
├── workers-db
│ ├── config.json
│ └── terraform_files
│ ├── main.tf
│ └── variables.tf
├── workers-large
│ └── config.json
└── zookeeper
└── config.json
8 directories, 7 files
Alternatively, when integrated with backends that have full revision history, such as git, the removal of a group can be determined from change log.
UX
Given the directory structure above, the UX / workflow is as follows:
- Create or edit a JSON config, add new folders as necessary. Or move folders to
.trash
for unwatch. - At the command line, do
$infrakit commit ./infrakit # assumes infrakit is the name of the top level directory.
This command invocation will tell the frontend to start traversing the directories and depending on the state of the config files (new or edits), call the group plugins watch
or update
endpoints accordingly. The commit
can also be tied to a commit trigger for a VCS (e.g. git).
Plugin Activation
As part of processing this hierarchy of files, the frontend will activate the plugin if it's not already running. This will relieve the user from having to start the plugins manually.
We may implement pruning (deactivating) plugins if they are no longer referenced / needed in the configs. That's for a later phase.
Other Concerns
- We will build integrations for git or other backends in the future
- Backup and replication can be accomplished using existing tooling such as S3FS, Infinit volumes
from deploykit.
The proposal above leaves me unclear on the UX problem being addressed. I was anticipating that this effort would focus on relieving the user of managing startup of plugins for their configs. If i'm reading correctly, this seems to be more of a configuration management effort.
I find the config approach described above more cumbersome than the single JSON document we use today. The single document stands on its own, is atomically shareable, and it is easy to see in one place how the pieces fit together. I'm not seeing benefits of splitting it up that outweigh the added complexity of interacting with the system.
from deploykit.
Addressed the point on automating plugin activation. It's implicit but was not called out explicitly in the doc.
I don't think this is any more cumbersome than a single JSON document. A single JSON document is not significantly more shareable than a tarball of the entire directory.
- Smaller JSONs are easier to work with. The directory structure is easily visualized using tools like File Explorer or
tree
. For counter example -- see large Cloudformation JSONs and how third party tools like troposphere tries to solve by reusable snippets of python that are used to stitch together a final, large document. - Discrete JSON files make transition from plugin documentation to a larger system easier -- just drop the examples to a folder and edit.
- Plugin implementations may have their own configs that are not always exposed in the config JSON but nevertheless are required -- for example, Terraform's
main.tf
files, etc. This provides a convenient mechanism to have a central place to manage all things related to the config.
Are there specific examples of 'added complexities' you are referring to?
from deploykit.
Updated the doc to make clear that a config.json
is the whole config, as seen in the documentation of the plugins. I am not proposing to split up fields or chunks within a logical, atomic config into separate files.
from deploykit.
Are there specific examples of 'added complexities' you are referring to?
I gave myself the impression that config files were splitting apart each plugin configuration into separate files. The fact that configs are 1:1 with Groups alleviates a bunch of my concern.
Other than that, I find the implicit side-effects of changing files to be surprising and difficult to work with (e.g. move to .trash
directory has side-effects, sounds like you may have more similar side-effects in mind).
The biggest benefit i can see is storing auxiliary files next to the relevant configuration, but my hunch is that is not big enough of a problem to be engineered away.
from deploykit.
A PR #283 is a start to address this. The manager should be the entity that handles this, in terms of roles and responsibility.
from deploykit.
PR #283 is merged -- this is the start of a single frontend. Note its data model at manager/spec.go has the notions of multiple groups.
from deploykit.
Related Issues (20)
- New local CLI returning 0 if the plugin does not exist
- Enrollment controller template indexing rendering "<no value>" HOT 2
- Move repo to infrakit/infrakit
- Group controller always calling instance plugin with "properties=true" HOT 4
- Force manager leadership change on during manager rolling update HOT 3
- Cascading deletes using the terraform plugin HOT 3
- Pacing of Rolling updates HOT 1
- Swarm node garbage collection HOT 4
- Error: Property 'Box' must be set
- Request canceled client timeout HOT 2
- Update group rolling update polices
- Confusing/Outdated LinuxKit tutorial HOT 5
- Issues with the Cloudformation Example
- feat: Homebrew HOT 2
- Tutorial Fails with Client.Timeout HOT 2
- Use case documentation needed: provisioning VM using libvirt HOT 9
- Libvirt init section doesn't work: Permission denied
- Please upgrade libvirt HOT 1
- Renaming this project HOT 6
- Is deploykit going to support Apache CloudStack?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deploykit.