Context

Currently, we maintain a CHANGELOG.md from https://keepachangelog.com, however the maintenance of this file is (IHMO) a pain:

• It's often outdated
• May not contain all changes since last release, one has to view the git history anyway
• Especially on GitHub, people often expect the changelog in the releases page. There we often find a copy of the CHANGELOG.md, but this is done manually and not even in all cases.
• Often requires "preparation" commits for releases (renaming Unreleased section to next SemVer, search git/PR history whether there are missing changes etc.)

My proposal:

1. Drop the CHANGELOG.md from the repositories, issue templates and other documentation
2. Use a GitHub Action to generate the changelog in the GH Release
   • Uses GH API to extract Titles from PRs and creates nice links
   • PRs can be grouped visibly in the changelog into categories (e.g. Fixes, Features, etc) with GH labels. Or be uncategorized if no labels

Both approaches (keepachangelog) and the generated one from the Action require discipline, but the generated one reduces this discipline to maintain nice PR titles (which we hopefully do already) and setting labels if categorization is required. But even without categorization, it's included in the changelog. More automation, more better :)

Alternatives

Go-based projects can use https://goreleaser.com/ to generate a changelog directly from commits. Other programming languages may or may not have something similar that can be integrated into GH Actions. This proposal is not about dropping Goreleaser or its feature in order to make every Syn-project have the same look of the changelog, but rather enabling automation for those projects that don't have any sort of changelog generation at all.

Context

Going through the getting started guide, I had some issues with placing the lieutenant API on port 80 on my local machine.

Using k3d as suggested, I have another service running on that port (though I still don't quite know which one, lsof didn't show me..).

I switched to port 8080 using k3d:

k3d cluster create synquickstart -p "8080:80@loadbalancer" -p "8443:443@loadbalancer" 

Therefore I adjusted the LIEUTENANT_URL to:

export LIEUTENANT_URL="lieutenant.${INGRESS_IP}.nip.io:8080"

Which works quite well.

However: Upon deploying steward, it tries to call lieutenant API on http://lieutenant.127.0.0.1.nip.io which doesn't work for some reason I'm not quite sure about.
I edited the deployment to switch it to port 8080 - doesn't work as well of course, because it tries to call the API on port 8080 but it actually runs on port 80.

In the end I switched STEWARD_API to http://lieutenant-api.lieutenant.svc.cluster.local (btw why is it not called LIEUTENANT_API?) which works probably in all cases (if you know the cluster domain).

Now my question: shouldn't the Lieutenant API always be called via the service within the cluster?

This is probably not really an issue with the getting started guide, but maybe it is? I'm not sure :)

Steps to Reproduce the Problem

1. run local cluster with another port as shown above
2. run all steps until steward
3. run kubectl get pods -n syn - it will show no other pod than steward
4. run kubectl logs $STEWARD_POD_NAME to have a look at the logs which will look similar to that:

» k logs -n syn steward-7c9697d8c-4bb2g
I1021 07:48:59.211936       1 main.go:26] Starting SYN cluster agent 🕵️
I1021 07:48:59.212039       1 main.go:27] Version v0.2.1-2-g0038ed0-dirty
W1021 07:48:59.466763       1 secret.go:39] Argo CD secret already exists, no update needed
E1021 07:48:59.548516       1 agent.go:94] Patch "http://lieutenant.127.0.0.1.nip.io/clusters/c-polished-field-6864": dial tcp 127.0.0.1:80: connect: connection refused

Actual Behavior

Steward couldn't talk to Lieutenant API.

Expected Behavior

Steward can talk to Lieutenant API in all cases.

Context

Lately the discussion came up that there is a need of having a Project Syn CLI which interacts with the different involved tools and makes the project much more accessible.

Task

Write an SDD describing the potential new Project Syn CLI, the intended initial features and how it will be implemented. Extract the already collected ideas from the internal VSHN Wiki and add it to the SDD.

It's currently not clearly defined how versioning across all the Project Syn tools is handled and how versions are "connected" to each others. It's also not clear where there are compatibility issues, when to raise the version number, and so on.

Therefore we need to define the versioning policy of the overall project so that all Project Syn tools use the same policy. This must also solve the compatibility issues so that the end-user experience gets much better.

Included in this issue is also a research how the documentation could be versioned together with the tooling.

In SDD 007 an idea is described how to collect and store (dynamic) facts on a cluster and store it in Lieutenant. This idea needs to be refined in a follow-up SDD, ready to be implemented after the SDD has been agreed upon.

Constraints:

• The use of a TSDB isn't considered important at this time
• The collected / dynamic facts need to be usable by Commodore Components
• The facts should be stored in the Cluster CRD, distinguishable by the manual facts

Define how we name commodore components and:

• where the name comes from
• that the name implies what a component contains

Maybe something like this, example (synsights):

• synsights-metrics
  • Kind: Prometheus (the SYN prometheus instance), namespace: syn-synsights
  • Kind: Alertmanager (the SYN Alertmanager), namespace: syn-synsights
  • Deployment: Pushgateway, namespace: syn-synsights
    • with corresponding Service and Route / Ingress to make it available at least within the cluster
  • Kind: ServiceMonitor to scrape the pushgateway, namespace: syn-synsights
  • Deployment: Blackbox_exporter, namespace: syn-synsights
• openshift4-monitoring, namespace: syn-synsights
  • Kind: ServiceMonitor to federate with openshift managed cluster prometheus
  • Kind: AlertRule for openshift specific alerts
  • configuration needed to actually enable the OpenShift cluster prometheus, etc.
• node-monitoring, namespace: syn-synsights
  • Kind: DaemonSet for node_exporter
  • Kind: ServiceMonitor for node_exporter
  • Alter rules for node specific alerts, e.g. disk space of system disk, load, memory usage, network stats
• kube-monitoring, namespace: syn-synsights
  • Kind: DaemonSet for "kube exporters"
  • Kind: ServiceMonitor for "kube exporters"
  • Alert rules for kubernetes component specific alerts, e.g. etcd, number of nodes, scheduler stuff, kublet running, etc.

Besides monitoring, these could look like this:

• node-maintenance
  • install and configure the "node agent" for automated "apt-get dist-upgrade"
• node-config
  • install and configure the "node agent" to do base setup stuff on the VMs like NTP, SSH, user accounts, etc.
• openshift4-oauth
• openshift4-whatever

Based on a discussion I had with @tobru:

Initially, commodore component were envisioned to be self contained. For that reason, initial components take care of everything including like S3 buckets and such by using Crossplane.

We need to change the default in a way, that all external referenced can be passed in as parameters. The reasons for that are:

• One might have Kubernetes clusters on infrastructure without Crossplane support. Adding support might not be an option.
• One might already have those external resources and do not want Crossplane to create new ones.

The how to "Getting Started with Project Syn" does not work on macOS. Docker is not supported natively so most will use Docker for Mac. The way this is set up, messes up networking for most things that just work ™️ on Linux. So this is what one needs to do differently:

1. Use Docker for Mac

2. Enable Kubernetes within its settings

3. Install Nginx ingress
   https://kubernetes.github.io/ingress-nginx/deploy/#docker-for-mac

   kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud/deploy.yaml
   

   NOTE: This will not work if you have other services running on port 80 and 443 on your macOS.

4. Use localhost for the ingress IP

   export K3S_INGRESS_IP=127.0.0.1
   

5. Use the injected docker service while compiling the cluster catalog

   export COMMODORE_API_URL=http://$LIEUTENANT_API_SERVICE_HOST
   

   NOTE: I was not able to find a way to define COMMODORE_API_URL while starting the Commodore container. So it has to be done once inside the container.

Context

Discovering Commodore Components is a very important topic to spread the word an make it more approachable to interested parties. Currently we do use GitHub topics to make them discoverable: https://github.com/search?q=topic%3Acommodore-component.

We should figure out if something like Artifact Hub could help to increase visibility of Commodore Components. Either by developing / hosting an own version of a / the artifact hub or by taking part of https://artifacthub.io/.

Alternatives

Stick with GitHub topics for discoverability.

The Getting Started guide gets bigger and bigger and should be split into subpages for better readability.

The different Project Syn tools should bring their own Getting Started guide and be integrated into the main syn.tools documentation using Asciidoc includes.

The getting started tutorial uses old versions of lieutenant-operator and lieutenant-api. The versions should be updated to the most recent versions.

Current versions are v0.5.3 for lieutenant-operator and v0.6.0 for lieutenant-api.

Context

SDD 0002 has outgrown and needs to be replaced by a new scheme which provides more descriptive names to the Project Syn tooling

Alternatives

Keep SDD 0002 but rework it as it's still draft. That has the drawback that history isn't preserved and doesn't follow the SDD process exactly.

We previously experimented with Doxygen to auto-generate some API documentation for components (especially components which provide libraries for other components to use). Integrate generating documentation from Doxygen comments into the documentation infrastructure and document how to document components (e.g. using file-level Doxygen comments).

This will only become relevant once components have been migrated from the VSHN internal GitLab to GitHub.

Task deliverables

• Commodore component Doxygen comments are rendered and displayed alongside the rest of the documentation
• How to document components is documented

Add the steps to the getting started guide to install Steward and to connect it to Lieutenant and to get Argo CD going.

As more people start writing Commodore components, questions regarding Jsonnet best practices are popping up more frequently.

While we already have #36 to document component architecture and interface principles and #40 to establish a Jsonnet style convention and guide, other best practice areas also need documentation, e.g.:

• best practices when using kube-libsonnet.

• Best practices for internal component structure, such as what belongs in a component library, when does it make sense to have multiple jsonnet files in component/, etc.

• Merging External Config Data

• Loops and Filters

• Component Library Functions

• Conditional Keys and Files

• Using kube-libsonnet

• Using Library functions

• CRD Group Versions

• Container Image & Helm Chart Versions

• Multiline Strings

• Component Structure

• Randomize (cron) schedules -> see #96

Create a page which describes the Project Syn glossary.

Context

The current Vault structure enforces the <tenant-id>/<cluster-id>/ structure. Nothing more is enforced or recommended.
We should document the best practices around secrets in Vault and how to structure them.
Some inputs:

• Use as less key-value pairs per secret as possible (it's not possible to update only single key-value pairs)
• Use descriptive names, so it's clear what a secret is used for
• Consistent naming (e.g. token vs. password, vs.
• ...

Alternatives

Implement more secrets generation via Lieutenant-operator which would enforce certain structures.

Context

As a CTO
I want to have a first impression in less than 30 minutes
So that I can evaluate Syn

The current "Getting Started Guide" (https://docs.syn.tools/syn/tutorials/getting-started.html) is big read and takes very long to do. There's no clear target audience: Is it meant for developers so they can start contributing or is it meant for someone who just wants to try out in action without understanding the behind-the-scenes and possibly buy commercial support for it?

The title "Getting Started" is already misleading. With a "Getting started" title my expectations are set to a point where I can setup something quick and easy. I'm not bothered about understanding the architecture, there can be links for further topics in the guide.

That being said, the setup should be as automated as much as possible. The best possible option, IMO, is like K3s does it:

curl -sfL https://get.syn.tools | bash -

This script checks if all pre-requirements are met and covers all of the installation: to get a control cluster up and synfecting the compute cluster, all locally.

The remaining content of the tutorial should be usage-focused:

• Where to find the catalogs and components
• How to make changes to the cluster or tenant, roll them out, verify the change (understanding the hierarchy of the configuration)
• See the applications in ArgoCD

The first impression of the current guide is rather devastating from my own experience: It's very complex, has high entry requirements, requires 18 "easy" steps to install something.

There should still be second guide that explains the internals and architecture. So maybe creating the install script and providing a completely new guide while renaming the current one is the way to go.

Alternatives

Rename the guide. Better titles for the current guide are:

• Tutorial: Understanding Syn Architecture
• ProjectSyn The Hard Way 😄

Require all jsonnet code to be compliant with jsonnetfmt using the default options. For CI/CD pipelines, the command jsonnetfmt --test **/*.jsonnet can be used, while autoformatting can be done using jsonnetfmt -i **/*.jsonnet.

jsonnetfmt is documented at https://jsonnet.org/learning/tools.html.

For YAML files, we should use yamllint For CI/CD pipelines, the command yamllint -f parsable --no-warnings **/*.yml can be used. For checking before commiting, yamllint **/*.yml will do the trick.

Rule wise, we might opt to make line-length a warning instead an error and or increase the character count from 80 to something like 120. Subject to be discussed.

As an author of Commodore components, operators and others, one easily can get lost when it comes to naming labels and annotations. Having a guide would be of help. It can:

• speed up the decision process for defining label names.
• prevent collisions and thus unintended side effects.

None conclusive list to be covered in that guide:

• labels to be used on different Kubernetes objects (e.g. Deployments, Services, etc.)
• labels to be used for matchSelectors (e.g. to make Prometheus picking up service monitors or alert rules).

See also https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/.

Write a SDD which describes the Project Syn documentation concept:

• How do we document?
• What is documented where?
• What structure does the documentation have?
• What is the form of the documentation?

And maybe even a lot more important points for having "the best" documentation ever!

Context

As of today (17.8.2022) the Syn documentation is quite a mess IMO.

As a component and package author the relevant pages are stretched out in various places. Some examples:

• The Tutorial to write components contains the whole Syn setup including local Lieutenant. However, currently at VSHN we nowadays develop components with local compilation without cluster deployments. Yet the tutorial for writing packages is in the Commodore Antora project. 🤯
• All the commodore component best practices are under the root Syn Antora project, not in Commodore. Yet when developing components, the main interaction happens within the Commodore tooling.
  
• The tutorial for Packages doesn't explain how a package is intended to be used. There is no reference page or explanation, but only a small section in a simplistic tutorial
• The "Creating a package" section in the package tutorial doesn't refer to the best practices, yet they belong together. That means, as a package author that wants to test packages also locally, I need to keep open at least 3 tabs with separate documentation locations.

(This list is incomplete, might be extended)

On some locations I question Divio's structure to put random articles into the 4 categories (Tutorial, How-to, Reference, Explanation). Or at least it's badly done as of today.
For example, in the following Screenshot the menu structure doesn't make sense:

• Under How-to guides, why are there top-level random articles, but submenus for Lieutenant Operator and API, and an empty Steward entry that isn't even clickable.
• Under Explanation same thing, seemingly random articles and then inclusion of completely different tools.

Over the years, it seems people can't make a clear distinction between a tutorial and a how-to, or an explanation and reference page. Often there are parameters and patterns explained in a how-to or tutorial page, but there's nothing in the references.

As Divio puts it: (https://documentation.divio.com/structure/#why-isn-t-this-obvious)

how-to guides and technical reference are both what we need when we are at work, coding

Yet when I code, the stuff I need is either a Tutorial or in a completely different Antora project xD

Alternatives

Most of the documentation is aimed towards engineers. Yet there are different kinds of engineers:

• Component or package authors that need to learn how to write good jsonnet and parameter structures.
• Cluster-admins that install components and packages
• Component or package maintainers/owners that review PRs and release new versions
• CTOs that want to learn what project syn is and does to evaluate whether this is something for them.

Maybe an alternative structure that is aimed towards the different personas is more helpful than strictly dividing into the 4 divio categories. Maybe it's still sensible to keep the nature of a page as a Tutorial, how-to, explanation or reference, but only change the menu/nav links to them. Some personas require the same pages, for example both cluster-admins and package authors need to know how to set parameters so their package or component gets installed.

Prose example (just an idea worth discussing):

For Component Contributors:
- Tutorial: Bootstrap new component
- How-To: Jsonnet problems and patterns
- How to: Running Commodore
- Reference: Style Guide
- Reference: Renovate version pins

For Component Maintainers:
- How to Release a component
- How-to: Prepare for modulesync(/cruft)
- How to: Running Commodore
- Reference: Review Guide
- Reference: Style Guide
- Reference: Renovate version pins

For Cluster-Administrators:
- How to: Compile a Cluster catalog
- How to: Change a parameter
- How to: Running Commodore

For new users:
- Tutorial: Getting started
- Reference: Concepts

The goal is the each persona basically has everything relevant for a certain task. A component author doesn't care about compiling catalogs at the time of authoring a PR. However, the same person might switch the persona to a cluster-admin when they try to actually deploy their component to a cluster, then the relevant documentation pages change.

Developing Lieutenant is a completely different topic and should be rather standalone than randomly put somewhere.

Randomize CronJob or any other time based schedules is a crucial topic.

Example:

local scope = "…" // Something that defines the scope of the schedule
local minute = std.foldl(function(x, y) x + y, std.encodeUTF8(std.md5(scope)), 0) % 60;
local job = kube.CronJob(name) {
  spec+: {
    schedule: '%d * * * *' % minute,
  …
}

Using a hash function makes the result deterministic. For that reason, the scope becomes important to ensure distribution of values.

Suggestions:

• For things running on clusters but act against a common system, use the cluster name. This will ensure, that all those jobs are executed at a different time.
• For things running on the same cluster but should be spread over time, use something like the namespace and or job name the job is running in. The same job will run at the same time on every cluster but different jobs on a cluster will run a different times
• Mixing in the the clusters name with namespace and or job names will ensure that different jobs run at different times both on the same cluster as well as the whole fleet of clusters.

See also appuio/component-openshift4-authentication@bfbff9d

Write an SDD which describes how the Argo CD instance used by Project Syn can also be used by the end-user of the Kubernetes cluster to do application deployment.

This is part of "GitOps for application deployment on a Project Syn enabled Kubernetes Cluster" from the Roadmap.

Context

In projectsyn/component-espejo#8 research has been done for testing Commodore Components. The discussion and outcome of this research needs to be preserved in an SDD.

Alternatives

Leave the outcome in the GitHub pull request, but that would make it hardly discoverable in the future.

Context

Currently we do multiple automated steps when a new cluster is created: a new git repo is created for the catalog, a token is stored in Vault, etc. This will be further extended in the future, for example to save arbitrary secrets in Vault (projectsyn/lieutenant-operator#79), create some CRs to automatically provision cluster via hive or Crossplane, set up authentication provider (e.g. Keycloak), Configure OpsGenie, etc.

Until now all of these features are implemented directly in the lieutenant-operator. Some of these automations are more generic than others and some will most likely only every be used by VSHN.
To make this more extensible and provide a scalable approach to add more automations as the need arises, we need a concept (SDD?) to define where and how such functionality should be implemented.

It should also be part of the concept how to leverage existing operators whenever possible. This includes for example the hive operator or the keycloak-operator.

Some Ideas

• Create a new operator for each feature (i.e. Vault operator, Keycloak operator, etc.)
• Create a new operator for all the VSHN specific features (Keycloak, OpsGenie)
• Implement the features in lieutenant-operator but in a generic way, like we currently do with the GitRepos (GitLab, GitHub, Gitea)

Alternatives

Continue to implement all functionality within the lieutenant-operator. This has the downside that it will become a complex operator which needs to support many different use-cases and might be too VSHN specific.