kubernetes-retired / cluster-registry Goto Github PK

/sig multicluster

The cluster registry should provide a way to guarantee that a cluster that has been added has not been changed out for another cluster at the same IP address. This could be via a certificate stored in the cluster that a controller validates against the registry, and that can be provided to the user to validate against the cluster in a defined way.

From https://github.com/kubernetes/community/blob/master/contributors/design-proposals/multicluster/cluster-registry/project-design-and-plan.md#later

From a comment on #21, there might be reason to check in the generated artifacts to enable people who do not use bazel to depend on this repository; and perhaps to make the test infrastructure easier to wrangle. I am quite opposed to checking in artifacts unless it really is blocking a workflow, so this issue is tracking whether enough need to outweigh the pain of checked-in generated code.

If this is causing you problems, please bump this issue!

Generic API servers can store state anywhere. The sample API server (the project I believe this project is based off of) happens only supports etcd, but since the cluster-registry expects to store very small amounts of data (~hundred of objects max?) custom resources don't create a crazy amount of overhead.

I'd like to propose that the clusterregistry should support storing state in custom resources. This would dramatically reduce the amount of overhead it takes to deploy it.

Hey Jonathan,
We looked at this yesterday. Do you want to connect the cluster-registry up to Reviewable?

"Please connect this repo at https://reviewable.io/repositories so that all PRs get linked to reviews. Thanks!"

/sig multicluster

We need to test version skew for the cluster registry, the hosting clusters and kubectl. Ideally, we will have upgrade tests and e2e tests that can verify many of these things, but we do need to decide what is worth testing and then write the tests/frameworks necessary.

As a principle, since the cluster registry itself is so simple and foundational, the value of having these sorts of tests is very high, since it allows users to upgrade with more confidence and because the tests themselves are not that complicated to write.

From https://github.com/kubernetes/community/blob/master/contributors/design-proposals/multicluster/cluster-registry/project-design-and-plan.md#later

/sig multicluster

This requires some API design work: the Status object is currently empty. Someone will need to drive building consensus around an initial design for this object. Note that there may be several controllers interacting with the cluster registry as an expected use case, and the status object needs to take this into account.

From https://github.com/kubernetes/community/blob/master/contributors/design-proposals/multicluster/cluster-registry/project-design-and-plan.md#beta

From #65 (comment), we should consider how the crinit tool can be named and how its interface should be organized.

• renaming crinit to kubecr (credit @font for the name) or something else
• renaming/regrouping the subcommands to prepare for adding more commands to interact with a cluster registry. This isn't necessarily straightforward, since the tool is intended to support both aggregated and standalone deployments, each of which may have different create/update/delete operations, which may imply a three level nesting of (e.g.) kubecr <aggregated|standalone> <init|update|delete>.

I think this is something we'd want to do on a beta timeframe, since at that point we will be expecting the tool's interface not to change significantly.

GoDoc seems to require doc.go files in packages that it scans. Add them where appropriate, and make sure that there are no other things to do to integrate with GoDoc.

Migrated from a question previously in the cluster API doc:

Are the auth configuration objects conceptually ordered? Is the semantic meaning of the ordering defined by the cluster registry itself, or are users free to define their own semantic meaning to the ordering of auth configurations?

@perotinus has repurposed kubefed (now called crinit in this repository) to simplify cluster registry installation in a host Kubernetes cluster. Add support to crinit so that it could be used to register clusters to or deregister clusters from a cluster registry.

cc @G-Harmon @perotinus @csbell @nikhiljindal

Migrated from the Cluster API doc:

How should the Status field work? How do users intend to use the status field? How does the API indicate that the status is being updated by a tool vs a user vs no-one? Is that useful information? For one particular example, CloudProvider, which can be provided by a user or provided by a tool, does it belong in status?

Cloned from kubernetes/kubernetes#50746.

crinit init should continue if related resources already exist in the host cluster; or it should provide a --force flag.

Migrated from the Cluster API doc:

How can the API expose information about the transport necessary to reach a cluster? Can/should the cluster registry provide info about proxies, etc. as a field? Is there enough need to make it a top-level field rather than an application-specific annotation?

cc @emaildanwilson

I can't open the Google doc in the readme. Please update permissions so anyone with the link can read it.

Cluster registry users need reference docs for both API server and crinit binaries. The help text accessible through --help command line flag serves its purpose, but it is also extremely valuable to have it on the web so that it is indexed and made quickly accessible through search engines. It saves users from downloading the binaries and running --help every time they want to look up something really quick.

cc @perotinus @font @pmorie

The proposed authentication configuration revolves around a map[string]string

// Config is a map of values that contains the information necessary for a
// client to authenticate to a Kubernetes API server.
// +optional
Config map[string]string `json:"config,omitempty" protobuf:"bytes,2,rep,name=config"

My understanding is that in general we are now avoiding untyped maps for anything intended to be system-meaningful. They are hard to version & update, and we risk repeating the same problems we hit when using labels/annotations where we probably should have been using fields.

Should we consider a pattern more akin to health checks or volumes, with an effective union of strongly-typed options-types?

Cloned from kubernetes/kubernetes#43123.

Today crinit waits infinitely for some of its operations to complete. This is suboptimal. We should introduce a hidden flag to allow timeouts.

/sig multicluster

Depends on #81.

From https://github.com/kubernetes/community/blob/master/contributors/design-proposals/multicluster/cluster-registry/project-design-and-plan.md#beta

/sig multicluster

All the [API] requirements under https://github.com/kubernetes/community/blob/master/contributors/design-proposals/multicluster/cluster-registry/project-design-and-plan.md#alpha need to be met.

This can be done before we actually have the tests; the main objective here is to get out of the habit of hitting the merge button by hand.

cc @perotinus

From #8 (comment):

How do we want to handle the different personalities that clusterregistry will assume when running as a delegated vs standalone apiserver?

Some things that will need to be addressed:

• Authn model: currently CR has built-in authn model. Will also need to delegate to core k8s apiserver. Need to allow both.
• Authz model: currently CR has always-allow authz model. Will also need to delegate to core k8s apiserver. Need to allow both.
• Necessary flags: handling common and mutually exclusive flags for standalone vs delegated
• Other stuff?

Some possible options:

• Add extra flag error checking. Possible confusion when user specifies combinations that are not compatible. This assumes we have several mutually exclusive options. It's not clear that we do yet but may in the future.
  • Roughly 8 for authn standalone and 8 for authn delegated.
• Do we want to add new sub-commands to handle the different flags that could then trigger different paths for the authn/authz setup? This would seem strange for an apiserver. Are there any other apiservers doing this?
  • This could be something similar to hyperkube where the first argument specifies which "personality" to execute.
• Two separate binaries for delegated vs standalone. Probably try to avoid this unless we follow a hyperkube type model.
• Other options?

/sig multicluster

This is already at least partially done, but could be improved.

Some ideas:

• examples of how to use the generated clients
• explanation of the code structure
• all packages have package-level documentation

Feel free to add more ideas!

From https://github.com/kubernetes/community/blob/master/contributors/design-proposals/multicluster/cluster-registry/project-design-and-plan.md#alpha-targeting-late-q4-17

Cloned from kubernetes/kubernetes#50817.

"It would be great to allow crinit init to accept a file just like what we do for kubectl apply -f <filename>, so that I can put all parameters in a file which is easy to maintain."

I'm trying to understand the use case for the CloudProvider field.

What is the cloud provider in a multi-repo, multi-cloudprovider-implementation world? If there are two AWS cloudprovider implementations, should they have the same name or a different name? How are the CloudProvider names registered?

What if a single cluster spans multiple clouds? Should this be of []CloudProvider?

Now that the cluster registry prototype is done, we should write a design based on this knowledge and circulate it more broadly.

Once the cluster registry is in a stable enough state, it should have a users guide distinct from the developers guide.

This may make sense to host here, or perhaps on a shared Kubernetes page.

cc @pmorie

cf this comment in the cluster registry API doc.

Cloned from kubernetes/kubernetes#50540

(Not sure if this is relevant for crinit.)

crinit does API version discovery and then uses the preferred API group version that server supports. This breaks when server supports a newer version that the generated clientset that crinit uses does not know about.

crinit should choose the version that both server and it knows about.

As part of the release process, we will need to make sure that licensing requirements for the compiled binaries and distributed source tarballs (if provided) are met.

/sig multicluster

Update cluster-registry rules_go to 0.6.0. This should help enable bazel debug builds enabled by bazelbuild/rules_go#743.

The cluster registry currently doesn't support namespaces, which are the main way that Kubernetes APIs support multi-tenancy. It's not clear whether we can use RBAC as-is to support multi tenancy in the cluster registry, or whether we will have to build some alternative support or modify the cluster registry in order to work more fluently with RBAC.

This is one reason that we would implement an authorizer other than "AlwaysAllow": in the simple single-tenant case, we can probably assume that authn==authz (that is, anyone who can authenticate is authorized), since there is only one resource type, the cluster,

Not a major issue now.

A testing plan will define what needs to be covered.

/sig multicluster

From https://github.com/kubernetes/community/blob/master/contributors/design-proposals/multicluster/cluster-registry/project-design-and-plan.md#beta-targeting-mid-q1-18

This tracks the work to modify crinit to deploy the cluster registry using API aggregation. See https://kubernetes.io/docs/tasks/access-kubernetes-api/setup-extension-api-server/ for required steps.

Once #19 goes in, there will be an incomplete but runnable suite of bazel tests that should be run before PR submission.

This involves adding some configuration to the Prow config (https://github.com/kubernetes/test-infra/blob/master/prow/config.yaml) and the job config (https://github.com/kubernetes/test-infra/blob/master/jobs/config.json).

Since it is currently necessary to run ./update-codegen.sh and bazel run //:gazelle before running the bazel tests, I expect that we will have to use a custom execution scenario (https://github.com/kubernetes/test-infra/blob/master/scenarios/execute.py) and call the bazel scenario directly after running the codegen commands.

Kubernetes API reference doc has been an invaluable resource not just for Kubernetes users, but also for the development team. We need similar a doc for the cluster registry API both for our users and the development team.

cc @perotinus @font @pmorie

/sig multicluster

This is mainly to support authorization on groups of clusters. Namespaces are the obvious mechanism for this, but they are not the only one and we need to do some design work here to find the right approach.

From https://github.com/kubernetes/community/blob/master/contributors/design-proposals/multicluster/cluster-registry/project-design-and-plan.md#beta

Based on @ericchiang's comment (#58 (comment)).

dep is currently not quite ready for use with all of Kubernetes, and using it will potentially expose us to incompatibilities; however, there is activity in other k8s repos to improve dep support. Since updating vendored deps is not a common activity, it's not critical to have it be a completely smooth experience, but it should also not be so arcane and messy as to be undoable. We need to evaluate further.

The cluster registry needs a way to authorize requests. Currently, it allows authenticated callers full access. It should at least support r/w vs readonly usage.

The canonical Kubernetes way to handle this would be RBAC, but we don't currently support creating RBAC policies in the cluster registry, and it's not clear that the cluster registry should have a parallel set of RBAC rules to the main Kubernetes API server (and it probably should defer to another API server if it is an aggregated API server).

Generate OpenAPI and/or Swagger definitions for the cluster API.

/sig multicluster

We should have a draft of user documentation for how to bootstrap/set up/use a cluster registry. This should probably include some examples as well.

From https://github.com/kubernetes/community/blob/master/contributors/design-proposals/multicluster/cluster-registry/project-design-and-plan.md#alpha-targeting-late-q4-17

/sig multicluster

This is a bit loosely defined, and will probably require a mini-design (or at least a comment with some ideas in this issue).

From https://github.com/kubernetes/community/blob/master/contributors/design-proposals/multicluster/cluster-registry/project-design-and-plan.md#beta

The cluster registry should have a way to handle authorization when it is not being run as a delegated API server. It doesn't seem ideal to implement the RBAC APIs in the cluster registry, given the potential for confusion and mismatches in implementing an API for a k8s object. One potential approach here is to use an authenticating/authorizing proxy.

Split from #8, which is tracking work for delegated authorization.

/sig multicluster

From https://github.com/kubernetes/community/blob/master/contributors/design-proposals/multicluster/cluster-registry/project-design-and-plan.md#beta

Cloned from kubernetes/kubernetes#50734.

crinit init will install etcd, but the etcd is not secured, we should enable secure mode for etcd.
Details about etcd TLS

/sig multicluster

Close criteria:

• unit tests for each non-trivial package
• integration test for the cluster CRD

From https://github.com/kubernetes/community/blob/master/contributors/design-proposals/multicluster/cluster-registry/project-design-and-plan.md#alpha-targeting-late-q4-17

Based on comments in #60, we plan to revert to the glog version used by k8s. However, because of all the outstanding repo infra work (#58, #61, #62) I want to defer this so as not to complicate those changes.

It would be extremely useful to have a helm chart to deploy the cluster-registry and the apiservice resource to add it into the aggregator.

/sig multicluster

The API should be stable enough at this point to support replacing the Federation cluster API with the cluster registry. The details of this integration will need some design: it's not clear whether the Federation API server will aggregate the cluster registry, or include the API types directly, or something else entirely.

From https://github.com/kubernetes/community/blob/master/contributors/design-proposals/multicluster/cluster-registry/project-design-and-plan.md#beta

/sig multicluster

From https://github.com/kubernetes/community/blob/master/contributors/design-proposals/multicluster/cluster-registry/project-design-and-plan.md#beta

@pmorie raised a few issues in #16 about the cluster registry API that were not addressed:

• being more explicit about how credentials are handled/whether credentials should be stored
  • work here is waiting for input from SIG-auth about how to handle cluster credentials in the cluster registry
• optional fields should be pointers
  • this seems reasonable, but would like to discuss with sig-arch and/or sig-api-machinery to make sure

Besides this, there are a few issues from the cluster registry API design that also need to be addressed:

• using a stronger type for the Kubernetes API server info
  • this was going to be a copy of api.Endpoints from k8s, for reasons described here, but after pushback more discussion is needed
• spec/status field names/presence
  • this is waiting on discussions with sig-arch and/or sig-api-machinery
• CABundle in KubeAPIServer
  • waiting for response on comment

And an issue from #29:

• figure out motivation for or remove CloudProvider field

cf https://docs.google.com/a/google.com/document/d/1Oi9EO3Jwtp69obakl-9YpLkP764GZzsz95XJlX1a960/edit?disco=AAAABFxkK0A

cc @pmorie @quinton-hoole