Git Product home page Git Product logo

garden-setup's Introduction

This repo is not actively maintained. We are working on a documentation for setting up a Gardener landscape based on the Gardener Operator

Install Gardener on your Kubernetes Landscape

Gardener uses Kubernetes to manage Kubernetes clusters. This documentation describes how to install Gardener on an existing Kubernetes cluster of your IaaS provider.

Where reference is made in this document to the base cluster, we are actually referring to the existing cluster where you will install Gardener. This helps to distinguish them from the clusters that you will create after the installation using Gardener. Once it's installed, it is also referred to as garden cluster. Whenever you create clusters, Gardener will create seed clusters and shoot clusters. In this documentation we will only cover the installation of clusters in one region of one IaaS provider. More information: Architecture.

Disclaimer: Productive Use

Please be aware that garden-setup was created with the intent of providing an easy way to install Gardener for the purpose of "having a look into it". While it offers lots of configuration options and can be used to create landscapes , garden-setup lacks some features which are usually expected from a 'productive' installer. The most prominent example is that garden-setup does not have any built-in support for upgrading an existing landscape. You can 'deploy over' an existing landscape with a new version of garden-setup (or one of its components), but this scenario is not tested or validated in any way, and might or might not work.

Prerequisites

  • The installation was tested on Linux and MacOS
  • You need to have the following tools installed:
  • You need a base cluster. Currently, the installation tools supports to install Gardener on the following Kubernetes clusters:
    • Kubernetes version >= 1.11 or enable the feature gate CustomResourceSubresources for 1.10 clusters
    • Google Kubernetes Engine (GKE) on Google Cloud Platform (GCP)
    • Elastic Container Service for Kubernetes (EKS) or Kubernetes Operations (kops) on Amazon Web Services (AWS)
      • Standard EKS clusters impose some additional difficulties for deploying a Gardener, one example being the EKS networking plugin that uses the same CIDR for nodes and pods, which Gardener can't handle. We are working on an improved documentation for this case. In the meantime, it is recommended to use other means for getting the initial cluster to avoid additional efforts.
    • Azure Kubernetes Service (AKS) on Microsoft Azure
  • Your base cluster needs at least 4 nodes with a size of 8GB for each node
    • This is only a rough estimate for the required resources, you can also use fewer or more nodes if the node size is adjusted accordingly
    • If you don't create additional seeds, all shoots' controlplanes will be hosted on your base cluster and these minimal requirements won't hold
  • You need a service account for the virtual machine instance of your IaaS provider where your Kubernetes version runs
  • You need to have permissions to access your base cluster's private key
  • You are connected to your Kubernetes cluster (environment variable KUBECONFIG is set)
  • You need to have the Vertical Pod Autoscaler (VPA) installed on the base cluster and each seed cluster (Gardener deploys it on shooted seeds automatically).

Procedure

To install Gardener in your base cluster, a command line tool sow is used. It depends on other tools to be installed. To make it simple, we have created a Docker image that already contains sow and all required tools. To execute sow you call a wrapper script which starts sow in a Docker container (Docker will download the image from eu.gcr.io/gardener-project/sow if it is not available locally yet). Docker executes the sow command with the given arguments, and mounts parts of your file system into that container so that sow can read configuration files for the installation of Gardener components, and can persist the state of your installation. After sow's execution Docker removes the container again.

Which version of sow is compatible with this version of garden-setup is specified in the SOW_VERSION file. Other versions might work too, but especially older versions of sow are probably incompatible with newer versions of garden-setup.

  1. Clone the sow repository and add the path to our wrapper script to your PATH variable so you can call sow on the command line.

    # setup for calling sow via the wrapper
    git clone "https://github.com/gardener/sow"
    cd sow
    export PATH=$PATH:$PWD/docker/bin
  2. Create a directory landscape for your Gardener landscape and clone this repository into a subdirectory called crop:

    cd ..
    mkdir landscape
    cd landscape
    git clone "https://github.com/gardener/garden-setup" crop
  3. If you don't have your kubekonfig stored locally somewhere yet, download it. For example, for GKE you would use the following command:

    gcloud container clusters get-credentials <your_cluster> --zone <your_zone> --project <your_project>
  4. Save your kubeconfig somewhere in your landscape directory. For the remaining steps we will assume that you saved it using file path landscape/kubeconfig.

  5. In your landscape directory, create a configuration file called acre.yaml. The structure of the configuration file is described below. Note that the relative file path ./kubeconfig file must be specified in field landscape.cluster.kubeconfig in the configuration file.

    Do not use file acre.yaml in directory crop. This file is used internally by the installation tool.

  6. The Gardener itself, but also garden-setup can only handle kubeconfigs with standard authentication methods (basic auth, token, ...). Authentication methods that require a third party tool, e.g. the aws or gcloud CLI, are not supported.

    If you don't have a kubeconfig with a supported method of authentication, you can use this workaround: create a serviceaccount, grant it cluster-admin privileges by adding it to the corresponding ClusterRoleBinding, and construct a kubeconfig using that serviceaccount's token. Here is an example on how to do it manually. Alternatively, you can use the following command:

    sow convertkubeconfig

    It will create a namespace with a serviceaccount in it and a clusterrolebinding which binds this serviceaccount to the cluster-admin role. Note that this replaces the kubeconfig which is referenced in the acre.yaml file. Also, there is currently no command to cleanup the resources created by this command, you will have to remove the namespace as well as the clusterrolebinding named garden-setup-auth manually if you want them cleaned up (the kubeconfig will then stop working).

  7. Open a second terminal window which current directory is your landscape directory. Set the KUBECONFIG environment variable as specified in landscape.cluster.kubeconfig, and watch the progress of the Gardener installation:

    export KUBECONFIG=./kubeconfig
    watch -d kubectl -n garden get pods,ingress,sts,svc
  8. In your first terminal window, use the following command to check in which order the components will be installed. Nothing will be deployed yet and you can test this way if your syntax in acre.yaml is correct:

    sow order -A
  9. If there are no error messages, use the following command to deploy Gardener on your base cluster:

    sow deploy -A
  10. sow now starts to install Gardener in your base cluster. The installation can take about 30 minutes. sow prints out status messages to the terminal window so that you can check the status of the installation. The other terminal window will show the newly created Kubernetes resources after a while and if their deployment was successful. Wait until the last component is deployed and all created Kubernetes resources are in status Running.

  11. Use the following command to find out the URL of the Gardener dashboard.

    sow url

More information: Most Important Commands and Directories

Concept: The 'Virtual' Cluster

As a part of garden-setup, a kube-apiserver and kube-controller-manager will be deployed into your base cluster, creating the so-called 'virtual' cluster. The name comes from the fact that it behaves like a kubernetes cluster, but there aren't any nodes behind this kube-apiserver and thus no workload will actually run on this cluster. This kube-apiserver is then extended by the Gardener apiserver.

Reasoning

At first glance, this feels unintuitive. Why do we create another kube-apiserver which needs its own kubeconfig? However, there are two major reasons for this approach:

Full Control

The kube-apiserver needs to be configured in a certain way so that it can be used for a Gardener landscape. For example, the Gardener dashboard needs some OIDC configuration to be set on the kube-apiserver, otherwise authentication at the dashboard won't work. However, since garden-setup relies on a base cluster created by other means, many people will probably use a managed kubernetes service (like GKE) to create the initial cluster - but most of the managed services do not grant access to the kube-apiserver to the end-users. By deploying an own kube-apiserver, garden-setup ensures full control over its configuration, which improves stability and reduces complexity of the landscape setup.

Disaster Recovery

Garden-setup also deploys an own etcd for the kube-apiserver. Because the kube-apiserver - and thus its etcd - is only being used for Gardener resources, restoring the state of a Gardener landscape from an etcd backup is significantly easier than it would be if the Gardener resources were mixed with other resources in the etcd.

Disadvantage: Two kubeconfigs

The major disadvantage of this approach is that two kubeconfigs are needed to operate the Gardener: one for the base cluster, where all the pods are running, and one for the 'virtual' cluster where the Gardener resources - shoot, seed, cloudprofile, ... - are maintained. The kubeconfig for the 'virtual' cluster can be found in the landscape folder at export/kube-apiserver/kubeconfig or it can be pulled from the secret garden-kubeconfig-for-admin in the garden namespace of the base cluster after the kube-apiserver component of garden-setup has been deployed.

TL;DR

Use the kubeconfig at export/kube-apiserver/kubeconfig to access the cluster where the Gardener resources - shoot, seed, cloudprofile, and so on - are maintained.

Configuration File acre.yaml

This file will be evaluated using spiff, a dynamic templating language for yaml files. For example, this simplifies the specification of field values that are used multiple times in the yaml file. For more information, see the spiff repository.

Please note that, for the sake of clarity, not all configuration options are listed in this readme. Instead, the more advanced configuration options have been moved into a set of additional documentation files. You can access these pages via their index and they are usually linked in their corresponding sections below.

landscape:
  name: <Identifier>                       # general Gardener landscape identifier, for example, `my-gardener`
  domain: <prefix>.<cluster domain>        # unique basis domain for DNS entries

  cluster:                                          # information about your base cluster
    kubeconfig: <relative path + filename>          # path to your `kubeconfig` file, rel. to directory `landscape` (defaults to `./kubeconfig`)
    networks:                                       # CIDR IP ranges of base cluster
      nodes: <CIDR IP range>
      pods: <CIDR IP range>
      services: <CIDR IP range>

  iaas:
    - name: (( iaas[0].type ))                   # name of the seed
      type: <gcp|aws|azure|alicloud|openstack|vsphere>    # iaas provider
      region: <major region>-<minor region>      # region for initial seed
      zones:                                     # remove zones block for Azure
        - <major region>-<minor region>-<zone>   # example: europe-west1-b
        - <major region>-<minor region>-<zone>   # example: europe-west1-c
        - <major region>-<minor region>-<zone>   # example: europe-west1-d
      credentials:                               # provide access to IaaS layer used for creating resources for shoot clusters
    - name:                                      # see above
      type: <gcp|aws|azure|alicloud|openstack>            # see above
      region: <major region>-<minor region>      # region for seed
      zones:                                     # remove zones block for Azure
        - <major region>-<minor region>-<zone>   # Example: europe-west1-b
        - <major region>-<minor region>-<zone>   # Example: europe-west1-c
        - <major region>-<minor region>-<zone>   # Example: europe-west1-d
      cluster:                                   # information about your seed's base cluster
        networks:                                # CIDR IP ranges of seed cluster
          nodes: <CIDR IP range>
          pods: <CIDR IP range>
          services: <CIDR IP range>
        kubeconfig:                              # kubeconfig for seed cluster
          apiVersion: v1
          kind: Config
          ...
      credentials:

  etcd:                                       # optional for gcp/aws/azure/alicloud/openstack, default values based on `landscape.iaas`
    backup:
      type: <gcs|s3|abs|oss|swift>                # type of blob storage
      resourceGroup:                          # Azure resource group you would like to use for your backup
      region: (( iaas.region ))               # region of blob storage (default: same as above)
      credentials: (( iaas.credentials ))     # credentials for the blob storage's IaaS provider (default: same as above)
    resources:                                # optional: override resource requests and limits defaults
      requests:
        cpu: 400m
        memory: 2000Mi
      limits:
        cpu: 1
        memory: 2560Mi

  dns:                                    # optional for gcp/aws/azure/openstack, default values based on `landscape.iaas`
    type: <google-clouddns|aws-route53|azure-dns|alicloud-dns|openstack-designate|cloudflare-dns|infoblox-dns>   # dns provider
    credentials: (( iaas.credentials ))   # credentials for the dns provider

  identity:
    users:
      - email:                                # email (used for Gardener dashboard login)
        username:                             # username (displayed in Gardener dashboard)
        password:                             # clear-text password (used for Gardener dashboard login)
      - email:                                # see above
        username:                             # see above
        hash:                                 # bcrypted hash of password, see above

  cert-manager:
    email:                                    # email for acme registration
    server: <live|staging|self-signed|url>    # which kind of certificates to use for the dashboard/identity ingress (defaults to `self-signed`)
    privateKey:                               # optional existing user account's private key

landscape.name

landscape:
  name: <Identifier>

Arbitrary name for your landscape. The name will be part of the names for resources, for example, the etcd buckets.

landscape.domain

domain: <prefix>.<cluster domain>

Basis domain for DNS entries. As a best practice, use an individual prefix together with the cluster domain of your base cluster.

landscape.cluster

cluster:
  kubeconfig: <relative path + filename>
  networks:
    nodes: <CIDR IP range>
    pods: <CIDR IP range>
    services: <CIDR IP range>

Information about your base cluster, where the Gardener will be deployed on.

landscape.cluster.kubeconfig contains the path to your kubeconfig, relative to your landscape directory. It is recommended to create a kubeconfig file in your landscape directory to be able to sync all files relevant for your installation with a git repository. This value is optional and will default to ./kubeconfig if not specified.

landscape.cluster.networks contains the CIDR ranges of your base cluster. Finding out CIDR ranges of your cluster is not trivial. For example, GKE only tells you a "pod address range" which is actually a combination of pod and service CIDR. However, since the kubernetes service typically has the first IP of the service IP range and most methods to get a kubernetes cluster tell you at least something about the CIDRs, it is usually possible to find out the CIDRs with a little bit of educated guessing.

landscape.iaas

iaas:
  - name: (( type ))                           # name of the seed
    type: <gcp|aws|azure|alicloud|openstack|vsphere>    # iaas provider
    region: <major region>-<minor region>      # region for initial seed
    zones:                                     # remove zones block for Azure
      - <major region>-<minor region>-<zone>   # example: europe-west1-b
      - <major region>-<minor region>-<zone>   # example: europe-west1-c
      - <major region>-<minor region>-<zone>   # example: europe-west1-d
    credentials:                               # provide access to IaaS layer used for creating resources for shoot clusters
  - name:                                      # see above
    type: <gcp|aws|azure|alicloud|openstack|vsphere>    # see above
    region: <major region>-<minor region>      # region for seed
    zones:                                     # remove zones block for Azure
      - <major region>-<minor region>-<zone>   # example: europe-west1-b
      - <major region>-<minor region>-<zone>   # example: europe-west1-c
      - <major region>-<minor region>-<zone>   # example: europe-west1-d
    cluster:                                   # information about your seed's base cluster
      networks:                                # CIDR IP ranges of seed cluster
        nodes: <CIDR IP range>
        pods: <CIDR IP range>
        services: <CIDR IP range>
      kubeconfig:                              # kubeconfig for seed cluster
        apiVersion: v1
        kind: Config
        ...
    credentials:

Contains the information where Gardener will create intial seed clusters and cloudprofiles to create shoot clusters.

Field Type Description Examples Iaas Provider Documentation
name Custom value Name of the seed/cloudprofile. Must be unique. gcp
type Fixed value IaaS provider for the seed. gcp
region IaaS provider specific Region for the seed cluster. The convention to use <major region>-<minor region> does not apply to all providers.

In Azure, use az account list-locations to find out the location name (name attribute = lower case name without spaces).
europe-west1 (GCP)

eu-west-1 (AWS)

eu-central-1 (Alicloud)

westeurope (Azure)
GCP (HowTo), GCP (overview); AWS (HowTo), AWS (Overview); Azure (Overview), Azure (HowTo)
zones IaaS provider specific Zones for the seed cluster. Not needed for Azure. europe-west1-b (GCP)

GCP (HowTo), GCP (overview); AWS (HowTo), AWS (Overview)
credentials IaaS provider specific Credentials in a provider-specific format. See table with yaml keys below. GCP, AWS, Azure
cluster.kubeconfig Kubeconfig The kubeconfig for your seed base cluster. Must have basic auth authentification.
cluster.networks CIDRs The CIDRs of your seed cluster. See landscape.cluster for more information.

Here a list of configurations can be given. The setup will create one cloudprofile and seed per entry. Currently, you will have to provide the cluster you want to use as a seed - in future, the setup will be able to create a shoot and configure that shoot as a seed. The type should match the type of the underlying cluster.

The first entry of the landscape.iaas list is special:

  • It has to exist - the list needs at least one entry.
  • Don't specify the cluster node for it - it will configure your base cluster as seed.
    • Its type should match the one of your base cluster.

See the advanced documentation for more advanced configuration options and information about Openstack.

Shooted Seeds

It's also possible to have the setup create shoots and then configure them as seeds. This has advantages compared to configuring existing clusters as seeds, e.g. you don't have to provide the clusters as they will be created automatically, the shooted seed clusters can leverage the Gardener's autoscaling capabilities, ...

How to configure shooted seeds is explained in the advanced documentation.

Credentials

The credentials will be used to give Gardener access to the IaaS layer:

  • To create a secret that will be used on the Gardener dashboard to create shoot clusters.
  • To allow the control plane of the seed clusters to store the etcd backups of the shoot clusters.

Use the following yaml keys depending on your provider (excerpts):

AWS GCP
credentials: 
accessKeyID: ...
secretAccessKey: ...
credentials: 
serviceaccount.json: |
{
"type": "...",
"project_id": "...",
...
}
Azure Openstack
credentials:
clientID: ...
clientSecret: ...
subscriptionID: ...
tenantID: ...
credentials:
username: ...
password: ...
tenantName: ...
domainName: ...
authURL: ...
region: ... # DNS only
Alicloud Other
credentials:
accessKeyID: ...
accessKeySecret: ...

The region field in the openstack credentials is only evaluated within the dns block (as iaas and etcd.backup have their own region fields, which will be used instead).

landscape.etcd

etcd:
  backup:
    # active: true
    type: <gcs|s3|abs|swift|oss>
    resourceGroup: ...
    region: (( iaas.region ))
    credentials: (( iaas.credentials ))
    # schedule: "0 */24 * * *"          # optional, default: 24h
    # maxBackups: 7                     # optional, default: 7

Configuration of what blob storage to use for the etcd key-value store. If your IaaS provider offers a blob storage you can use the same values for etc.backup.region and etc.backup.credentials as above for iaas.region and iaas.credentials correspondingly by using the (( foo )) expression of spiff. If the type of landscape.iaas[0] is one of gcp, aws, azure, alicloud, or openstack, this block can be defaulted - either partly or as a whole - based on values from landscape.iaas. The resourceGroup, which is necessary for Azure, cannot be defaulted and must be specified. Make sure that the specified resourceGroup is empty and unused as deleting the cluster using sow delete all deletes this resourceGroup.

Field Type Description Example Iaas Provider Documentation
backup.active Boolean If set to false, deactivates the etcd backup for the virtual cluster etcd. Defaults to true. true n.a.
backup.type Fixed value Type of your blob store. Supported blob stores: gcs (Google Cloud Storage), s3 (Amazon S3), abs (Azure Blob Storage), oss (Alicloud Object Store), and swift (Openstack Swift). gcs n.a.
backup.resourceGroup IaaS provider specific Azure specific. Create an Azure blob store first which uses a resource group. Provide the resource group here. my-Azure-RG Azure (HowTo)
backup.region IaaS provider specific Region of blob storage. (( iaas.region )) GCP (overview), AWS (overview)
backup.credentials IaaS provider specific Service account credentials in a provider-specific format. (( iaas.creds )) GCP, AWS, Azure

landscape.dns

dns:
  type: <google-clouddns|aws-route53|azure-dns|openstack-designate|cloudflare-dns|infoblox-dns>
  credentials:

Configuration for the Domain Name Service (DNS) provider. If your IaaS provider also offers a DNS service you can use the same values for dns.credentials as for iaas.creds above by using the (( foo )) expression of spiff. If they belong to another account (or to another IaaS provider) the appropriate credentials (and their type) have to be configured. Similar to landscape.etcd, this block - and parts of it - are optional if the type of landscape.iaas[0] is one of gcp, aws, azure, alicloud, or openstack. Missing values will be derived from landscape.iaas.

Field Type Description Example IaaS Provider Documentation
type Fixed value Your DNS provider. Supported providers: google-clouddns (Google Cloud DNS), aws-route53 (Amazon Route 53), alicloud-dns (Alicloud DNS), azure-dns (Azure DNS), openstack-designate (Openstack Designate), cloudflare-dns (Cloudflare DNS), and infoblox-dns (Infoblox DNS). google-clouddns n.a.
credentials IaaS provider specific Service account credentials in a provider-specific format (see above). (( iaas.credentials )) GCP, AWS, Azure

Cloudflare Credentials

The credentials to use Cloudflare DNS consist of a single key apiToken, containing your API token.

Infoblox Credentials and Configuration

For Infoblox DNS, you have to specify USERNAME, PASSWORD, and HOST in the credentials node. For a complete list of optional credentials keys see here

landscape.identity

identity:
  users:
    - email:
      username:
      password:
    - email:
      username:
      hash:

Configures the identity provider that allows access to the Gardener dashboard. The easiest method is to provide a list of users, each containing email, username, and either a clear-text password or a bcrypted hash of the password. You can then login into the dashboard using one of the specified email/password combinations.

landscape.ingress

ingress:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"  # example for internal loadbalancers on aws
    ...

You can add annotations for the ingress controller load balancer service. This can be used for example to deploy an internal load balancer on your cloud provider (see the example for aws above).

landscape.cert-manager

  cert-manager:
    email:                                   
    server: <live|staging|self-signed|url>
    privateKey: # optional

The setup deploys a cert-manager to provide a certificate for the Gardener dashboard, which can be configured here.

The entire landscape.cert-manager block is optional.

If not specified, landscape.cert-manager.server defaults to self-signed. This means, that a selfs-signed CA will be created, which is used by the cert-manager (using a CA issuer) to sign the certificate. Since the CA is not publicly trusted, your webbrowser will show a 'untrusted certificate' warning when accessing the dashboard. The landscape.cert-manager.email field is not evaluated in self-signed mode.

If set to live, the cert-manager will use the letsencrypt ACME server to get trusted certificates for the dashboard. Beware the rate limits of letsencrypt. Letsencrypt requires an email address and will send information about expiring certificates to that address. If landscape.cert-manager.email is not specified, landscape.identity.users[0].email will be used. One of the two fields has to be present.

If set to staging, the cert-manager will use the letsencrypt staging server. This is for testing purposes mainly. The communication with letsencrypt works exactly as for the live case, but the staging server does not produce trusted certificates, so you will still get the browser warning. The rate limits are significantly higher for the staging server, though.

If set to anything else, it is assumed to be the URL of an ACME server and the setup will create an ACME issuer for it.

See the advanced configuration for more configuration options.

If the given email address is already registers at letsencrypt, you can specify the private key of the associated user account with landscape.cert-manager.privateKey.

Uninstall Gardener

  1. Run sow delete -A to delete all components from your base Kubernetes cluster in inverse order.

  2. During the deletion, the corresponding contents in directories gen, export, and state in your landscape directory are deleted automatically as well.

Adding a Service Account Authentication Token to a Kubeconfig File

  1. create a new service account in the kube-system namespace
    kubectl -n kube-system create serviceaccount <service-account-name>
  2. Create a new clusterrolebinding with cluster administration permissions and bind it to the service account you just created
    kubectl create clusterrolebinding <binding-name> --clusterrole=cluster-admin --serviceaccount=kube-system:<service-account-name>
  3. Obtain the name of the service account authentication token and assign its value to an environment variable
    TOKENNAME=`kubectl -n kube-system get serviceaccount/<service-account-name> -o jsonpath='{.secrets[0].name}'`
  4. Obtain the value of the service account authentication token and assign its value (decoded from base64) to an environment variable. These instructions assume you specify TOKEN as the name of the environment variable.
    TOKEN=`kubectl -n kube-system get secret $TOKENNAME -o jsonpath='{.data.token}'| base64 --decode`
  5. Add the service account (and its authentication token) as a new user definition in the kubeconfig file
    kubectl config set-credentials <service-account-name> --token=$TOKEN
  6. Set the user specified in the kubeconfig file for the current context to be the new service account user you created
    kubectl config set-context --current --user=<service-account-name>
  7. Your Kubeconfig should now contain the token to authenticate the Service Account

Example of the SA user in the kubeconfig:

users:
- name: <service-account-name>
  user:
    token: <service-account-token>

Most Important Commands and Directories

Commands

These are the most important sow commands for deploying and deleting components:

Command Use
sow <component> Same as sow deploy <component>.
sow delete <component> Deletes a single component
sow delete -A Deletes all components in the inverse order
sow delete all Same as sow delete -A
sow delete -a <component> Deletes a component and all components that depend on it (including transitive dependencies)
sow deploy <component> Deploys a single component. The deployment will fail if the dependencies have not been deployed before.
sow deploy -A Deploys all components in the order specified by sow order -A
sow deploy -An Deploys all components that are not deployed yet
sow deploy all Same as sow deploy -A
sow deploy -a <component> Deploys a component and all of its dependencies
sow help Displays a command overview for sow
sow order -a <component> Displays all dependencies of a given component (in the order they should be deployed in)
sow order -A Displays the order in which all components can be deployed
sow url Displays the URL for the Gardener dashboard (after a successful installation)

Directories

After using sow to deploy the components, you will notice that there are new directories inside your landscape directory:

Directory Use
gen Temporary files that are created during the deployment of components, for example, generated manifests.
export Allows communication (exports and imports) between components. It also contains the kubeconfig for the virtual cluster that handles the Gardener resources.
state Important state information of the components is stored here, for example, the terraform state and generated certificates. It is crucial that this directory is not deleted while the landscape is active. While the contents of the export and gen directorys will be overwritten when a component is deployed again, the contents of state will be reused instead. In some cases, it is necessary to delete the state of a component before deploying it again, for example if you want to create new certificates for it.

garden-setup's People

Contributors

afritzler avatar berendt avatar christianhuening avatar dergeberl avatar diaphteiros avatar einfachnuralex avatar gardener-robot-ci-1 avatar gardener-robot-ci-2 avatar gardener-robot-ci-3 avatar harshcommits avatar herrmannhinz avatar ialidzhikov avatar je-hal avatar jguipi avatar juergenmair avatar kayrus avatar kevin-lacoo avatar lotharbach avatar mandelsoft avatar martinweindel avatar mssedusch avatar mward29 avatar n-boshnakov avatar petersutter avatar raphaelvogel avatar rootfs avatar sayakmukhopadhyay avatar simonkienzler avatar sluetze avatar vchrisst avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

garden-setup's Issues

document all configuration options available in acre.yaml

The user has to provide an acre.yaml file containing the configuration. Most relevant parts are documented here. However, some components support additional options or specific edge cases that require further configuration - which is partly not documented at all.

To not overstrain new users with a huge README, these features and their corresponding configuration options should be documented in an extra document (or multiple documents).

Structure of the documentation can be discussed here.

Dashboard connectivity issues on Azure

When the Gardener is deployed on Azure, the dashboard sometimes doesn't work properly.

Dashboard logs show something like this:

2019-04-16T11:11:12.924Z - debug: watch closed 4000 Client network socket disconnected before secure TLS connection was established
2019-04-16T11:11:12.923Z - error: watch projects disconnected { Error: Client network socket disconnected before secure TLS connection was established
    at TLSSocket.onConnectEnd (_tls_wrap.js:1088:19)
    at Object.onceWrapper (events.js:277:13)
    at TLSSocket.emit (events.js:194:15)
    at endReadableNT (_stream_readable.js:1125:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  code: 'ECONNRESET',
  path: undefined,
  host: 'api.ingress.mycluster.org',
  port: 443,
  localAddress: undefined }

The problem will disappear/reappear from time to time.
Everything else seems to work fine.

Print diff before performing a helm upgrade or kubectl apply

What would you like to be added:

I'd like to see the diff of what is going to be changed during the helm upgrade or kubectl apply -f file.yaml

Why is this needed:

It is not clear what has been changed, e.g.

controllerregistration.core.gardener.cloud/dns-external configured

or

Running helm upgrade for gardener ...
Release "gardener" has been upgraded. Happy Helming!

Some references:

Running the installation using kind

What would you like to be added:
I would like to be able to run the installation as part of my CI/CD process using kind as my k8s infrastructure and trvis-ci as my executor

Why is this needed:
In order to test applications developed on k8s that depend on Gardener, without utilizing a real cluster in a promise to reduce cost.

Deletion of a garden cluster does not cleanup shoots

When a gardener cluster contains shoots and garden-setup wants to delete it the deletion process stuck if there are still shoots in the cluster.
The garden-setup should rather delete all shoots first before deleting the cluster.

Wait for tiller pod to be ready

*** sowing species ingress-controller/controller
Plugin: helm[deploy] (/sow/plugins/helm/plugin)
    using /src/./kubeconfig
Running helm upgrade for nginx-ingress-controller ...
Error: could not find a ready tiller pod

Gardener does not get cleaned up correctly if the previous deployment failed

What happened:
Garden setup is unable to delete its deployed resources properly when garden-setup fails during creation before gardener resources (like shoots) can be deployed into the virtual api-server.
Garden-setup checks if there are any shoots left in the virtual apiserver before it is clenaed up. But when the apiserver is not aware of a resource like shoot the deletion fails with this error.

*** deleting species gardener/runtime
using local action script
Plugin: shoot-check[delete] (/tmp/garden/crop/plugins/shoot-check/plugin)
   using inline kubeconfig
Error: error: the server doesn't have a resource type "shoots"

What you expected to happen:
Garden-setup should cleanup correctly when shoots cannot be found or are unknown.

Cloud profiles update doesn't work properly

What happened:

sow uses kubectl apply -f cloudprofile.yaml, which does only the update of the yaml file, but doesn't remove the elements, which are not listed in the yaml file anymore.

What you expected to happen:

If you remove vulnerable/outdated k8s versions from the cloud profile yaml file, they are not actually removed.

How to reproduce it (as minimally and precisely as possible):

create a cloud profile with the following k8s versions:

  • 1.15.0
  • 1.15.3
  • 1.15.4

then remove the outdated k8s versions and keep only:

  • 1.15.4

run the sow, it will produce the following output:

manifest for CloudProfile(garden.sapcloud.io/v1beta1): region1
cloudprofile.garden.sapcloud.io/region1 configured

but the 1.15.0 and 1.15.3 k8s versions are not removed from the cloud profile.

Environment:

  • Version of garden-setup (release or commit): 1.15.0

sowing species gardener/virtual failing

What happened:
Followed the installation steps found gardner/garden-setup but once I reached to:

`


*** sowing species gardener/virtual
executing configured plugins for action prepare
using config from path git
Plugin: git[prepare] (/sow/plugins/git/plugin)
Cloning repo https://github.com/gardener/gardener.git (tag v1.1.4)...
Initialized empty Git repository in /mounted/Users/reza/projects/landscape/gen/gardener/virtual/git/repo/.git/
Reinitialized existing Git repository in /mounted/Users/reza/projects/landscape/gen/gardener/virtual/git/repo/.git/
remote: Enumerating objects: 8642, done.
remote: Counting objects: 100% (8642/8642), done.
remote: Compressing objects: 100% (6666/6666), done.
remote: Total 8642 (delta 2363), reused 5074 (delta 1433), pack-reused 0
Receiving objects: 100% (8642/8642), 11.94 MiB | 662.00 KiB/s, done.
Resolving deltas: 100% (2363/2363), done.
From https://github.com/gardener/gardener

  • tag v1.1.4 -> FETCH_HEAD
    executing configured plugins for action deploy
    Plugin: pinned[deploy] (builtin)
    using config from path gardener
    Plugin: helm[deploy]{gardener} (/sow/plugins/helm/plugin)
    using /mounted/Users/reza/projects/landscape/./kubeconfig
    Running helm template for gardener ...
    walk.go:74: found symbolic link in path: /mounted/Users/reza/projects/landscape/gen/gardener/virtual/git/repo/charts/gardener/controlplane/charts/application/charts/utils-common resolves to /mounted/Users/reza/projects/landscape/gen/gardener/virtual/git/repo/charts/gardener/controlplane/charts/utils-common
    walk.go:74: found symbolic link in path: /mounted/Users/reza/projects/landscape/gen/gardener/virtual/git/repo/charts/gardener/controlplane/charts/application/charts/utils-common/charts/utils-templates resolves to /mounted/Users/reza/projects/landscape/gen/gardener/virtual/git/repo/charts/utils-templates
    walk.go:74: found symbolic link in path: /mounted/Users/reza/projects/landscape/gen/gardener/virtual/git/repo/charts/gardener/controlplane/charts/application/charts/utils-common/values.yaml resolves to /mounted/Users/reza/projects/landscape/gen/gardener/virtual/git/repo/charts/gardener/controlplane/values.yaml
    walk.go:74: found symbolic link in path: /mounted/Users/reza/projects/landscape/gen/gardener/virtual/git/repo/charts/gardener/controlplane/charts/application/charts/utils-templates resolves to /mounted/Users/reza/projects/landscape/gen/gardener/virtual/git/repo/charts/utils-templates
    walk.go:74: found symbolic link in path: /mounted/Users/reza/projects/landscape/gen/gardener/virtual/git/repo/charts/gardener/controlplane/charts/application/values.yaml resolves to /mounted/Users/reza/projects/landscape/gen/gardener/virtual/git/repo/charts/gardener/controlplane/values.yaml
    using config from path kubectl_apply

Plugin: kubectl[deploy]{kubectl_apply} (/sow/plugins/kubectl/plugin)
manifest for Namespace(v1): garden
error: unable to recognize "/mounted/Users/reza/projects/landscape/gen/gardener/virtual/kubectl_apply/manifests/0-0-Namespace.garden.yaml": Get https://api.ing.gardner.jupitercloud.dev:443/api?timeout=32s: EOF
command failed, waiting for 15 seconds and trying again (4 times)
error: unable to recognize "/mounted/Users/reza/projects/landscape/gen/gardener/virtual/kubectl_apply/manifests/0-0-Namespace.garden.yaml": Get https://api.ing.gardner.jupitercloud.dev:443/api?timeout=32s: EOF
command failed, waiting for 15 seconds and trying again (3 times)
error: unable to recognize "/mounted/Users/reza/projects/landscape/gen/gardener/virtual/kubectl_apply/manifests/0-0-Namespace.garden.yaml": Get https://api.ing.gardner.jupitercloud.dev:443/api?timeout=32s: EOF
command failed, waiting for 15 seconds and trying again (2 times)
error: unable to recognize "/mounted/Users/reza/projects/landscape/gen/gardener/virtual/kubectl_apply/manifests/0-0-Namespace.garden.yaml": Get https://api.ing.gardner.jupitercloud.dev:443/api?timeout=32s: EOF
command failed, waiting for 15 seconds and trying again (1 times)
error: unable to recognize "/mounted/Users/reza/projects/landscape/gen/gardener/virtual/kubectl_apply/manifests/0-0-Namespace.garden.yaml": Get https://api.ing.gardner.jupitercloud.dev:443/api?timeout=32s: EOF
Error: kubectl command failed and retry limit reached
~/projects/landscape ❯ sow url 19m 20s ⎈ gardner-cluster 22:12:51
cat: can't open '/mounted/Users/reza/projects/landscape/export/dashboard/dashboard_url': No such file or directory
~/projects/landscape ❯ 7s ⎈ gardner-cluster 22:14:07

`

All installation screen logs are here: Installation Log

How to reproduce it (as minimally and precisely as possible):

acre.yaml and Installation Log

Environment: POC

  • Version of garden-setup (release or commit) Latest from the setup instruction page
  • Versions of components (only needed if you overwrote the defaults)
  • Where does the underlying base cluster come from and which operating system does it use? ubuntu
  • Which cloud provider is configured for the setup? GCP

Add an ability to define ClusterRoleBinding for the garden.sapcloud.io:system:project-creation ClusterRole

What would you like to be added:

By default the project creation is available only for admin: https://github.com/gardener/garden-setup/blob/90980da3eaef4af204776b852d8a64779960f6a0/components/dashboard/deployment.yaml#L89..L97

would be great to have an ability to specify the ClusterRoleBinding for:

subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:authenticated

so every authenticated user can create a project.

Why is this needed:

simplify the the landscape setup

Cannot connect to virtual garden API-server

What happened:
When setting the KUBECONFIG to target the API-server of the virtual garden cluster and executing any kubectl-command fails with:

Unable to connect to the server: net/http: TLS handshake timeout

What you expected to happen:
Expect kubectl to successfully execute, e.g. show pods.

Anything else we need to know?:
Executing kubectl-commands used to work for the virtual garden cluster, so the KUBECONFIG should be correct. Not sure why this is no longer the case. Maybe due to hibernation of the base cluster?

Environment:

  • Version of garden-setup: 8aabff3
  • Where does the underlying base cluster come from and which operating system does it use? The base cluster is also a gardener cluster.
  • Which cloud provider is configured for the setup? AWS

Support multiple cloudproviders

What would you like to be added:
Add initial support (cloudprofile, secrets and extension controllers) for multiple cloudproviders.

Why is this needed:
This is needed to validate multiple cloudprovider with one gardener installation.

Deploy another shoot as seed

To test specific behavior of seeds it would be great when the gardener-setup creates a shoot and registers the existing seed as a soil and the created shoot as a seed.

Maybe make this behavior configurable as it is not needed in every use case.

Multiple landscape profiles are not supported

For some reason only the one profile is handled by the sow deploy gardencontent/profiles.

acre.yaml:

... ... ...
  profiles:
  - name: region1
    type: openstack
    config:
      floatingPools: (( landscape.iaas.openstack.floatingPools ))
      loadBalancerProviders: (( landscape.iaas.openstack.loadBalancerProviders ))
      machineImages: (( landscape.iaas.openstack.machineImages ))
      machineTypes: (( landscape.iaas.openstack.machineTypes ))
      keystoneURL: (( landscape.iaas.credentials.authURL ))
      zones:
      - region: (( landscape.iaas.region ))
        names: (( landscape.iaas.zones ))
  - name: region2
    type: openstack
    config:
      floatingPools: (( landscape.iaas.openstack.floatingPools ))
      loadBalancerProviders: (( landscape.iaas.openstack.loadBalancerProviders ))
      machineImages: (( landscape.iaas.openstack.machineImages ))
      machineTypes: (( landscape.iaas.openstack.machineTypes ))
      keystoneURL: https://region2
      zones:
      - region: region2
        names: ['az1']
... ... ...

In this case only one region2 profile will be created. If you swap the region1 and region2 order, then region1 will be created, but region2 will be removed:

$ sow -v deploy gardencontent/profiles
... ... ...
cluster https://api.ingress:443
  found deployed objects: 2
  found currect objects: 1
  found deployed obsolete objects: 1
    CloudProfile(garden.sapcloud.io/v1beta1): region2
kubectl --kubeconfig landscape/gen/gardencontent/profiles/kubectl/kubeconfig.tmp delete -f landscape-1/gen/gardencontent/profiles/kubectl/manifest.tmp
cloudprofile.garden.sapcloud.io "region2" deleted
generating exports
*** species gardencontent/profiles deployed

Dynamically nested seeds

What happened:

I'd like to have the following structure:

root: (kubernikus cluster)
  seeds:
    - soil:
        seeds:
          - region1
          - region2

Unfortunately so far I can create only soil, the seeds inside the soil are not allowed.

Enable centralized gardenlet

Currently, garden-setup supports only having a gardenlet for every seed and not a setup that uses only one central gardenlet. Both variants should be supported, with the current one being the default.

Dashboard doesn't work with self-signed certificates

When trying to login into the dashboard, after entering username and password, one gets redirected to the main page and a red error pops up saying "self-signed certificate in certificate chain". Logging in is not possible.
This doesn't happen when trusted certificates are used (using the letsencrypt live server).

It's probably an error in the dashboard that was introduced with the upgrade to 1.33.0, so 5423504 should be the last commit that works.

Add a retry for the kube api connection

What happened:

Deploying gardener from scratch may fail because of the loadbalancer readiness:

*** sowing species gardener/virtual
Plugin: git[deploy] (/sow/plugins/git/plugin)
Cloning repo https://github.com/gardener/gardener.git (tag 0.27.6
)...
Initialized empty Git repository in /mounted/dev/gen/gardener/virtual/gardener_git/repo/.git/
Reinitialized existing Git repository in /mounted/dev/gen/gardener/virtual/gardener_git/repo/.git/
remote: Enumerating objects: 7255, done.
remote: Counting objects: 100% (7255/7255), done.
remote: Compressing objects: 100% (5575/5575), done.
remote: Total 7255 (delta 2049), reused 3976 (delta 1210), pack-reused 0
Receiving objects: 100% (7255/7255), 9.49 MiB | 8.29 MiB/s, done.
Resolving deltas: 100% (2049/2049), done.
From https://github.com/gardener/gardener
 * tag               0.27.6     -> FETCH_HEAD
Plugin: helm[deploy] (/sow/plugins/helm/plugin)
    using /mounted/dev/kubeconfig_sa
Running helm template for gardener ...
Plugin: kubectl[deploy] (/sow/plugins/kubectl/plugin)
manifest for Namespace(v1): garden
error: unable to recognize "/mounted/dev/gen/gardener/virtual/kubectl_apply/manifests/0-0-Namespace.garden.yaml": Get https://gardener:443/api?timeout=32s: net/http: TLS handshake timeout

Sometimes it takes time for the LB to be available.

What you expected to happen:

sow should show that the LB is not available, but it will retry the action.

Configuration of OpenStack CloudProfile

What would you like to be added:
In the acre.yamlwe should allow the user to configure the following things:

  • Machine Image names
  • Provider network
  • Upcomming Provider pool Lbaas classes
  • etc.

Why is this needed:
Since every OpenStack installation is different, we need a way to customize the way the CloudProfile will look like.

sow deploy -A errors at bucket creation

What happened:

google_storage_bucket.bucket: Creating...
  bucket_policy_only: "" => "<computed>"
  force_destroy:      "" => "true"
  location:           "" => "EU-WEST4"
  name:               "" => "xxxx-gardener-etcd-bucket"
  project:            "" => "<computed>"
  self_link:          "" => "<computed>"
  storage_class:      "" => "COLDLINE"
  url:                "" => "<computed>"

Error: Error applying plan:

1 error(s) occurred:

* google_storage_bucket.bucket: 1 error(s) occurred:

* google_storage_bucket.bucket: googleapi: Error 400: Invalid Value, invalid

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

What you expected to happen:
the install to finish

How to reproduce it (as minimally and precisely as possible):
cert-manager.server: self-signed

Anything else we need to know?:

Environment:

  • Version of garden-setup (release or commit)
    sow 1.0.10-dev

  • Versions of components (only needed if you overwrote the defaults)

  • Where does the underlying base cluster come from and which operating system does it use?
    gke (cos)

  • Which cloud provider is configured for the setup?
    gcp

  • Other:

Backup garden-setup state

What would you like to be added:
It would be good if the state of a garden-setup installation (state, export and gen) is backuped somewhere that is not the current local disk. (e.g. S3/GCS or kubernetes itself)

Why is this needed:
This is needed to recover or delete a cluster (maybe broken) if the state on the local machine is gone.

Gardener extensions are not cleaned up properly

What happened:
Deleted all components with sow delete -A. Afterwards, sow order -A shows a still partially existent gardener/extensions component. The state folder does still exist for the extensions. The extension namespaces and pods in there are still there.

What you expected to happen:
Extensions should be removed from cluster and state folder.

Environment:

  • Version of garden-setup: 1.3.0

sow deploy -A halts

What happened:
sow -A halts at

generating exports
*** species cert-manager/solver deployed

*****************************
*** sowing species cert-manager/cert
using local action script
Plugin: kubectl[deploy] (/sow/plugins/kubectl/plugin)
manifest for Certificate(certmanager.k8s.io/v1alpha1): garden/dashboard-identity-ingress
certificate.certmanager.k8s.io/dashboard-identity-ingress configured
cluster https://IPMASKED
  found deployed objects: 1
  found currect objects: 1
The export generation waits for the certificate, which may take some time.
generating exports

What you expected to happen:
for the installer to continue and finish

How to reproduce it (as minimally and precisely as possible):
cert-manager.server: live

Anything else we need to know?:

Environment:

  • Version of garden-setup (release or commit)
    sow: 1.0.10-dev

  • Versions of components (only needed if you overwrote the defaults)

  • Where does the underlying base cluster come from and which operating system does it use?
    gke (cos)

  • Which cloud provider is configured for the setup?
    gcp

  • Other:

sow delete doesn't delete the soil cluster

What happened:

during the shoot deletion the must have a "confirmation.garden.sapcloud.io/deletion" annotation to delete error occurs

What you expected to happen:

annotation have to be performed automatically

Automatically Roll Deployments When ConfigMaps/Secrets are changed

What would you like to be added:

Add a checksum annotation inside the helm deployment templates (pods), e.g.

annotations:
  checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
  checksum/secret: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }}

Why is this needed:

helm doesn't restart pods, when secrets/configmaps are changed.

Create garden project

What would you like to be added:
In the initial setup we should create a garden project and also create the corresponding labels on the garden namespace.

garden namespace

labels:
  garden.sapcloud.io/role: project
  project.garden.sapcloud.io/name: garden

Why is this needed:
To create seeds and soils in the garden namespace.

sowing species ingress-controller has no deployed releases

What happened:
sow deploy -A
What you expected to happen:
*** sowing species ingress-controller no plugins found for action prepare executing configured plugins for action deploy using config from path ingresscontroller Plugin: helm[deploy]{ingresscontroller} (/sow/plugins/helm/plugin) using /mounted/root/jiawen/landscape/./kubeconfig Running helm upgrade for nginx-ingress-controller ... Error: UPGRADE FAILED: "nginx-ingress-controller" has no deployed releases command failed, waiting for 15 seconds and trying again (4 times) Error: UPGRADE FAILED: "nginx-ingress-controller" has no deployed releases command failed, waiting for 15 seconds and trying again (3 times) Error: UPGRADE FAILED: "nginx-ingress-controller" has no deployed releases command failed, waiting for 15 seconds and trying again (2 times) Error: UPGRADE FAILED: "nginx-ingress-controller" has no deployed releases command failed, waiting for 15 seconds and trying again (1 times) Error: UPGRADE FAILED: "nginx-ingress-controller" has no deployed releases Error: helm command failed and retry limit reached
How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • Version of garden-setup (release or commit):origin/master
  • Versions of components (only needed if you overwrote the defaults)
  • Where does the underlying base cluster come from and which operating system does it use?
  • Which cloud provider is configured for the setup?:openstack
  • Other:

Unable to deploy gardener

$ sow order -A
Unable to find image 'eu.gcr.io/gardener-project/sow:1.0.7-dev' locally
docker: Error response from daemon: manifest for eu.gcr.io/gardener-project/sow:1.0.7-dev not found.
See 'docker run --help'.

Use different/specific kubeconfigs for accessing the virtual cluster

Currently, garden-setup generates one admin kubeconfig for the 'virtual' cluster when it is created. This kubeconfig is then given to each component that needs access to the virtual cluster, e.g. Gardener itself, the dashboard, ...

Instead of having the same admin kubeconfig everywhere, it would be cleaner to create a serviceaccount for each component and generate a kubeconfig specifically for that component using the serviceaccount. This would allow to restrict the privileges of the components to the necessary minimum and provide other smaller benefits.

Support for new AWS region us-gov-west-1

What would you like to be added:
While there are AMIs setup for us-gov-west-1 AWS region, there is no profile setup.

Why is this needed:
Adding the profile will allow deploying Gardener to us-gov-west-1 without the user having to specifically define the region in their acre.yaml.

etcd statefulset upgrade has been failed

What happened:

Latest garden setup fails to upgrade the etcd clusters. It leads to unresponsive etcd service in all the gardener clusters.

I fixed the root cluster, but then the new deployment has been propagated to the soil and seed clusters.

https://gist.github.com/kayrus/31551685d523cbf84039c31ee6857366

For some reason there are no backup sidecars defined and probes fail.

What you expected to happen:

An upgrade should happen without the broken etcd service.

How to reproduce it (as minimally and precisely as possible):

I guess that the upgrade from 0.28.0 to 0.28.1 caused the issue.

/cc @afritzler

"no dns label" error while installing on aws from https://gardener.cloud/installer/

In the installer I set up all fields according description and video from https://gardener.cloud/installer/
I am using eks cluster on aws.

After executing:
kubectl apply -f gardener-deploy.yaml

I get in the logs:

error generating manifest: unresolved nodes:
	(( validate(landscape.name, "dnslabel") ))	in /landscape/crop/acre.yaml	validation.name	()	*condition 1 failed: is no dns label: [must be no more than 63 characters] 

I am not able to find where the dnslabel comes from nor what is the prerequisite of setting it.

Anything I should check?

acre.yaml conversion not picking up resourceGroup on Azure/AKS

What happened:

Seeing the following error, after preparing the gardener-deploy.yaml from https://gardener.cloud/installer/ page for Azure/AKS.

Components:
2020/04/22 13:52:37 error generating manifest: unresolved nodes:
(( validate(landscape.etcd.backup, (types.etcd_backup.[landscape.etcd.backup.type]).config) )) in /landscape/crop/acre.yaml validation.etcd.backup.provider_config () *condition 1 failed: has no field "resourceGroup"
error classification:
*: error in local dynaml expression
@: dependent of or involved in a cycle
-: depending on a node with an error
�[0;91m�[1mError: processing acre.yaml�[0m

What you expected to happen:
Resource Group is not asked on the installer page and this seems to be missing in acre.yaml. Even after updating the resourceGroup in the section:
etcd:
backup:
resourceGroup: gardener
type: abs
region: eastus
credentials: *ref_0

and updating the secret, it is not picked up by the installer.

How to reproduce it (as minimally and precisely as possible):

Provide all the details for Azure on the installer page https://gardener.cloud/installer/
And use the gardener-deploy.yaml to deploy into the AKS cluster

Anything else we need to know?:

Environment:
Azure/AKS

  • Gardener version: stable
  • Kubernetes version (use kubectl version):
    Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:20:10Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"windows/amd64"}
    Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.10", GitCommit:"150f36044fe31bee3891b5a3fae69c17237e022c", GitTreeState:"clean", BuildDate:"2020-02-21T10:02:50Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration:
    Azure
  • Others:

Support cutomization of the cloudprofile

What would you like to be added:
Add the feature to cutomize specific sections of the deployed cloudprofile (e.g. k8s versions)

Why is this needed:
This feature is needed if we want to validate new kubernetes versions (patch or minor) without modifying garden-setup

error generating manifest: unresolved nodes:

What happened:
sow deploy -A
What you expected to happen:
generating exports 2020/03/12 02:21:46 error generating manifest: unresolved nodes: (( sync(exec_uncached(command)) )) in /mounted/root/jiawen/landscape/crop/components/ingress-controller/export.yaml temp.target () *sync timeout reached (( defined(.temp.target.ip) ? "A" : "CNAME" )) in /mounted/root/jiawen/landscape/crop/components/ingress-controller/export.yaml export.type () (( .temp.target.ip || .temp.target.hostname )) in /mounted/root/jiawen/landscape/crop/components/ingress-controller/export.yaml export.address () -'.temp.target' unresolved (( .temp.target.ip || .temp.target.hostname )) in /mounted/root/jiawen/landscape/crop/components/ingress-controller/export.yaml export.address () -'.temp.target' unresolved error classification: *: error in local dynaml expression @: dependent of or involved in a cycle -: depending on a node with an error Error: processing exports
How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:
acre.yaml some config:

landscape:
  name: svc-cluster-local
  domain: svc.cluster.local
  cluster:
    kubeconfig: ./kubeconfig
    iaas: openstack
    networks:
      pods: 100.64.0.0/10
      nodes: 172.16.55.0/24
      services: 10.96.0.0/12 # change to under actual undercloud service ip cidr
  iaas:
    - name: openstack
      type: openstack
      mode: seed
      region: RegionOne
      zones:
      - nova
      credentials: *openstack
      floatingPools:
      - name: public
      loadBalancerProviders:
      - name: haproxy
      extensionConfig:
        machineImages:
          - name: coreos
            versions:
              - image: coreos-2345.3.0
                version: 2345.3.0
          - name: ubuntu
            versions:
              - image: ubuntu-18.04
                version: 18.04
      machineImages:
        - name: coreos
          versions:
          - version: 2345.3.0
        - name: ubuntu
          versions:
          - version: "18.04"
      machineTypes:
        - name: 2C4G50G
          cpu: "2"
          gpu: "0"
          memory: 4Gi
          usable: true
          storage:
            class: ""
            type: default
            size: 50Gi
        - name: 4C8G50G
          cpu: "4"
          gpu: "0"
          memory: 8Gi
          usable: true
          storage:
            class: ""
            type: default
            size: 50Gi

Environment:

  • Version of garden-setup (release or commit)
    2.2.0-dev
  • Versions of components (only needed if you overwrote the defaults)
  • Where does the underlying base cluster come from and which operating system does it use?
  • Which cloud provider is configured for the setup?
  • Other:

How to set Shoot cluster's Pod/Service CIDR when deploying via auto-installer

I am trying to install Gardener on existing Kubernetes Cluster (configured via KOPS) via automatic installer (https://gardener.cloud/installer/).

But I am facing an issue: "100.64.0.0/13": must not be a subset of "spec.networks.shootDefaults.services" ("100.64.0.0/13")

Tried with different subnet (100.80.0.0/11) too, then it gives error:
Error: release gardener failed: Service "gardener-apiserver" is invalid: spec.clusterIP: Invalid value: "100.80.0.20": provided IP is not in the valid range. The range of valid IPs is 100.64.0.0/13

As per the ticket gardener/gardener#1386 "Shoot pod/service CIDR defaults per seed", "It is now possible to specify shoot default pod/service CIDRs per seed. Please configure spec.networks.shootDefaults on your Seed resources if you want to use this."

But I am not able to understand how to configure the same when using the automatic installer (https://gardener.cloud/installer/).

Please help me to achieve a successful deployment of Gardner on existing Kubernetes Cluster (configured via Kops).

Introduce soil and seeds clusters management

What would you like to be added:

acre.yaml has to have an ability to create soil and underlying seeds clusters.

Why is this needed:

simplify the overall cascade gardener installation

Add retry logic

What would you like to be added:

Some operation should have retry logic so that temporarily unavailable endpoints like ak8s are retried instead of a directly exit.

sow deploy -A: errors on exec: exec: "aws-iam-authenticator": executable file not found in $PATH

I am trying to deploy gardener on AWS EKS. I have created Kubeconfig file that works well with kubectl and i get proper responses from EKS, but with gardener i am getting the following error message:

[root]# sow deploy tiller
*****************************
*** sowing species tiller
Plugin: kubectl[deploy] (/sow/plugins/kubectl/plugin)
manifest for ServiceAccount(v1): kube-system/tiller
error: unable to recognize "/mounted/root/landscape/gen/tiller/kubectl/manifests/0-0-ServiceAccount.kube-system.tiller.yaml": Get https://XYZ/api?timeout=32s: getting credentials: exec: exec: "aws-iam-authenticator": executable file not found in $PATH
command failed, waiting for 15 seconds and trying again (4 times)

I have that file in my PATH variable: /root/bin/aws-iam-authenticator
[root@ landscape]# echo $PATH /root/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/sow/docker/bin:/usr/local/bin

I am not sure what wrong am doing:

sow order -A command works fine for me

Please can somebody suggest what wrong am doing?

Regards,
Jay

Cannot delete the cluster with aws DNS provider

What happened:

DNS controller deletion hangs:

*** deleting species dns-controller
Plugin: kubectl[delete] (/sow/plugins/kubectl/plugin)
manifest for DNSProvider(dns.gardener.cloud/v1alpha1): kube-system/aws
manifest for Secret(v1): kube-system/garden-setup-dns-controller-manager-aws
cluster https://garden-root-k8s-c7a345cee89b457cb03178db0a9f7882.kubernikus.eu-nl-1.cloud.sap
  found deployed objects: 2
  found currect objects: 0
  found deployed obsolete objects: 2
    DNSProvider(dns.gardener.cloud/v1alpha1): kube-system/aws
dnsprovider.dns.gardener.cloud "aws" deleted

Controller logs:

kubectl -n extension-dns-external-zf4p2 logs seed-dns-controller-manager-5674b9796f-l5zgt
failed to create CRD dnsentries.dns.gardener.cloud: customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:extension-dns-external-zf4p2:seed-dns-controller-manager" cannot create resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope

Removing the finalizer helps:

kubectl -n kube-system edit dnsproviders aws

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.