Git Product home page Git Product logo

yawol's Introduction

yawol

Do OpenStack Load Balancing the Kubernetes Way.

yawol (yet another working OpenStack Load Balancer) is a Load Balancer solution for OpenStack, based on the Kubernetes controller pattern.

Key Features

  • Replacement for OpenStack Octavia Load Balancing
  • Provides Load Balancers for Kubernetes Services
  • Fully manages the instance lifecycle of Load Balancer VMs
  • Kubernetes-native approach: All the benefits of CRDs and controllers

How It Works

yawol uses kubebuilder as the controller framework and gophercloud for the OpenStack integration. The actual load balancing is done by Envoy.

For a more in-detail description, see the components documentation.

Installation

If this installation guide doesn't work for you, or if some instructions are unclear, please open an issue!

We provide a Helm chart for yawol in charts/yawol-controller that you can use for a quick installation on a Kubernetes cluster. In order to get yawol going, however, you need a yawol OpenStack VM image first.

yawol OpenStack Image

Create alpine base image

We use an openstack alpine base image which can be created with this packer file.

Preparation of packer environment

To create the necessary environment to build the image, you can use the terraform code located within hack/packer-infrastructure.

You can run this terraform code with the Earthly step +build-packer-environment. To be able to log in to OpenStack make sure you source your OpenStack Credentials. The following OpenStack ENV variables are needed to build the image: OS_AUTH_URL OS_PROJECT_ID OS_PROJECT_NAME OS_USER_DOMAIN_NAME OS_PASSWORD OS_USERNAME OS_REGION_NAME

To connect the OpenStack network with the internet, a floating IP is needed. You can specify the floating IP network with the Earthly argument FLOATING_NETWORK_NAME (default is floating-net).

earthly +build-packer-environment \
   --OS_AUTH_URL="$OS_AUTH_URL" \
   --OS_PROJECT_ID="$OS_PROJECT_ID" \
   --OS_PROJECT_NAME="$OS_PROJECT_NAME" \
   --OS_USER_DOMAIN_NAME="$OS_USER_DOMAIN_NAME" \
   --OS_PASSWORD="$OS_PASSWORD" \
   --OS_USERNAME="$OS_USERNAME" \
   --OS_REGION_NAME="$OS_REGION_NAME"
#  --FLOATING_NETWORK_NAME=floating-net

Note that the terraform state is locally in the hack/packer-infrastructure folder.

To clean up the resources the Earthly step +destroy-packer-environment can be used.

earthly +destroy-packer-environment \
   --OS_AUTH_URL="$OS_AUTH_URL" \
   --OS_PROJECT_ID="$OS_PROJECT_ID" \
   --OS_PROJECT_NAME="$OS_PROJECT_NAME" \
   --OS_USER_DOMAIN_NAME="$OS_USER_DOMAIN_NAME" \
   --OS_PASSWORD="$OS_PASSWORD" \
   --OS_USERNAME="$OS_USERNAME" \
   --OS_REGION_NAME="$OS_REGION_NAME"
#  --FLOATING_NETWORK_NAME=floating-net

Build yawol OpenStack Image

Before running our Earthly targets, set the needed environment variables:

export OS_NETWORK_ID=<from your openstack environment>
export OS_FLOATING_NETWORK_ID=<from your openstack environment>
export OS_SECURITY_GROUP_ID=<from your openstack environment>
export OS_SOURCE_IMAGE=<from your openstack environment>
export IMAGE_VISIBILITY=<private or public>

Like in the step above, to be able to log in to OpenStack make sure you source your OpenStack Credentials. To specify the machine flavor and volume type the Earthly arguments MACHINE_FLAVOR and VOLUME_TYPE can be used (default is MACHINE_FLAVOR=c1.2 and VOLUME_TYPE=storage_premium_perf6).

Then validate and build the image:

earthly +validate-yawollet-image
earthly +build-yawollet-image \
   --OS_NETWORK_ID="$OS_NETWORK_ID" \
   --OS_FLOATING_NETWORK_ID="$OS_FLOATING_NETWORK_ID" \
   --OS_SECURITY_GROUP_ID="$OS_SECURITY_GROUP_ID" \
   --OS_SOURCE_IMAGE="$OS_SOURCE_IMAGE" \
   --IMAGE_VISIBILITY="$IMAGE_VISIBILITY" \
   --OS_AUTH_URL="$OS_AUTH_URL" \
   --OS_PROJECT_ID="$OS_PROJECT_ID" \
   --OS_PROJECT_NAME="$OS_PROJECT_NAME" \
   --OS_USER_DOMAIN_NAME="$OS_USER_DOMAIN_NAME" \
   --OS_PASSWORD="$OS_PASSWORD" \
   --OS_USERNAME="$OS_USERNAME" \
   --OS_REGION_NAME="$OS_REGION_NAME"
#   --MACHINE_FLAVOR=c1.2
#   --VOLUME_TYPE=storage_premium_perf6

Cluster Installation

The in-cluster components of yawol (yawol-cloud-controller andyawol-controller) can now be installed.

  1. Optional: Install VerticalPodAutoscaler. If installed you can enable the VerticalPodAutoscaler resources in the helm values.
    1. VPA install guide
  2. Create a Kubernetes Secret that contains the contents of an .openrc file underneath the cloudprovider.conf key. The .openrc credentials need the correct permission to be able to create instances and request floating IPs.

Note: At most one of domain-id or domain-name and project-id or project-name must be provided.

apiVersion: v1
kind: Secret
metadata:
  name: cloud-provider-config
type: Opaque
stringData:
  cloudprovider.conf: |-
    [Global]
    auth-url="<OS_AUTH_URL>"
    domain-name="<OS_USER_DOMAIN_NAME>"
    domain-id="<OS_DOMAIN_ID>"
    # Deprecated (tenant-name): Please use project-name, only used if project-name is not set.
    tenant-name="<OS_PROJECT_NAME>"
    project-name="<OS_PROJECT_NAME>"
    project-id="<OS_PROJECT_ID>"
    username="<OS_USERNAME>"
    password="<OS_PASSWORD>"
    region="<OS_REGION_NAME>"

Assuming you saved the secret as secret-cloud-provider-config.yaml, apply it with:

kubectl apply -f secret-cloud-provider-config.yaml
  1. Configure the Helm values according to your OpenStack environment:

Values for the yawol-cloud-controller

# the name of the Kubernetes secret we created in the previous step
#
# Placed in LoadBalancer.spec.infrastructure.authSecretRef.name
yawolOSSecretName: cloud-provider-config

# floating IP ID of the IP pool that yawol uses to request IPs
#
# Placed in LoadBalancer.spec.infrastructure.floatingNetID
yawolFloatingID: <floating-id>

# OpenStack network ID in which the Load Balancer is placed
#
# Placed in LoadBalancer.spec.infrastructure.networkID
yawolNetworkID: <network-id>

# OpenStack subnet ID in which the Load Balancer is placed.
# If not set, the subnet is chosen automatically.
#
# Placed in LoadBalancer.spec.infrastructure.subnetID
yawolSubnetID: <subnet-id>

# default value for flavor that yawol Load Balancer instances should use
# can be overridden by annotation
#
# Placed in LoadBalancer.spec.infrastructure.flavor.flavor_id
yawolFlavorID: <flavor-id>

# default value for ID of the image used for the Load Balancer instance
# can be overridden by annotation
#
# Placed in LoadBalancer.spec.infrastructure.image.image_id
yawolImageID: <image-id>

# default value for the AZ used for the Load Balancer instance
# can be overridden by annotation. If not set, empty string is used.
#
# Placed in LoadBalancer.spec.infrastructure.availabilityZone
yawolAvailabilityZone: <availability-zone>

Values for the yawol-controller

# URL/IP of the Kubernetes API server that contains the LoadBalancer resources
yawolAPIHost: <api-host>

To check out all available values have a look into the Helm values

  1. With the values correctly configured, you can now install the Helm chart.
helm install yawol ./charts/yawol-controller

This will also install the CRDs needed by yawol.

After successful installation, you can request Services of type: LoadBalancer and yawol will take care of creating an instance, allocating an IP, and updating the Service resource once the setup is ready.

You can also specify custom annotations on the Service to further control the behavior of yawol.

apiVersion: v1
kind: Service
metadata:
  name: loadbalancer
  annotations:
    # Override the default  OpenStack image ID.
    yawol.stackit.cloud/imageId: "OS-imageId"
    # Override the default OpenStack machine flavor.
    yawol.stackit.cloud/flavorId: "OS-flavorId"
    # Overwrites the default openstack network for the loadbalancer.
    # If this is set to a different network ID than defined as default in the yawol-cloud-controller
    # the default from the yawol-cloud-controller will be added to the additionalNetworks.
    yawol.stackit.cloud/defaultNetworkID: "OS-networkID"
    # If set to true it do not add the default network ID from
    # the yawol-cloud-controller to the additionalNetworks.
    yawol.stackit.cloud/skipCloudControllerDefaultNetworkID: "false"
    # Overwrites the projectID which is set by the secret.
    # If not set the settings from the secret binding will be used.
    # This field is immutable and can not be changed after the service is created.
    yawol.stackit.cloud/projectID: "OS-ProjectID"
    # Overwrites the openstack floating network for the loadbalancer.
    yawol.stackit.cloud/floatingNetworkID: "OS-floatingNetID"
    # Override the default OpenStack availability zone.
    yawol.stackit.cloud/availabilityZone: "OS-AZ"
    # Specify if this should be an internal LoadBalancer .
    yawol.stackit.cloud/internalLB: "false"
    # Run yawollet in debug mode.
    yawol.stackit.cloud/debug: "false"
    # Reference the name of the SSH key provided to OpenStack for debugging .
    yawol.stackit.cloud/debugsshkey: "OS-keyName"
    # Allows filtering services in cloud-controller.
    # Deprecated: Use service.spec.loadBalancerClass instead.
    yawol.stackit.cloud/className: "test"
    # Specify the number of LoadBalancer machines to deploy (default 1).
    yawol.stackit.cloud/replicas: "3"
    # Specify an existing floating IP for yawol to use.
    yawol.stackit.cloud/existingFloatingIP: "193.148.175.46"
    # Specify the loadBalancerSourceRanges for the LoadBalancer like service.spec.loadBalancerSourceRanges (comma separated list).
    # If service.spec.loadBalancerSourceRanges is set this annotation will NOT be used.
    yawol.stackit.cloud/loadBalancerSourceRanges: "10.10.10.0/24,10.10.20.0/24"
    # Enable/disable envoy support for proxy protocol.
    yawol.stackit.cloud/tcpProxyProtocol: "false"
    # Defines proxy protocol ports (comma separated list).
    yawol.stackit.cloud/tcpProxyProtocolPortsFilter: "80,443"
    # Enables log forwarding.
    yawol.stackit.cloud/logForward: "true"
    # Defines loki URL for the log forwarding.
    yawol.stackit.cloud/logForwardLokiURL: "http://example.com:3100/loki/api/v1/push"
    # Defines labels that are added when forwarding logs
    # The prefix "logging.yawol.stackit.cloud/" will be trimmed
    # and only "foo": "bar" will be added as a label
    logging.yawol.stackit.cloud/foo: "bar"
    # Setting multiple labels is also supported.
    logging.yawol.stackit.cloud/env: "testing"
    # Defines the TCP idle Timeout as duration, default is 1h.
    # Make sure there is a valid unit (like "s", "m", "h"), otherwise this option is ignored.
    yawol.stackit.cloud/tcpIdleTimeout: "5m30s"
    # Defines the UDP idle Timeout as duration, default is 1m.
    # Make sure there is a valid unit (like "s", "m", "h"), otherwise this option is ignored.
    yawol.stackit.cloud/udpIdleTimeout: "5m"
    # Defines the openstack server group policy for a LoadBalancer.
    # Can be 'affinity', 'anti-affinity' 'soft-affinity', 'soft-anti-affinity' depending on the OpenStack Infrastructure.
    # If not set openstack server group is disabled.
    yawol.stackit.cloud/serverGroupPolicy: anti-affinity
    # Defines additional openstack networks for the loadbalancer (comma separated list).
    yawol.stackit.cloud/additionalNetworks: "OS-networkID1,OS-networkID2"

To create a first LoadBalancer you can create a nginx deployment with a Service of type LoadBalancer:

kubectl create deploy --image nginx --port 80 nginx
kubectl expose deployment nginx --port 80 --type LoadBalancer

Development

See the development guide.

yawol's People

Contributors

breuerfelix avatar dependabot[bot] avatar dergeberl avatar einfachnuralex avatar gfelbing avatar hikhvar avatar kumm-kai avatar maboehm avatar malt3 avatar michaeleischer avatar nschad avatar renovate[bot] avatar robinschneider avatar simonkienzler avatar timebertt avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar

Forkers

hikhvar timebertt malt3 gfelbing berendt

yawol's Issues

Removing all gardener-related values

We should remove all gardener related settings/values in the helm chart.

Valid point. In this case, I suggest closing this PR and removing all gardener-related values in a new PR.
The gardener extension should be responsible for injecting all gardener-related values in its controller.
Originally posted by @timebertt in #87 (comment)

  • #89
  • drop gardener monitoring config in this repository and move it to extension

target-controller: service-controller: possible bug in deletion logic

According to

yawol/controllers/yawol-cloud-controller/targetcontroller/service_controller.go

Line 71 in 80f74e6

// only trigger deletion routine if the lb still exists

the deletion routine for services of different classes should only be triggered if the loadbalancer object still exists.
As there is a comment, I think there was a conscious decision towards it.

If the deletion shouldn't be triggered, then why is the deletion routine still triggered for other services, e.g. with different types?

yawol/controllers/yawol-cloud-controller/targetcontroller/service_controller.go

Line 78 in 80f74e6

return r.deletionRoutine(ctx, svc, infraDefaults)

For the example above, it would even break a test if I add the check for lb existence, as it doesn't remove the finalizer of the service (which is valid).

IMHO the check for lb existence when reconciling deletions of services with different classes is obselete / is a bug in the case when there is no lb object (anymore) but still a finalizer left.

Openstack Secret handling

We use tenant-name in the OpenStack Secret and use it in the background as project-name. tenant-name is not used anymore in OpenStack.

We should also add the possibility to use project-name, project-id, domain-id.

yawol/internal/openstack/client.go

Lines 222 to 227 in 422e1d2

authURL := strings.TrimSpace(cfg.Section("Global").Key("auth-url").String())
authInfo.AuthURL = authURL
authInfo.Username = strings.TrimSpace(cfg.Section("Global").Key("username").String())
authInfo.Password = strings.TrimSpace(cfg.Section("Global").Key("password").String())
authInfo.DomainName = strings.TrimSpace(cfg.Section("Global").Key("domain-name").String())
authInfo.ProjectName = strings.TrimSpace(cfg.Section("Global").Key("tenant-name").String())

Improve documentation

Issue to track documentation topics from #96

  • hack/packer-infrastructure/ - openstack specific settings (floating-net, dns-nameservers) #113
  • yawollet image build - openstack specific settings (flavor name, storage-class) #113
  • link to VPA docs #114
  • more details about the openstack secret #114
  • add example LoadBalance to try it out after installation, that work without yawol.stackit.cloud/className: "test" #114

Installation not working

I tried to follow the installation as described in the README and failed at several points.

Image Build

First I failed at the terraform apply for the build infrastructure and hat to adopt the name of the floating ip network:

diff --git a/hack/packer-infrastructure/variables.tf b/hack/packer-infrastructure/variables.tf
index 1ea4f4c..c366e21 100644
--- a/hack/packer-infrastructure/variables.tf
+++ b/hack/packer-infrastructure/variables.tf
@@ -1,4 +1,4 @@
 variable "floating_ip_network_name" {
-  default = "floating-net"
+  default = "extnet"
   description = "Name of the network your floating IPs are hosted in"
 }

And set a DNS server for the created network:

diff --git a/hack/packer-infrastructure/network.tf b/hack/packer-infrastructure/network.tf
index 3fa8859..b90df5e 100644
--- a/hack/packer-infrastructure/network.tf
+++ b/hack/packer-infrastructure/network.tf
@@ -8,6 +8,7 @@ resource "openstack_networking_subnet_v2" "packer" {
   network_id = openstack_networking_network_v2.packer.id
   cidr       = "192.168.48.0/24"
   ip_version = 4
+  dns_nameservers = [ "8.8.8.8" ]
 }

Next fail was at the earthy build. To make that work I had to adopt the flavor type and volume type:

diff --git a/image/alpine-yawol.pkr.hcl b/image/alpine-yawol.pkr.hcl
index 13531da..b231088 100644
--- a/image/alpine-yawol.pkr.hcl
+++ b/image/alpine-yawol.pkr.hcl
@@ -35,7 +35,7 @@ variable "security_group_id" {

 variable "machine_flavor" {
   type        = string
-  default     = "c1.2"
+  default     = "my_tiny_flavor"
   description = "The ID, name, or full URL for the desired flavor for the server to be created."
 }

@@ -65,7 +65,7 @@ source "openstack" "yawollet" {
   ssh_username            = "alpine"
   use_blockstorage_volume = true
   volume_size             = 1
-  volume_type             = "storage_premium_perf6"
+  volume_type             = "__DEFAULT__"
   ssh_timeout             = "10m"
   image_tags              = var.image_tags
 }

After that I got the yawol-alpine-v0.11.0-1 image.

Cluster Installation

At the cluster installation it would be nice to have a link how to install VerticalPodAutoscaler (which is not that hard to find, but would make life easier 😸 ).

I had to create the Secret in namespace kube-system otherwise the controller did not find it.

In the Secret there is some tenant-name which is no (more) used in OpenStack. So it's not clear was needs to be provided here.

The images referenced in the helm chart do not exist, I adopted the values.yaml to make that work:

diff --git a/charts/yawol-controller/values.yaml b/charts/yawol-controller/values.yaml
index acc96a9..a423ff8 100644
--- a/charts/yawol-controller/values.yaml
+++ b/charts/yawol-controller/values.yaml
@@ -17,13 +17,13 @@ yawolCloudController:
   clusterRoleEnabled: true
   image:
     repository: ghcr.io/stackitcloud/yawol/yawol-cloud-controller
-    tag: latest
+    tag: v0.11.0

 yawolController:
   gardenerMonitoringEnabled: false
   image:
     repository: ghcr.io/stackitcloud/yawol/yawol-controller
-    tag: latest
+    tag: v0.11.0

The provided example service is not working out-of-the-box, because of the set yawol.stackit.cloud/className: "test". After commenting this out, the service gets handled by the yawol controller.

Finally I have to give up with this error:

1.6742993640043612e+09	ERROR	Reconciler error	{"controller": "loadbalancer", "controllerGroup": "yawol.stackit.cloud", "controllerKind": "LoadBalancer", "loadBalancer": {"name":"yawol-test--lb-test","namespace":"kube-system"}, "namespace": "kube-system", "name": "yawol-test--lb-test", "reconcileID": "bd3a2b22-c3ff-48f4-bc0f-a7d897195d18", "error": "You must provide exactly one of DomainID or DomainName to authenticate by Username"}

Any hints are welcome as I would like to see it running 👾

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

dockerfile
Dockerfile
  • docker/dockerfile 1.7-labs
  • golang 1.22
github-actions
.github/workflows/helmchart.yaml
  • actions/checkout v4
  • azure/setup-helm v4
  • helm/chart-releaser-action v1.6.0
.github/workflows/images.yaml
  • actions/checkout v4
.github/workflows/tests.yaml
  • actions/checkout v4
gomod
go.mod
  • go 1.22.0
  • go 1.22.3
  • github.com/cncf/xds/go v0.0.0-20240423153145-555b57ec207b@555b57ec207b
  • github.com/envoyproxy/go-control-plane v0.12.0
  • github.com/go-logr/logr v1.4.1
  • github.com/golang/protobuf v1.5.4
  • github.com/gophercloud/gophercloud v1.11.0
  • github.com/onsi/ginkgo/v2 v2.17.3
  • github.com/onsi/gomega v1.33.1
  • github.com/prometheus/client_golang v1.19.1
  • github.com/shirou/gopsutil/v3 v3.24.4
  • go.uber.org/zap v1.27.0
  • golang.org/x/time v0.5.0
  • google.golang.org/grpc v1.64.0
  • google.golang.org/protobuf v1.34.1
  • gopkg.in/ini.v1 v1.67.0
  • gopkg.in/yaml.v3 v3.0.1
  • k8s.io/api v0.30.0
  • k8s.io/apimachinery v0.30.0
  • k8s.io/client-go v0.30.0
  • k8s.io/utils v0.0.0-20240502163921-fe8a2dddb1d0@fe8a2dddb1d0
  • sigs.k8s.io/controller-runtime v0.18.2
  • sigs.k8s.io/controller-tools v0.15.0
helm-values
charts/yawol-controller/values.yaml
terraform
hack/packer-infrastructure/main.tf
  • openstack >= 1.47
  • template 2.2.0
  • hashicorp/terraform >= 1.1.7
  • Check this box to trigger a request for Renovate to run again on this repository

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.