how to deploy a Avi Controller. without a vCenter cloud

Describe the bug

Hey guys,
when I try to follow the steps here: https://avinetworks.github.io/avi-helm-charts/docs/AKO/install/helm.html

then I can't reach the helm repo.
In your markdown files in the code base you are referencing to oci://projects.registry.vmware.com/ako/helm-charts/ako, that link seems to work.
Working doc:
https://github.com/vmware/load-balancer-and-ingress-services-for-kubernetes/blob/master/docs/install/helm.md

Reproduction steps

Follow the steps from the github page. The helm install doesn't work

Expected behavior

A fixed documentation

Additional context

No response

Problem Description

Kubernetes has been deprecating API(s), which will be removed and are no longer available in 1.22. Operators projects using these APIs versions will not work on Kubernetes 1.22 or any cluster vendor using this Kubernetes version(1.22), such as OpenShift 4.9+. Following the APIs that are most likely your projects to be affected by:

• apiextensions.k8s.io/v1beta1: (Used for CRDs and available since v1.16)
• rbac.authorization.k8s.io/v1beta1: (Used for RBAC/rules and available since v1.8)
• admissionregistration.k8s.io/v1beta1 (Used for Webhooks and available since v1.16)

Therefore, looks like this project distributes solutions via the Red Hat Connect with the package name as ako-operator and does not contain any version compatible with k8s 1.22/OCP 4.9. Following some findings by checking the distributions published:

• ako-operator.v1.4.2: this distribution is using APIs which were deprecated and removed in v1.22. More info: https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-22. Migrate the API(s) for CRD: (["akoconfigs.ako.vmware.com"]),ClusterRole: (["ako-operator-metrics-reader"]),

NOTE: The above findings are only about the manifests shipped inside of the distribution. It is not checking the codebase.

How to solve

It would be very nice to see new distributions of this project that are no longer using these APIs and so they can work on Kubernetes 1.22 and newer and published in the Red Hat Connect collection. OpenShift 4.9, for example, will not ship operators anymore that do still use v1beta1 extension APIs.

Due to the number of options available to build Operators, it is hard to provide direct guidance on updating your operator to support Kubernetes 1.22. Recent versions of the OperatorSDK greater than 1.0.0 and Kubebuilder greater than 3.0.0 scaffold your project with the latest versions of these APIs (all that is generated by tools only). See the guides to upgrade your projects with OperatorSDK Golang, Ansible, Helm or the Kubebuilder one. For APIs other than the ones mentioned above, you will have to check your code for usage of removed API versions and upgrade to newer APIs. The details of this depend on your codebase.

If this projects only need to migrate the API for CRDs and it was built with OperatorSDK versions lower than 1.0.0 then, you maybe able to solve it with an OperatorSDK version >= v0.18.x < 1.0.0:

$ operator-sdk generate crds --crd-version=v1
INFO[0000] Running CRD generator.
INFO[0000] CRD generation complete.

Alternatively, you can try to upgrade your manifests with controller-gen (version >= v0.4.1) :

If this project does not use Webhooks:

$ controller-gen crd:trivialVersions=true,preserveUnknownFields=false rbac:roleName=manager-role paths="./..."

If this project is using Webhooks:

1. Add the markers sideEffects and admissionReviewVersions to your webhook (Example with sideEffects=None and admissionReviewVersions={v1,v1beta1}: memcached-operator/api/v1alpha1/memcached_webhook.go):

2. Run the command:

$ controller-gen crd:trivialVersions=true,preserveUnknownFields=false rbac:roleName=manager-role webhook paths="./..."

For further info and tips see the blog.

Thank you for your attention.

Currently, AKO can be deployed only in a particular namespace like avi-system, vmware-system.

Ideally, AKO should support deployed in a namespace that the user can specify.

Is your feature request related to a problem? Please describe.

Currently, to use a health monitor in AKO, it must be pre-created in the Avi controller. This poses a hindrance to automation.

Describe the solution you'd like

A way to create health monitors via AKO would alleviate the need for external custom automation, and allow me to keep more automation in one place.

A separate CR specifically for the creation of custom healthmonitors would greatly assist in providing users the ability to "keep it in Kubernetes".

Describe alternatives you've considered

Currently, to circumvent this issue, and to provide my platform users good experience, I relegated the creation/deletion of the healthmonitor to a controlscript that fires on a pool CONFIG_CREATE/CONFIG_DELETE event.

While this works, it doesn't allow for much customization, so my users are forced to use a single, pre-defined path for their healthchecks

Additional context

No response

Describe the bug

I am trying to follow the install guide for AKO with helm and receiving an error with version 1.10.3 not found.

Reproduction steps

Following the install guide:

helm show chart oci://projects.registry.vmware.com/ako/helm-charts/ako --version 1.10.3
Error: projects.registry.vmware.com/ako/helm-charts/ako:1.10.3: not found

Expected behavior

Helm release 1.10.3 chart output.

Additional context

No response

Describe the bug

When trying to deploy AKO v1.7.1 (also tried with v1.6.1) logs show the following error:

2022-06-24T11:57:45.163Z	INFO	lib/lib.go:287	Setting AKOUser: ako-tkg-prods-cluster for Avi Objects
2022-06-24T11:57:45.387Z	INFO	cache/controller_obj_cache.go:3182	Setting VRF global found from network Tanzu-Workload
2022-06-24T11:57:45.387Z	WARN	cache/controller_obj_cache.go:2575	Invalid input detected, AKO will be rebooted to retry Labels does not match with cluster name for SE group :Default-Group. Expected Labels: [{"key":"clustername","value":"tkg-prods-cluster"}]
2022-06-24T11:57:45.387Z	INFO	api/api.go:68	Shutting down the API server
2022-06-24T11:57:45.387Z	INFO	api/api.go:113	API server shutdown: http: Server closed
2022-06-24T11:57:45.387Z	INFO	record/event.go:282	Event(v1.ObjectReference{Kind:"Pod", Namespace:"avi-system", Name:"ako-0", UID:"ffa3379f-0f5a-48b8-bd65-fb7c26e198e4", APIVersion:"v1", ResourceVersion:"53795", FieldPath:""}): type: 'Warning' reason: 'AKOShutdown' Invalid user input [Labels does not match with cluster name for SE group :Default-Group. Expected Labels: [{"key":"clustername","value":"tkg-prods-cluster"}]]
2022-06-24T11:57:45.388Z	ERROR	k8s/ako_init.go:292	Error while validating input: Labels does not match with cluster name for SE group :Default-Group. Expected Labels: [{"key":"clustername","value":"tkg-prods-cluster"}]
2022-06-24T11:57:45.388Z	ERROR	ako-main/main.go:271	Handle configmap error during reboot, shutting down AKO. Error is: sync is disabled because of configmap unavailability during bootup

I've tried numerous configurations but the issue is always the same. We are using AVI LB 21.1.4 with only one SE group (Default-Group).

Reproduction steps

1. Follow configuration for AKO according to AVI Advanced LB documentation
2. Install helm chart and deploy using values.yaml
3. Check the logs for the ako-0 container

Here are the values for the values.yaml file:

AKOSettings:
  clusterName: tkg-prods-cluster # A unique identifier for the kubernetes cluster, that helps distinguish the objects for this cluster in the avi controller. // MUST-EDIT
  cniPlugin: 'antrea' # I'm using Antrea as CNI Plugin in my TKGS environment
  layer7Only: true # If this flag is switched on, then AKO will only do layer 7 loadbalancing. The Ako on the Supervisor Control Plane is still responsible for L4 loadbalancing.
NetworkSettings:
  ## This list of network and cidrs are used in pool placement network for vcenter cloud.
  ## Node Network details are not needed when in nodeport mode / static routes are disabled / non vcenter clouds.
  nodeNetworkList:
    - networkName: "Tanzu-Front"
      cidrs:
        - 10.30.40.0/23
  vipNetworkList:  # Network information of the VIP network. Multiple networks allowed only for AWS Cloud.
    - networkName: "Tanzu-Workload"
      cidr: 10.100.8.0/21

### This section outlines settings on the Avi controller that affects AKO's functionality.
ControllerSettings:
  serviceEngineGroupName: Default-Group   # Name of the ServiceEngine Group.
  controllerVersion: '21.1.4' # The controller API version
  cloudName: Default-Cloud   # The configured cloud name on the Avi controller.
  controllerHost: '10.20.30.130' # IP address or Hostname of Avi Controller
  tenantName: admin   # Name of the tenant where all the AKO objects will be created in AVI.
avicredentials:
  username: '<AVI-USERNAME>'
  password: '<AVI-PASSWORD>'

Expected behavior

We should be able to get connected to the AVI LB with AKO

Additional context

We are using vSphere with Tanzu clusters:

NAME                                                  STATUS   ROLES                  AGE     VERSION            INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                 KERNEL-VERSION   CONTAINER-RUNTIME
tkg-prods-cluster-control-plane-4zglp                 Ready    control-plane,master   4h19m   v1.22.9+vmware.1   10.100.9.16   <none>        VMware Photon OS/Linux   4.19.225-3.ph3   containerd://1.5.11
tkg-prods-cluster-control-plane-8cstq                 Ready    control-plane,master   4h15m   v1.22.9+vmware.1   10.100.9.17   <none>        VMware Photon OS/Linux   4.19.225-3.ph3   containerd://1.5.11
tkg-prods-cluster-control-plane-txrr9                 Ready    control-plane,master   4h24m   v1.22.9+vmware.1   10.100.9.7    <none>        VMware Photon OS/Linux   4.19.225-3.ph3   containerd://1.5.11
tkg-prods-cluster-nodepool-b1-ccvwn-56d5dcfff-bxn4q   Ready    <none>                 4h20m   v1.22.9+vmware.1   10.100.9.10   <none>        VMware Photon OS/Linux   4.19.225-3.ph3   containerd://1.5.11
tkg-prods-cluster-nodepool-b1-ccvwn-56d5dcfff-s9nmg   Ready    <none>                 4h20m   v1.22.9+vmware.1   10.100.9.15   <none>        VMware Photon OS/Linux   4.19.225-3.ph3   containerd://1.5.11
tkg-prods-cluster-nodepool-b1-ccvwn-56d5dcfff-tvbtd   Ready    <none>                 4h20m   v1.22.9+vmware.1   10.100.9.12   <none>        VMware Photon OS/Linux   4.19.225-3.ph3   containerd://1.5.11

It's a common request from customers. For now, assigning a static IP in service.spec.loadBalancerIP is a noop.

In the AKO chart the image tag is hard coded:

image: "{{ .Values.image.repository }}:{{ .Chart.AppVersion }}"

We rebuild all of our open source images in our own repository and apply a build date suffix. We can't use the chart as it is without modifying it.

Standard practice is to allow the image tag to be overridden and have the image tag supplied in the default values.yaml file. Like this:

image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"

Currently, when upgrading AKO to a newer version, all the new features are on by default.

AKO should provide a flag or feature gate to control if newly released features should be enabled by users.

Is your feature request related to a problem? Please describe.

With the changes made for compatibility between Avi Essential and external-dns it's not possible to use DNS with L4 unless autoFQDN is enabled.

DNS on L4 and automatic FQDN shouldn't be configured together. There are many use cases for DNS on L4 without enabling autoFQDN on every service.

Describe the solution you'd like

Create a method for users to create a DNS entry for L4 services without automatically creating it for every service. This could be another option for autoFQDN, or using a different annotation to external-dns.alpha.kubernetes.io/hostname.

Describe alternatives you've considered

No response

Additional context

No response

Describe the bug

Ako pod is unable to read ingressclasses in v1.6.1 resulting in repeating:

E0106 07:29:13.478723 1 reflector.go:138] github.com/vmware/load-balancer-and-ingress-services-for-kubernetes/internal/k8s/controller.go:1087: Failed to watch *v1.IngressClass: failed to list *v1.IngressClass: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:avi-system:ako-sa" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope

Reproduction steps

1. helm install ako v 1.6.1
2. get logs of ako pod


Adding ingressclass manualy in the clusterrole fixes the error and allows ako to continue.

Expected behavior

That the ako-sa serviceaccount are able to read ingressclasses using the provided clusterrole.

Additional context

Going back to 1.5.1 works.

Is your feature request related to a problem? Please describe.

svc.spec.loadBalancerIP has been deprecated in k8s 1.24: kubernetes/kubernetes#107235. This may lead to problems in the future.

Describe the solution you'd like

I'd like to see this project add an annotation to assign a Load Balancer IPs. Please see metallb/metallb#1174 for inspiration.

Describe alternatives you've considered

No response

Additional context

No response

When cluster has no LB SVC workload, setting deleteConfig to True will leave the StatefulSet status stuck in:

status:
  collisionCount: 0
  conditions:
  - lastTransitionTime: "2021-01-12T22:16:45Z"
    message: Started deleting objects
    reason: Started
    status: "True"
    type: ako.vmware.com/ObjectDeletionInProgress

which eventually become

status:
  collisionCount: 0
  conditions:
  - lastTransitionTime: "2021-01-12T20:36:57Z"
    message: Error, timed out while deleting objects
    reason: Timeout
    status: Unknown
    type: ako.vmware.com/ObjectDeletionInProgress

Describe the bug

We noticed the installation of the CRDs for gatewayclass.networking.x-k8s.io are installed on the cluster. These resources don't seem to be used/referenced by AKO anymore and they are also outdated.

Reproduction steps

1. Install AKO
2. Run kubectl get crds | grep networking.x-k8s.io

Expected behavior

kubectl get crds | grep networking.x-k8s.io to not show any resources

Additional context

No response

There's a case where client is send requests with custom headers. If host header contains port information. Avi will fail to redirect traffic to backend.

e.g.
Client is sending a request with host header "www.test.com:443"

Other open source solutions like Nginx does not have this issue. Nginx uses server name(i.e. FQDN) to proxy traffic to endpoint and it doesn't care which port Host header contains.

Original Datascript ako is using:

host = avi.http.get_host_tokens("MODIFIED", 1)
path = avi.http.get_path_tokens(1)
if host and path then
lbl = host.."/"..path
else
lbl = host.."/"
end
avi.poolgroup.select("ako-ncp2-cluster--Shared-L7-0", string.lower(lbl) )

Fix:

host = avi.http.get_host_tokens("MODIFIED", 1)
if string.contains(host,":") then
      for match in string.gmatch(host, ".*:") do
        host = string.sub(match,0,-2)
      end
end
path = avi.http.get_path_tokens(1)
if host and path then
lbl = host.."/"..path
else
lbl = host.."/"
end
avi.poolgroup.select("ako-ncp2-cluster--Shared-L7-0", string.lower(lbl) )

Datascript reference:
https://github.com/avinetworks/datascript-library/blob/master/availability/location_header_rewrite_with_nonstandard_port.md

I have noticed that CTRL_USERNAME and CTRL_PASSWORD variables got removed from 1.5+ AKO statefulset template.

After talking with @sudswasavi , it looks like AKO will fetch those values directly from avi-secret now, could we document this somewhere?
I think other users may also be confused about this.

Thanks!

Describe the bug

Hi they are any solution to passe the subnetuuid to ako ?

i try to create a simple l4 lb service in k8s with ako on oci/linux server cloud and the networkref is never given in call api of controller

ako can work with linux server cloud ?

Reproduction steps

install ako and config to cloud with linux server cloud

Expected behavior

create vip

Additional context

ako 1.10.3 / k8s 1.25 / avi controller 21.1.3 / Oracle cloud infrastructure

Summary:
When a LB service without a selector is created and a few minutes later, the corresponding Endpoint object is created, AKO 1.4.1 doesn't map the service to the endpoint.

Detail:
When a LB service without a selector is created and a few minutes later, the corresponding Endpoint object is created, msg: error while retrieving endpoints: endpoints "ha-lb" not found still shows up in the log of pod "ako-0". AKO needs a restart to map the service to the endpoint.

Describe the bug

I got We found potential security vulnerabilities in your dependencies alert when I import github.com/vmware/load-balancer-and-ingress-services-for-kubernetes in my go project.

Reproduction steps

run `go get github.com/vmware/load-balancer-and-ingress-services-for-kubernetes` in your repo.

Expected behavior

No security vulnerabilities after run go get github.com/vmware/load-balancer-and-ingress-services-for-kubernetes

Additional context

No response

Describe the bug

The helm chart repo returns 404.

# helm repo update ako
Hang tight while we grab the latest from your chart repositories...
...Unable to get an update from the "ako" chart repository (https://projects.registry.vmware.com/chartrepo/ako):
        failed to fetch https://projects.registry.vmware.com/chartrepo/ako/index.yaml : 404 Not Found
Update Complete. ⎈Happy Helming!⎈
```

### Reproduction steps

1. helm repo update ako


### Expected behavior

The repo should return the helm chart.

### Additional context

_No response_

I have a TKG environment running on a vCenter and AVI controller and SE running on a different vCenter environment. When I configure an LB using NodePort the IP addresses used in the pool are the internal (non routable) IP addresses. E.g. the non-routable IPs in the screenshot below get used to populate the pool addresses. Note that a TKG deployment always uses the same ip for internal and external. When I tun the TKG on the same vCenter as the AVI environment the correct IP gets used.

Is this scenario using two different vCenters supported and if so how do the correct IP addresses get populated in the AKO operator?

AKO should reconcile AVIInfraSettings conditionally. Currently it's crashing if AVIInfraSettings CRD is not created beforehand.

Describe the bug

AKO 1.8.2 logs mention is supports up till AVI Controller version 22.1.2. However, AKO compatibility guide suggests it is compatible till 22.1.4
Verified AKO 1.8.2 working fine with our AVI 22.1.3 version also. therefore reported message in log is incorrect.

Please find attached ako pod log file and support bundle

As per AVI Ref Link - https://avinetworks.com/docs/ako/1.8/ako-compatibility-guide/

AKO Release(s) Avi Controller Versions Supported
============== ==================================
1.8 21.1.3 - 22.1.4 (AKO 1.8.2)

ako pod logs details (kubectl logs ako-0 -n avi-system):

2023-07-15T18:11:09.859Z INFO ako-main/main.go:71 AKO is running with version: v1.8.2
:
2023-07-15T18:11:10.504Z INFO lib/lib.go:1730 Setting the client version to AVI Max supported version 22.1.2
2023-07-15T18:11:10.504Z INFO cache/avi_ctrl_clients.go:71 Setting the client version to 22.1.2

Following issue observed:

2023-07-15T18:11:09.859Z INFO api/api.go:52 Setting route for GET /api/status
2023-07-15T18:11:09.859Z INFO ako-main/main.go:71 AKO is running with version: v1.8.2
2023-07-15T18:11:09.859Z INFO ako-main/main.go:81 We are running inside kubernetes cluster. Won't use kubeconfig
files.
2023-07-15T18:11:09.859Z WARN lib/lib.go:710 Unable to fetch Blocked namespaces from environment variables. unexpec
ted end of JSON input
2023-07-15T18:11:09.860Z INFO api/api.go:110 Starting API server at :8080
2023-07-15T18:11:09.870Z INFO lib/control_config.go:198 ako.vmware.com/v1alpha1/AviInfraSetting enabled on clu
ster
2023-07-15T18:11:09.903Z INFO lib/control_config.go:207 ako.vmware.com/v1alpha1/HostRule enabled on cluster
2023-07-15T18:11:09.934Z INFO lib/control_config.go:216 ako.vmware.com/v1alpha1/HTTPRule enabled on cluster
2023-07-15T18:11:09.946Z INFO ako-main/main.go:150 Kubernetes cluster apiserver version 1.24
2023-07-15T18:11:09.953Z INFO utils/utils.go:168 Initializing configmap informer in avi-system
2023-07-15T18:11:09.953Z INFO lib/dynamic_client.go:118 Skipped initializing dynamic informers antrea
2023-07-15T18:11:10.102Z INFO k8s/ako_init.go:455 Successfully connected to AVI controller using existing AKO se
cret
2023-07-15T18:11:10.102Z INFO ako-main/main.go:261 Valid Avi Secret found, continuing ..
2023-07-15T18:11:10.504Z INFO lib/lib.go:1730 Setting the client version to AVI Max supported version 22.1.2
2023-07-15T18:11:10.504Z INFO cache/avi_ctrl_clients.go:71 Setting the client version to 22.1.2
2023-07-15T18:11:10.504Z INFO ako-main/main.go:279 SEgroup name found, continuing ..
2023-07-15T18:11:11.650Z INFO cache/controller_obj_cache.go:2340 Avi cluster state is CLUSTER_UP_NO_HA
2023-07-15T18:11:11.785Z INFO cache/controller_obj_cache.go:2901 Setting cloud vType: CLOUD_VCENTER
2023-07-15T18:11:11.785Z INFO cache/controller_obj_cache.go:2904 Setting cloud uuid: cloud-ba12c31e-dc83-4a73-9

Reproduction steps

1. Check ako-o pod logs using kubectl logs command
   kubectl logs ako-0 -n avi-system
2. Observer message : Setting the client version to AVI Max supported version 22.1.2 ( But actually ako 1.8.2 supported version is 22.1.4)

Expected behavior

If the CTRL_VERSION (controllerVersion) is set a large version than the AviMaxSupportedVersion such as 22.1.4, it will get the 22.1.2 version

if CompareVersions(controllerVersion, ">", GetAviMaxSupportedVersion()) {
		utils.AviLog.Infof("Setting the client version to AVI Max supported version %s", GetAviMaxSupportedVersion())
		controllerVersion = GetAviMaxSupportedVersion()
	}

Additional context

No response

Summary:
When adding or removing node in Endpoints object, ako doesn’t update backend server correspondingly. Since in control plane HA use case, the control plane node would be added and removed from cluster dynamically, it is critical that AKO should watch Endpoints object and update corresponding Virtual Service backend Server.

Detail:
After a LB service without a selector and the corresponding Endpoint object (which contains one node) are created, "Servers(up/total)" is "1/1" for this service on Avi Controller UI.

Then, we update the Endpoints object and add a new node in subnets, but “Servers(up/total)” is still “1/1” , which is expected to be “2/2".

Then, we update the Endpoints object and remove the node added to the Endpoint object, but "Servers(up/total)" is still "1/1", which is expected to be "0/0".

Is your feature request related to a problem? Please describe.

I am unable to use an existing secret generated by a Secrets Manager such as the external-secret operator for the AKO avi-secret.

Describe the solution you'd like

The ako Helm Chart currently captures avicredentials properties and uses them to generate a Secret named avi-secret. In some environments, it is preferred to allow definition of an existing secret rather than the generation of a new secret.

In order to support this case, consider introducing a new Helm property entitled avicredentials.existingSecret. See PR https://github.com/grafana/helm-charts/pull/542/files for a similar feature that was implemented in the Grafana Loki Helm Chart.

Describe alternatives you've considered

No response

Additional context

No response

Is your feature request related to a problem? Please describe.

I want to limit access to an ingress by source ip (like with this nginx annotation nginx.ingress.kubernetes.io/whitelist-source-range). If i understand it correctly this can be managed by network security rules.
But with the current CRDs i can only assign defined network security rules.

It would be nice if there is a way to define the rules inside the kubernetes cluster.

Describe the solution you'd like

Provide a way to define network security rules inside a kubernetes cluster.
One way is to define a new CRD to create the rules.

Describe alternatives you've considered

No response

Additional context

No response

Is your feature request related to a problem? Please describe.

When I provide a hostname that isn't in the DNS profile as a subdomain, I would still like AKO to spin up a VIP even if it can't provision the DNS name via the profile. Our DNS profile is integrated with infoblox.

Example:
Requested hostname via ingress: myname.mydomain
Subdomains configured: .tkg.mydomain

Error:
WARN nodes/validator.go:58 Didn't find match for hostname :myname.mydomain Available sub-domains:[tkg.mydomain]

Nothing is configured. The idea is that I would just go in after the fact and assign DNS to the VIP that gets spun up. This is an edge case from our normal usage, for subdomains I don't want developers to be able to provision to without some kind of oversight. Maybe there is another solution I'm not considering?

Describe the solution you'd like

A flag in the helm chart would work. It would mean the AKO operator spun up the ingress anyways. I'm not sure if there is an issue on the AVI side where it won't let you do this?

Describe alternatives you've considered

No response

Additional context

No response

Problem Description

When the enableEVH option is 'true' in AKO, the datascript in HostRule cannot be loaded into VS;
However, httpPolicySet is normal with EVH mode,datascript normal with SNI mode.

Environmental information

* vSphere with Tanzu 7.0U2a
* Guest Cluster: 1.20.2
* AKO: 1.4.2
* AVI: 20.1.5

hostrule yaml

apiVersion: ako.vmware.com/v1alpha1
kind: HostRule
metadata:
  name: avi-hello-kubernetes-hostrule
spec:
  virtualhost:
    fqdn: hello.avi.corp.local
    enableVirtualHost: true
    httpPolicy:
      policySets:
      - corp-httppolicy-rewrite-api
      overwrite: false
    tls: # optional
      sslKeyCertificate:
        name: dc03-corp-local
        type: ref
      sslProfile: System-Standard
      termination: edge
    datascripts:
    - corp-rewrite-api
    wafPolicy: dc03-vds-ako-waf-policy
    applicationProfile: dc03-vds-ako-http

After deleteConfig is set to True, AKO will clean up all the resources it manages and stops reconciling. There is NO way for an external component who instructed AKO's clean up to know the current state.

It must be supported so the external component could safely move forward once it knows reliably AKO has finished all its work.

Currently AKO will just fail to connect to an endpoint with errors, but not crash, if any of the following information changes:

• The Avi Controller IP
• The username
• The password
• The CA data

Instead, AKO should exit with a non-zero exit code so that Kubernetes will restart AKO and allow it to pick up any new, configuration data via its environment variables.

Is your feature request related to a problem? Please describe.

Currently, AKO by default adds some markers to some objects it creates, such as

• clustername
• Namespace
• Host
• ServiceName
• Path
• IngressName

While useful in their own right, being able to add some custom information here, would allow me to pass some info to other automation.

Describe the solution you'd like

Adding a parameter to either the HostRule or HTTPRule CR, accepting a map of custom markers.

Also, it may be worthwhile to have the ability to add "global" custom markers (possibly as values in the AKO helm chart)

Being able to pass in custom markers, can allow me to use them in custom controlscripts when/if necessary.

Describe alternatives you've considered

No response

Additional context

No response

Summary:
After restarting AKO, External-IP are pending for all LoadBalancer type of services.

Detail:

Before restarting AKO:

$ kubectl get svc                 
NAME         TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)          AGE
my-svc-1   LoadBalancer   100.70.222.241     10.186.50.92    80:30008/TCP     17h
my-svc-2   LoadBalancer   100.66.166.195     10.186.52.155   6443:31031/TCP   20m
kubernetes   ClusterIP      100.64.0.1       <none>          443/TCP          17h

After restarting AKO:

$ kubectl get svc         
NAME         TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
my-svc-1   LoadBalancer   100.70.222.241     <pending>     80:30008/TCP     17h
my-svc-2   LoadBalancer   100.66.166.195     <pending>     6443:31031/TCP   29m
kubernetes   ClusterIP      100.64.0.1       <none>        443/TCP          17h

Rerun $ kubectl get svc a few minutes later and the result doesn't change.
But curl <ip> still works fine:

$ curl -k https://10.186.52.155:6443/healthz
ok%     

I create two clusters(one named foo second named bar ) with the same configurations.

  disableStaticRouteSync: "false"
  fullSyncFrequency: "1800"
  networkName: test
  nodeNetworkList: '[{"networkName":"test","cidrs":["xxx.xxx.xxx.xxx/24"]}]'
  serviceEngineGroupName: Default-Group
  serviceType: ClusterIP

The first cluster works well and I am also able to create a L7 ingress, but when I create second cluster using the same configurations, AKO complains:

2021-05-06T23:26:36.162Z	WARN	cache/controller_obj_cache.go:2581	Labels does not match with cluster name for SE group :Default-Group. Expected Labels: [{"key":"clustername","value":"default-test"}]
2021-05-06T23:26:36.385Z	WARN	cache/controller_obj_cache.go:2474	Invalid input detected, AKO will be rebooted to retry
2021-05-06T23:26:36.387Z	INFO	api/api.go:68	Shutting down the API server
2021-05-06T23:26:36.387Z	INFO	api/api.go:113	API server shutdown: http: Server closed
2021-05-06T23:26:36.887Z	WARN	cache/controller_obj_cache.go:2477	Invalid input detected, sync will be disabled.
2021-05-06T23:26:36.888Z	ERROR	ako-main/main.go:134	Handleconfigmap error during reboot, shutting down AKO

However, if I set disableStaticRouteSync: "true", then AKO can be deployed successfully, it would not throw errors.

Any ideas on this one?

Is your feature request related to a problem? Please describe.

I'd like to pull avi credentials from a vault and then create a secret with those credentials. This can be achieved using sealed-secrets.

The problem is, ako doesn't wait for the secret to appear and use it, instead it just creates its own secret overwriting the secret if it already exists.

Describe the solution you'd like

1. Allow a secretName: to be specified rather than require the secret to be added to the values.yaml in order to be used.

avicredentials:
  # specify an existing secret to use, otherwise a secret will be created automatically
  existingSecretName: "<secretName>"
  # if an 'existingSecretName' was not specified, the following values will be used to create a secret named 'avi-secret'
  username: ""
  password: ""
  authtoken: ""
#  certificateAuthorityData: |
#      PEM-encoded public certificate of avi server with Common Name attribute set as the FQDN of the avi server

If 'existingSecretName' is empty then use the passed in values, otherwise expect a secret with the name 'secretName' to exist with the needed values.

2. (a) Also allow an initContainer to be specified so that I can add an initContainer which will wait for the secret to appear. This can be added by adding the vmware 'common' library as a dependency to this helm chart. Like so:

dependencies:
  - name: common
    repository: oci://registry-1.docker.io/bitnamicharts
    tags:
      - bitnami-common
    version: 2.x.x

2. (b) Or, you could go ahead and add the needed initContainer if 'existingSecretName' was specified:

  initContainers:
  - name: wait-for-secret
    image: radial/busyboxplus
    imagePullPolicy: Always
    command: ['sh', '-c']
    args:
    - |
      # Point to the internal API server hostname
      APISERVER=https://kubernetes.default.svc

      # Path to ServiceAccount token
      SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount

      # Read this Pod's namespace
      NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)

      # Read the ServiceAccount bearer token
      TOKEN=$(cat ${SERVICEACCOUNT}/token)

      # Reference the internal certificate authority (CA)
      CACERT=${SERVICEACCOUNT}/ca.crt

      # Explore the API with TOKEN
      while [ 200 -ne $(curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" \
        -X GET ${APISERVER}/api/v1/namespaces/avi-system/secrets/<secretName>\
        --head \
        --silent \
        --output /dev/null \
        --write-out '%{http_code}') ]
      do
        echo "waiting for secret: <secretName>"
        sleep 5
      done
      echo "waiting for secret: <secretName>"
      echo "  detected"

Describe alternatives you've considered

I tried creating the 'avi-secret' secret prior to deploying ako but it was simply overwritten.

Additional context

ESXI 8u2, Tanzu, K8s version v1.26.5

AKO should add support for selector-less LoadBalacaner type of service for customizing endpoints.

AKO needs to use IPs and Ports directly from Endpoints object to create Pool Members for a selector-less LoadBalancer type of Service.

For example, if I want to specify endpoints by my own like this:

apiVersion: v1
kind: Service
metadata:
 name: HA-lb
spec:
  ports:
  - protocol: TCP
    port: 6443
    targetPort: 6443
  type: LoadBalancer
---
apiVersion: v1
kind: Endpoints
metadata:
  name: ha-lb-eps
  namespace: default
subsets:
- addresses:
  - ip: 100.96.0.2
    nodeName: controlplane-node-1
  - ip: 100.96.0.7
    nodeName: controlplane-node-2
  ports:
  - port: 6443
    protocol: TCP

AKO should directly add IPs and Ports from Endpoints ha-lb-eps to virtual service's pool members.

Hi,

I've was recently experimenting with using Avi with the Essential License and using External-dns and noticed an incompatibility with using these together.

External-dns uses the external-dns.alpha.kubernetes.io/hostname to reconcile dns names for a LoadBalancer Service. However, AKO also uses the external-dns.alpha.kubernetes.io/hostname and sends the dns to AVI as well. With the essential license this will fail with the following error:

2021-08-20T18:50:50.262Z        WARN    rest/rest_operation.go:268      RestOp method POST path /api/vsvip tenant admin Obj {"cloud_ref":"/api/cloud?name=Default-Cloud","dns_info":[{"fqdn":"kuard.k8s.example.org"}],"east_west_placement":false,"name":"default-tkg-vc-antrea--default-kuard","tenant_ref":"/api/tenant/?name=admin","vip":[{"auto_allocate_ip":true,"ipam_network_subnet":{"network_ref":"/api/network/?name=VM Network","subnet":{"ip_addr":{"addr":"10.206.80.0","type":"V4"},"mask":20}},"vip_id":"0"}],"vrf_context_ref":"/api/vrfcontext?name=global","vsvip_cloud_config_cksum":"2835294718"} returned err {"code":0,"message":"map[error:VsVip.dns_info: Field VsVip.dns_info cannot be set in ESSENTIALS license tier.]","Verb":"POST","Url":"https://10.206.94.240//api/vsvip","HttpStatusCode":400} with response null                                                                                                                                            
2021-08-20T18:50:50.263Z        INFO    rest/rest_operation.go:163      Failed to remove VsVip ref, object is not of type Virtualservice                                                                                                        
2021-08-20T18:50:50.263Z        WARN    rest/dequeue_nodes.go:522       key: admin/default-tkg-vc-antrea--default-kuard, msg: there was an error sending the macro Error during POST: Encountered an error on POST request to URL https://10.206.94.240//api/vsvip: HTTP code: 400; error from Avi: map[error:VsVip.dns_info: Field VsVip.dns_info cannot be set in ESSENTIALS license tier.]

This will prevent the LoadBalancer service from providing an ExternalIP and I won't be able to use External-dns to provide me DNS.

It would be nice if I could tell AKO to not send dns_info if I'm using Essentials or if it can detect that it is using Essentials and not send the dns_info.

Is your feature request related to a problem? Please describe.

Currently there is no possibility of setting the icap profile similarly to f.e. setting an errorpageprofile or applicationProfile.

Describe the solution you'd like

Allow the addition of an icapProfile in the same way as an erorpageprofile or an applicationprofile

Describe alternatives you've considered

No response

Additional context

No response

Is your feature request related to a problem? Please describe.

We are using kubernetes 1.29 and wan't a proof of concept with AKO. When deploying AKO version 1.11.3 we receive the error that K8s 1.29 API is not supported.

Describe the solution you'd like

Faster AKO releases or the possibility to disable the check by environment variable.

Describe alternatives you've considered

Deploying k8s cluster 1.28 for POC but we would like to use 1.29 because this is out prod k8s version

Additional context

No response

For now, after creating a LoadBalancer type Service, kubectl doesn't show the EXTERNAL-IP for the Service:

➜  playground k get service
NAME              TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes        ClusterIP      100.64.0.1       <none>        443/TCP        12d
nginx-service-1   LoadBalancer   100.70.172.108   <pending>     80:31759/TCP   10m

The loadBalancer in service.status is not updated with the VIP allocated by AVI Controller:

➜  playground k get service nginx-service-1 -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"nginx-service-1","namespace":"default"},"spec":{"ports":[{"port":80,"protocol":"TCP","targetPort":80}],"selector":{"app":"nginx-1"},"type":"LoadBalancer"},"status":{"loadBalancer":{"ingress":[{"ip":"10.184.74.12"}]}}}
  creationTimestamp: "2020-10-26T16:57:52Z"
...
  name: nginx-service-1
  namespace: default
  resourceVersion: "4223172"
  selfLink: /api/v1/namespaces/default/services/nginx-service-1
  uid: c3e3f580-3b16-4707-8ac5-070997a51be6
spec:
  clusterIP: 100.70.172.108
  externalTrafficPolicy: Cluster
  ports:
  - nodePort: 31759
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx-1
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer: {}

Summary:
enable_rhi can’t be configured with ESSENTIALS license since it is a feature for ENTERPRISE. If omit this field, AKO also throws error that this is a must-have field.

Detail:
When LoadBalancer type of service is created in AKO 1.4.1 using essential license, external-ip is pending and the error below shows up:

2021-04-30T23:03:29.091Z	WARN	rest/dequeue_nodes.go:522	key: admin/tkg-vc-antrea--default-corgi-test, msg: there was an error sending the macro Error during POST: Encountered an error on POST request to URL https://10.186.48.73//api/virtualservice/: HTTP code: 400; error from Avi: map[error:VirtualService.enable_rhi: Field VirtualService.enable_rhi cannot be set in ESSENTIALS license tier.] 

If enable_rhi is removed from configmap/avi-k8s-config and restart ako, ako pod shows CreateContainerConfigError:

state:
      waiting:
        message: couldn't find key enableRHI in ConfigMap avi-system/avi-k8s-config
        reason: CreateContainerConfigError

Describe the bug

I am trying to use AKO with NSX-T cloud in a dedicated non-admin tenant. I have deployed all the required infrastructure in NSX-T and the AVI tenant. A manually created VS works fine. But AKO cannot set up static routes on VRF because it tries to look for them in the admin tenant (note the words tenant admin):

ako-0 ako 2024-03-19T15:23:37.717Z      INFO    api/api.go:52   Setting route for GET /api/status
ako-0 ako 2024-03-19T15:23:37.718Z      INFO    ako-main/main.go:77     AKO is running with version: v1.11.3
ako-0 ako 2024-03-19T15:23:37.718Z      INFO    api/api.go:110  Starting API server at :8080
ako-0 ako 2024-03-19T15:23:37.718Z      INFO    ako-main/main.go:87     We are running inside kubernetes cluster. Won't use kubeconfig files.
ako-0 ako 2024-03-19T15:23:37.718Z      INFO    lib/control_config.go:314       Setting the client version to AVI Max supported version 22.1.3
ako-0 ako 2024-03-19T15:23:37.730Z      INFO    ako-main/main.go:174    Kubernetes cluster apiserver version 1.27
ako-0 ako 2024-03-19T15:23:37.736Z      INFO    utils/utils.go:171      Initializing configmap informer in avi-system
ako-0 ako 2024-03-19T15:23:39.674Z      INFO    cache/controller_obj_cache.go:2359      Avi cluster state is CLUSTER_UP_HA_ACTIVE
ako-0 ako 2024-03-19T15:23:39.978Z      INFO    cache/controller_obj_cache.go:3131      Setting cloud vType: CLOUD_NSXT
ako-0 ako 2024-03-19T15:23:39.978Z      INFO    cache/controller_obj_cache.go:3134      Setting cloud uuid: cloud-77594459-fc93-467c-9588-5821fb8d9360
ako-0 ako 2024-03-19T15:23:39.978Z      INFO    lib/lib.go:301  Setting AKOUser: ako-albVS-ORG0001-k8s for Avi Objects
ako-0 ako 2024-03-19T15:23:40.085Z      INFO    cache/controller_obj_cache.go:3413      Skipping the check for Node Network
ako-0 ako 2024-03-19T15:23:40.365Z      INFO    cache/controller_obj_cache.go:3583      Setting VRF T1_ORG0001-001-DATA found that matches the T1Lr /infra/tier-1s/2990ecf4-f802-4157-8ec9-8b28abfac69f
ako-0 ako 2024-03-19T15:23:40.366Z      INFO    record/event.go:285     Event(v1.ObjectReference{Kind:"Pod", Namespace:"avi-system", Name:"ako-0", UID:"7dad8c70-4398-4a51-9066-8675a2753ee5", APIVersion:"v1", ResourceVersion:"53128292", FieldPath:""}): type: 'Normal' reason: 'ValidatedUserInput' User input validation completed.
ako-0 ako 2024-03-19T15:23:40.370Z      INFO    lib/lib.go:240  Setting Disable Sync to: false
ako-0 ako 2024-03-19T15:23:40.373Z      INFO    k8s/ako_init.go:276     avi k8s configmap created
ako-0 ako 2024-03-19T15:23:41.725Z      WARN    lib/avi_api.go:65       msg: Unable to fetch data from uri /api/ipamdnsproviderprofiledomainlist?cloud_uuid=cloud-77594459-fc93-467c-9588-5821fb8d9360 Encountered an error on GET request to URL https://nsx-alb.tld.de//api/ipamdnsproviderprofiledomainlist?cloud_uuid=cloud-77594459-fc93-467c-9588-5821fb8d9360: HTTP code: 400; error from Avi: map[error:Ipam Type invalid or configuration not found in Avi]
ako-0 ako 2024-03-19T15:23:41.816Z      WARN    lib/avi_api.go:65       msg: Unable to fetch data from uri /api/ipamdnsproviderprofiledomainlist?cloud_uuid=cloud-77594459-fc93-467c-9588-5821fb8d9360 Encountered an error on GET request to URL https://nsx-alb.tld.de//api/ipamdnsproviderprofiledomainlist?cloud_uuid=cloud-77594459-fc93-467c-9588-5821fb8d9360: HTTP code: 400; error from Avi: map[error:Ipam Type invalid or configuration not found in Avi]
ako-0 ako 2024-03-19T15:23:41.907Z      WARN    lib/avi_api.go:65       msg: Unable to fetch data from uri /api/ipamdnsproviderprofiledomainlist?cloud_uuid=cloud-77594459-fc93-467c-9588-5821fb8d9360 Encountered an error on GET request to URL https://nsx-alb.tld.de//api/ipamdnsproviderprofiledomainlist?cloud_uuid=cloud-77594459-fc93-467c-9588-5821fb8d9360: HTTP code: 400; error from Avi: map[error:Ipam Type invalid or configuration not found in Avi]
ako-0 ako 2024-03-19T15:23:41.907Z      WARN    cache/controller_obj_cache.go:2521      DNSProperty Get uri /api/ipamdnsproviderprofiledomainlist?cloud_uuid=cloud-77594459-fc93-467c-9588-5821fb8d9360 returned err msg: AviGet retried 3 times, aborting
ako-0 ako 2024-03-19T15:23:41.907Z      WARN    cache/controller_obj_cache.go:2466      Cloud: albC-ORG0001 does not have a dns provider configured
ako-0 ako 2024-03-19T15:23:43.059Z      WARN    nodes/avi_model_routeingr_hostname_shard.go:359 key: Ingress/nginx/nginx, msg: nothing to delete for route: nginx
ako-0 ako 2024-03-19T15:23:43.059Z      WARN    nodes/avi_model_routeingr_hostname_shard.go:359 key: Ingress/oauth/dex, msg: nothing to delete for route: dex
ako-0 ako 2024-03-19T15:23:43.059Z      WARN    nodes/avi_model_l7_translator.go:253    key: Ingress/oauth/dex, msg: secret: dex-tls has been deleted, err: secret "dex-tls" not found
ako-0 ako 2024-03-19T15:23:43.071Z      WARN    status/svc_status.go:39 key: syncstatus, msg: Service hostname not found for service [nginx-ingress-controller/nginx-ingress-controller] status update
ako-0 ako 2024-03-19T15:23:43.100Z      WARN    rest/dequeue_nodes.go:65        key: ORG0001/DummyVSForStaleData, msg: no model found for the key
ako-0 ako 2024-03-19T15:23:43.406Z      WARN    rest/rest_operation.go:304      key: ORG0001/T1_ORG0001-001-DATA, msg: RestOp method PUT path /api/vrfcontext/vrfcontext-d59343ab-8a36-4208-a3c9-42cb30b36b71 tenant admin Obj {"_last_modified":"1710833802833745","attrs":[{"key":"tier1path","value":"/infra/tier-1s/2990ecf4-f802-4157-8ec9-8b28abfac69f"}],"cloud_ref":"https://nsx-alb.tld.de/api/cloud/cloud-77594459-fc93-467c-9588-5821fb8d9360","lldp_enable":true,"name":"T1_ORG0001-001-DATA","static_routes":[{"next_hop":{"addr":"10.70.27.33","type":"V4"},"prefix":{"ip_addr":{"addr":"0.0.0.0","type":"V4"},"mask":0},"route_id":"1"},{"labels":[{"key":"clustername","value":"albVS-ORG0001-k8s"}],"next_hop":{"addr":"10.70.27.35","type":"V4"},"prefix":{"ip_addr":{"addr":"10.244.1.0","type":"V4"},"mask":24},"route_id":"albVS-ORG0001-k8s-1"},{"labels":[{"key":"clustername","value":"albVS-ORG0001-k8s"}],"next_hop":{"addr":"10.70.27.34","type":"V4"},"prefix":{"ip_addr":{"addr":"10.244.2.0","type":"V4"},"mask":24},"route_id":"albVS-ORG0001-k8s-2"},{"labels":[{"key":"clustername","value":"albVS-ORG0001-k8s"}],"next_hop":{"addr":"10.70.27.36","type":"V4"},"prefix":{"ip_addr":{"addr":"10.244.0.0","type":"V4"},"mask":24},"route_id":"albVS-ORG0001-k8s-3"},{"labels":[{"key":"clustername","value":"albVS-ORG0001-k8s"}],"next_hop":{"addr":"10.70.27.40","type":"V4"},"prefix":{"ip_addr":{"addr":"10.244.4.0","type":"V4"},"mask":24},"route_id":"albVS-ORG0001-k8s-4"},{"labels":[{"key":"clustername","value":"albVS-ORG0001-k8s"}],"next_hop":{"addr":"10.70.27.41","type":"V4"},"prefix":{"ip_addr":{"addr":"10.244.5.0","type":"V4"},"mask":24},"route_id":"albVS-ORG0001-k8s-5"},{"labels":[{"key":"clustername","value":"albVS-ORG0001-k8s"}],"next_hop":{"addr":"10.70.27.42","type":"V4"},"prefix":{"ip_addr":{"addr":"10.244.6.0","type":"V4"},"mask":24},"route_id":"albVS-ORG0001-k8s-6"}],"system_default":false,"tenant_ref":"https://nsx-alb.tld.de/api/tenant/tenant-1e7619a4-a24f-4e9c-baa7-edd2c2dab69f","url":"https://nsx-alb.tld.de/api/vrfcontext/vrfcontext-d59343ab-8a36-4208-a3c9-42cb30b36b71","uuid":"vrfcontext-d59343ab-8a36-4208-a3c9-42cb30b36b71"} returned err {"code":0,"message":"map[error:VrfContext object not found!]","Verb":"PUT","Url":"https://nsx-alb.tld.de//api/vrfcontext/vrfcontext-d59343ab-8a36-4208-a3c9-42cb30b36b71","HttpStatusCode":404} with response null
ako-0 ako 2024-03-19T15:23:43.406Z      WARN    rest/dequeue_nodes.go:624       key: ORG0001/T1_ORG0001-001-DATA, msg: there was an error sending the macro Error during PUT: Encountered an error on PUT request to URL https://nsx-alb.tld.de//api/vrfcontext/vrfcontext-d59343ab-8a36-4208-a3c9-42cb30b36b71: HTTP code: 404; error from Avi: map[error:VrfContext object not found!]
ako-0 ako 2024-03-19T15:23:43.406Z      WARN    rest/dequeue_nodes.go:659       key: ORG0001/T1_ORG0001-001-DATA, msg: Avi model not set, possibly a DELETE call
ako-0 ako 2024-03-19T15:23:44.098Z      WARN    nodes/avi_model_l7_translator.go:253    key: Endpoints/oauth/dex, msg: secret: dex-tls has been deleted, err: secret "dex-tls" not found

Reproduction steps

1. Create all required networks and firewall rules in NSX-T manager
2. Create tenant, user, cloud, IPAM profile, SEG, network profiles in NSXALB manager (VRFs are auto-created during cloud creation)
3. Deploy AKO with nodeNetworkList and ClusterIP mode

AKOSettings:
  clusterName: 'k8stest'   # A unique identifier for the kubernetes cluster, that helps distinguish the objects for this cluster in the avi controller. // MUST-EDIT
  cniPlugin: 'cilium' # Set the string if your CNI is calico or openshift or ovn-kubernetes. For Cilium CNI, set the string as cilium only when using Cluster Scope mode for IPAM and leave it empty if using Kubernetes Host Scope mode for IPAM. enum: calico|canal|flannel|openshift|antrea|ncp|ovn-kubernetes|cilium
  disableStaticRouteSync: 'false' # If the POD networks are reachable from the Avi SE, set this knob to true.

### This section outlines the network settings for virtualservices.
NetworkSettings:
  nsxtT1LR: '/infra/tier-1s/2990ecf4-f802-4157-8ec9-8b28abfac69f' # Unique ID (note: not display name) of the T1 Logical Router for Service Engine connectivity. Only applies to NSX-T cloud.

  # Network information of the VIP network. Multiple networks allowed only for AWS Cloud.
  # Either networkName or networkUUID should be specified.
  # If duplicate networks are present for the network name, networkUUID should be used for appropriate network.
  vipNetworkList:
   - networkName: 'segO-T1_ORG0001-001-VIP-0001'

  ## This list of network and cidrs are used in pool placement network for vcenter cloud.
  ## Node Network details are not needed when in nodeport mode / static routes are disabled / non vcenter clouds.
  ## Either networkName or networkUUID should be specified.
  ## If duplicate networks are present for the network name, networkUUID should be used for appropriate network.
  nodeNetworkList:
    - networkName: "segO-T1_ORG0001-001-0001"
      cidrs:
        - 10.244.0.0/16

L4Settings:
  defaultDomain: '' # If multiple sub-domains are configured in the cloud, use this knob to set the default sub-domain to use for L4 VSes.
  autoFQDN: disabled  # ENUM: default(<svc>.<ns>.<subdomain>), flat (<svc>-<ns>.<subdomain>), "disabled" If the value is disabled then the FQDN generation is disabled.

#L7Settings:
#  serviceType: NodePort        # enum NodePort|ClusterIP|NodePortLocal

### This section outlines settings on the Avi controller that affects AKO's functionality.
ControllerSettings:
  serviceEngineGroupName: 'albSEG-ORG0001'   # Name of the ServiceEngine Group.
  cloudName: 'albC-ORG0001'   # The configured cloud name on the Avi controller.
  controllerHost: 'nsx-alb.tld.de' # IP address or Hostname of Avi Controller
  controllerVersion: '22.1.5' # The controller API version
  tenantsPerCluster: true    # If set to true, AKO will map each k8s cluster uniquely to a tenant in AVI.
  tenantName: 'ORG0001'   # Name of the tenant where all the AKO objects will be created in AVI.

avicredentials:
  username: 'ako-ORG0001'
  password: ''
  authtoken: 'redacted'
  certificateAuthorityData: |-
    -----BEGIN CERTIFICATE-----
    redacted
    -----END CERTIFICATE-----

4. See that AKO is unable to set up static routes on VRF in non-admin tenant

Expected behavior

AKO is able to set up static routes on VRF in non-admin tenant

Additional context

AVI Controller Version: 22.1.5
NSX-T Version: 4.1.2.3

Is your feature request related to a problem? Please describe.

We currently use AVI with vSphere and utilize the VRF features. We have all of our networks in one of two non-default VRFs. We don't use the global VRF.

We're experimenting with using AKO in a generic on-prem K8s cluster and have discovered that there is no way to specify the VRF that the objects are created in (we're using NodePort mode). It just always creates the objects (VIPs, VSs, Pools, etc) in the global VRF. This leads to an issue where we can't use our existing VIP network which already has IP addresses allocated and IPAM configured.

I've looked through the values.yaml for the latest version of the chart and I can't find any way to set this. It seems like the VRF is just hard-coded into AKO. I also don't see a way to configure this in the AviInfraSetting CRD.

Describe the solution you'd like

The ability to specify the VRF where objects should be created. This could either be during install using the values.yaml file, or after install with the AviInfraSetting CRD. I don't know which one makes more sense.

Describe alternatives you've considered

To use AVI we would need to manually expose our services as NodePorts and do the AVI configuration outside of Kubernetes.

Additional context

I submitted this as a ticket through VMware support and was told this is not currently possible, but that RFE-2860 has been opened regarding this issue. I wanted to also open it here for increased visibility.

Customer is asking why is AKO exposing a port? There should be no need to access port 80 on the ako pod? In fact, there is no service object and if you curl the pod ip at port 80 you get a connection refused. I raised this on slack. Per Sudipto Biswas.. This is pure noise and an issue should be created.

$kubectl get po ako-0 -n avi-system -oyaml
...
    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /api/status
        port: 8080
        scheme: HTTP
      initialDelaySeconds: 5
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    name: ako-foo
    ports:
    - containerPort: 80
      name: http
      protocol: TCP
    resources:
...

Current AKO does not support TCP/UDP profile customization at Service type LoadBalancer creation time. There will be some benefits as below, if TCP/UDP profiles can be selected by application operator at Service LB vreation rather than changing the assigned profile manually:

• Application operator can select a TCP Fast Path profile with customized Session Idle Timeout value.
• Application operator can deploy UDP LoadBalancer with Per Packet Load Balancing enabled.
• Application operator can select a profile with customized proxy settings.

Currently, if DNS Profile is not configured, Service Status can not be patched successfully.

2020-12-23T00:56:25.425Z	ERROR	status/svc_status.go:37	Service hostname not found for service %v status update[default/test-ip-service]

AKO 1.4.1 introduced Gateway API to support advanced L4 service expose. But after enable this feature, all service type LoadBalancer will not work and have to switch to Gatway.

In some cases, customers only want to use Gateway for specific application, like multiple service sharing the same vip with different port. And for other majority application, service type LoadBalancer is sufficient and easy to use.

In additional, Gateway and service type LoadBalancer can't co-exists will affect current kubernetes users as they already using lots of service type LoadBalancer. If they want to use Gateway it'll be difficult to switch from service type to LoadBalancer without interruption.

I think the best way is there's a switch or selector(like amko implemention) that user can specify to let partial service expose via Gateway and other service expose via Service type LoadBalancer by default.

Describe the bug

If the cloud configuration does not define an ipam provider a panic occurs.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1741002]
 
goroutine 1 [running]:
github.com/vmware/load-balancer-and-ingress-services-for-kubernetes/ako-infra/avirest.AviCloudCachePopulate(0x0?, {0x1b83ff7, 0xd})
        /build/mts/release/bora-21883866/cayman_load-balancer-and-ingress-services-for-kubernetes/build/release/lin64/load-balancer-and-ingress-services-for-kubernetes/build/src/ako-infra/avirest/handle_netinfo.go:398 +0x3c2
github.com/vmware/load-balancer-and-ingress-services-for-kubernetes/ako-infra/avirest.getAviCloudFromCache(...)
        /build/mts/release/bora-21883866/cayman_load-balancer-and-ingress-services-for-kubernetes/build/release/lin64/load-balancer-and-ingress-services-for-kubernetes/build/src/ako-infra/avirest/handle_netinfo.go:345
github.com/vmware/load-balancer-and-ingress-services-for-kubernetes/ako-infra/avirest.SyncLSLRNetwork()
        /build/mts/release/bora-21883866/cayman_load-balancer-and-ingress-services-for-kubernetes/build/release/lin64/load-balancer-and-ingress-services-for-kubernetes/build/src/ako-infra/avirest/handle_netinfo.go:51 +0x110
main.InitializeAKOInfra()
        /build/mts/release/bora-21883866/cayman_load-balancer-and-ingress-services-for-kubernetes/build/release/lin64/load-balancer-and-ingress-services-for-kubernetes/build/src/cmd/infra-main/main.go:114 +0x86a
main.main()
        /build/mts/release/bora-21883866/cayman_load-balancer-and-ingress-services-for-kubernetes/build/release/lin64/load-balancer-and-ingress-services-for-kubernetes/build/src/cmd/infra-main/main.go:39 +0x17

Reproduction steps

1. Setup a cloud without IPAM provider
2. Run ako pod
3. Panic occurs

Expected behavior

No panic should occur.

Additional context

I think the problem is this line:
https://github.com/vmware/load-balancer-and-ingress-services-for-kubernetes/blob/master/ako-infra/avirest/handle_netinfo.go#L399

It deferences the CloudCache.IPAMProviderRef without checking for nil.

In statefulset.yaml line 19~21, imagePullSecrets's format is wrong, it should be - name: {{ toYaml . }}

Original:

    {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}

Fix:

    {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        - name: {{  toYaml . }}
    {{- end }}

References:

Official doc:
https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry

Another helm example:
https://github.com/Azure-Samples/helm-charts/blob/master/chart-source/image-pull-secret/templates/acs-helloworld.yaml