Test if Prometheus metrics telemetry - https://istio.io/docs/tasks/telemetry/metrics-logs/ is working fine

Describe the bug
Unable to attach cert to ELB

Need to attach these annotations:

  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-west-2:{act_num}:certificate/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx"
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "300"

The good news is the ELB already comes up as TCP. :) This is a huge improvement.

Add configurable node affinities per Istio components

We should consider HPA made changes to prevent the reconciliation logic reverting those changes.

• Pilot
• Gateways
• Mixer
• Galley

Istio config CRD status should show the current state (components available, etc)

e.g.: control plane security

In Istio 1.1 lighstep and jaeger/zipkin are both supported.
By default the zipkin address is set to zipkin.istio-system:9411, we should make it configurable to integrate with custom zipkin/jager, (and lightstep) installations.
Related to #41.

• flat federation
• gateway federation

... but they seem to be working fine

the Helm chart can install all of these, but it's probably not the scope of the operator to manage these components, but we should figure out how to integrate them.

Nice work on the operator folks.. With regard to the multicluster setup, we have updated the multicluster setup a while ago. You no longer need a vpn for the single control plane-noHA option - i.e. no longer need unique pod CIDRs..

Secondly, there is a much more simpler multicluster option with two control planes (one on each cluster), and pods communicating with each other via the gateways. I believe the operator can be of great help in this setup to spit out the global services (services that are visible across clusters), and assign unique VIPs to these services..

https://preliminary.istio.io/docs/setup/kubernetes/multicluster-install/gateways/
I would be happy to give you more details or the workflow.

it is now set to ALLOW_ANY

Parallelize CRD config creation, now it's rather slow

Test if objects are actually created

Describe the bug
Applying routing rules does not route as expected -- get a 404 on all routes.

In the code yaml files below I've replaced identifying information (hosts and various id numbers) with placeholders.

Steps to reproduce the issue:
Install the istio-operator by following the instructions i.e. running make deploy. Apply the following istio configuration.

apiVersion: istio.banzaicloud.io/v1beta1
kind: Istio
metadata:
  labels:
    controller-tools.k8s.io: "1.0"
  name: istio
  namespace: istio-system
spec:
  mtls: false
  includeIPRanges: "*"
  excludeIPRanges: ""
  autoInjectionNamespaces:
  - "app-stg"
  - "app-prd"
  - "httpbin"
  controlPlaneSecurityEnabled: false
  gateways:
    ingress:
      serviceAnnotations:
        service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:{REGION}:{ACCT_NUMBER}:certificate/{CERT_UUID}"
        service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
        service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "300"

Then create a virtual service as below

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: app-stg
  namespace: default
spec:
  hosts:
  - "app-stg.example.com"
  - "app-stg.blue.example.com"
  - "app-stg.green.example.com"
  gateways:
  - istio-autogenerated-k8s-ingress
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        port:
          number: 80
        host: httpbin.httpbin.svc.cluster.local

Expected behavior
The hosts listed above should be routed to httpbin (which is running in the namespace httpbin).

Additional context
Add any other context about the problem like release number version, branch, etc.
Using Istio operator 0.0.6.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.6", GitCommit:"ab91afd7062d4240e95e51ac00a18bd58fddd365", GitTreeState:"clean", BuildDate:"2019-02-26T12:59:46Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.7", GitCommit:"65ecaf0671341311ce6aea0edab46ee69f65d59e", GitTreeState:"clean", BuildDate:"2019-01-24T19:22:45Z", GoVersion:"go1.10.7", Compiler:"gc", Platform:"linux/amd64"}

Once Istio will have a proper client package, use that instead of dynamic clients

Enable Control Plane security

Some secrets are created by Citadel and not managed by the operator directly.
These resources are left over when Istio is deleted from a cluster.

This scenario should work:

• A cluster is created with mTLS disabled. (the default MeshPolicy is created)
• mTLS value is enabled in the config - the operator triggers reconciling, the default MeshPolicy should be updated, DestinationRules should be created
• mTLS value is disabled - the operator triggers reconciling, the default MeshPolicy should be updated, DestinationRules should be deleted

(use escalate from 1.12)

Istio defaults to generating trace spans for 1 out of every 100 requests (sampling rate of of 1%).

We should allow this value to be configured in the CR.

are there any releases/branches that support the 1.1 series (or the rc0)? if not, we would be happy to help..
cc @costinm @sdake

Remove privileged Envoy containers

Is your feature request related to a problem? Please describe.

Add support/integration with the Banzai Cloud Kafka Operator. The Kafka Operator already orchestrates an Envoy proxy in order to route external broker access towards the broker pod - instead of relying on the load balancer's e.g. round-robin feature. While brokers come and go (autoscaling, failovers, etc) the Kafka operator already generates complex Envoy configs - adding mTLS, etc complicates things even more.

Describe the solution you'd like to see

Add support for Kafka, by wiring brokers into the mesh. Also, add support for the Envoy Kafka protocol filter once is pushed upstream.

Enable mTLS

Prior to Istio 1.1, the keys and certificates of Istio workloads were generated by Citadel and distributed to sidecars through secret-volume mounted files. These issues are addressed in Istio 1.1 through the SDS identity provision flow.
Related docs: https://preliminary.istio.io/docs/tasks/security/auth-sds/

A new component (nodeagent) should be added to the operator, and SDS should be configurable where needed.

for now only the main config CR is watched, so if anything changes with the Istio resources, the operator won't kick off the reconciling logic.

Add support for Istio master (1.1) instead of 1.0.5 only

When the Istio config CR is deleted during reconciling, some resources are left over

• check if updating the ResourceVersion is needed
• check if we can use deepCopy or 2WayMerge to compare the current state with the desired and only update if necessary

Wait until galley is up and running and only create CRDs after that