When long GRPC routes are used, the certificate generation fails with the following error:

--------------------------- Ansible Task StdOut -------------------------------

TASK [microcks : The Microks GRPC certs are generated] *************************
task path: /opt/ansible/roles/microcks/tasks/main.yml:334


-------------------------------------------------------------------------------
{"level":"info","ts":1626808452.0495095,"logger":"logging_event_handler","msg":"[playbook task]","name":"some-microcks","namespace":"xxxxx-yyyyyyy-common","gvk":"microcks.github.io/v1alpha1, Kind=MicrocksInstall","event_type":"playbook_on_task_start","job":"6129484611666145821","EventData.Name":"microcks : The Microks GRPC certs are generated"}
{"level":"error","ts":1626808452.779658,"logger":"logging_event_handler","msg":"","name":"some-microcks","namespace":"xxxxx-yyyyyyy-common","gvk":"microcks.github.io/v1alpha1, Kind=MicrocksInstall","event_type":"runner_on_failed","job":"6129484611666145821","EventData.Task":"The Microks GRPC certs are generated","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/microcks/tasks/main.yml:334","error":"[playbook task failed]","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/[email protected]/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/events.loggingEventHandler.Handle\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/events/log_events.go:87"}

--------------------------- Ansible Task StdOut -------------------------------

 TASK [The Microks GRPC certs are generated] ******************************** 
fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["openssl", "req", "-x509", "-nodes", "-days", "3650", "-newkey", "rsa:2048", "-keyout", "microcks-grpc.key", "-out", "microcks-grpc.crt", "-subj", "/CN=some-microcks-xxxxx-yyyyyyy-common.apps.mw-ocp4.cloud.lab.eng.bos.redhat.com", "-extensions", "san", "-config", "microcks-grpc.cnf"], "delta": "0:00:00.068440", "end": "2021-07-20 19:14:12.739156", "msg": "non-zero return code", "rc": 1, "start": "2021-07-20 19:14:12.670716", "stderr": "Generating a RSA private key\n.............+++++\n.........................+++++\nwriting new private key to 'microcks-grpc.key'\n-----\nproblems making Certificate Request\n140581912262464:error:0D07A097:asn1 encoding routines:ASN1_mbstring_ncopy:string too long:crypto/asn1/a_mbstr.c:107:maxsize=64", "stderr_lines": ["Generating a RSA private key", ".............+++++", ".........................+++++", "writing new private key to 'microcks-grpc.key'", "-----", "problems making Certificate Request", "140581912262464:error:0D07A097:asn1 encoding routines:ASN1_mbstring_ncopy:string too long:crypto/asn1/a_mbstr.c:107:maxsize=64"], "stdout": "", "stdout_lines": []}

As of today, the Operator backed installation process of MicrocksInstall custom resources allows:

• Generating self-signed certificates for ingress,
• OR using an existing secret to reference a pre-populated secret.

We should provide options for allowing user to place annotations allowing:

• To specify ingress classes or some other customization markers,
• To trigger some certificate generation process like the one provided by https://cert-manager.io/

Got it published on OperatorHub.io and tagged in GitHub

This is the consequence of microcks/microcks#468

In order to quickly setup a new test environment, we should have a all-in-one scripts for OpenShift and Minikube that allow quick installation and setup of Microcks with async features.

Kubernetes has been deprecating apiextensions.k8s.io/v1beta1 API, which will be removed and are no longer available in 1.22. For our operator to support 1.22 and other, we have to move the CRD definition to apiextensions.k8s.io/v1beta.

When installing the Kafka broker on a vanilla Kubernetes cluster, the broker URL is not correctly reported into the features.properties configuration file. This is not crucial as this property is for display only but it gave bad ifnormation to users. See microcks/microcks#332

Next release 0.9.0 of Microcks will need an extra configuration file called features.properties that should be present into the ConfigMap. Add this as well as the default value into CRD.

Embed following issues:

• #3 Create Route with specific host on OpenShift
• #4 Kubernetes installation not complete regarding TLS ingresses

Currently installation is expecting TLS routes for Microcks and Keycloak access but Ingresses are not created using TLS. The setup is expected to be reviewed and complete by a cluster administrator.

We should provide fully configured installation, creating TLS ingresses - at least using self generated and signed certificates for TLS support in ingresses.

With the latest versions of Strimzi, there is a new way to define the listeners for the Kafka custom resource. The current usage is still supported but will be removed in the future.

Create a couple files with the examples mentioned in the README.

Create a single file install for all the resources required by the operator. For example, like strimzi does.

This is the consequence on Operator of microcks/microcks#408

Report in operator the parameters that have been added into Helm Charts. See microcks/microcks#481

I think it would be very useful to have a simple CRD that allows managing importer jobs.

Thanks!

For now, OpenShift routes are created using the default URI schema handled by the Router.
That is because whenever I try adding a custom hostname in templates, OpenShift complains about not being possible to modify the spec.host field of Route for security reasons.

Will have to investigate...

Following microcks/microcks#322 we should add correct securityContext to MongoDB deployment when running on vanilla Kubernetes.

I would like to use different image registries or internal ones for deploying Microcks. I think the image tag could be added to the CRD to allow that.

  mongodb:
    image: myregistry:5000/microcks/mongodb-34-centos7:latest
    install: true

Following the Spring 2.4.5 version update (see microcks/microcks#406) logs are much more verbose as we got DEBUG levels for some Spring and Netty components. We should lower that by default.

Also it appears that we no longer have timestamp information on log message so fix that.

Currently you only get the keycloak master realm user as part of the operator install. It could be useful if we can also add a microcks user from the beginning to avoid new users the need to deep dive on keycloack to get started with microcks.

Before 0.2.0 release is out, complete Operator documentation on all available options.

Since microcks/microcks#169, version 0.2.0 of the operator cannot be deployed anymore. Upgrade operator to use the new keycloak images.

As a consequence of microcks/microcks#407, Keycloak version should be upgraded

When the features.async.enabled flag of the CR is set to false, the Operator is checking that a CR corresponding to Microcks Kafka broker is not present. However for doing that, the Strimzi CRD must be defined on the cluster. Otherwise it raises an Error and makes the Operator crash.

This issue was not detected by our CI tests because we start creating a new cluster and setting up with dependency CRDs (Strimzi included). Then we start creating a CR with features.async.enabled to false. The test succeeds but we do not detect this hard dependency to CRD presence.

As of today, Ingress TLS certificate and key are automatically generated when creating the Ingress on Kubernetes. We should be able to retrieve them from an existing Secret into the namespace.

This is a sub-task of microcks/microcks#424.

This is the report on Operator of microcks/microcks#315

When deploying keycloak with privateURL in the MicrocksInstall CR, the Async minion deployment stills try to use the public url (not accessible) to connect.

Got it published on OperatorHub.io and tagged in GitHub

Got it published on OperatorHub.io and tagged in GitHub

Since #14 the Operator SDK base image is no longer shipping openssl that makes the operator fails when generating self-signed certificates for a rapid test drive on Minikube (for example).

Fix this!

This is the consequence of microcks/microcks#337.

This is a sub-task of microcks/microcks#433.

Got it published on OperatorHub.io and tagged in GitHub

Upgrade to the latest SDK version

Just after having fixed #7, we need to prepare operator deployment upgrading CSV and package description.

We found on some OpenShift clusters that the API Groups Information is not always available making the Operator crash on the following command because of HTTP response 502:

api_groups: "{{ lookup('k8s', cluster_info='api_groups') }}"

A more stable (but maybe more crappy ?) solution could be to look at the available annotations on namespace to try detecting some OpenShift specific ones like this:

- name: Retrieve and set watch namespace
  set_fact:
    watch_namespace: "{{ lookup('env','WATCH_NAMESPACE') }}"

- name: Get watched namespace infos
  set_fact:
    watched_namespace: "{{ lookup('k8s', kind='Namespace', resource_name=watch_namespace) }}"

- name: Determine is this is an OpenShift or Kubernetes namespace
  set_fact:
    is_openshift : "{{ watched_namespace.metadata.annotations['openshift.io/display-name'] is defined }}"
    is_kubernetes : "{{ watched_namespace.metadata.annotations['openshift.io/display-name'] is not defined }}"

Maybe we should consider this as a fallback in case of official API returning an error?

Certificates generation fails when Microcks is installed with long urls. This has been analyse through #39 and #41.
This is due to certificate CN that is set to urls but shout not be longer to 64 characters according the spec.
Because we use subjectAltName to specify URL in certificate, we do not need to have CN set to URL and can truncate URL to 64 characters.

This issue for tracking the changes related to microcks/microcks#314

This is the consequence of microcks/microcks#464

As of today we support already existing installation of MongoDB but the Secret used should have a fixed name corresponding to {{name}}-mongodb-connection with fixed keys being username and password. We should allow more flexible ways of referencing an existing Secret.

Following microcks/microcks#345 we should add correct securityContext to PostgreSQL deployment when running on vanilla Kubernetes.

Describe the bug

When I run this container on an ARM system I see the following error:
standard_init_linux.go:207: exec user process caused "exec format error"

I think it may be related to the system being ARM based, and your image was not compiled in ARM based systems.
Is it possible to add an ARM build? For example using the tag arm and arm64 for each system.

Tested on Kubernetes 1.17

When using the latest version of the nginx ingress controller there is warning about the configuration of the self signed certs because of the use of CN.

For some reason I have not been able to provision the async feature connected to an external Kafka and an external Keycloak.
It doesn't seem to work with an internal Keycloak either.

2021-07-22 12:29:16,169 INFO  [io.git.mic.min.asy.AsyncMinionApp] (main) Microcks Keycloak server url {https://keycloak-myproject.apps.example.com/auth} and realm {mercury}
2021-07-22 12:29:16,176 INFO  [io.git.mic.min.asy.AsyncMinionApp] (main) Use locally defined Keycloak Auth URL: Optional[http://keycloak-discovery.myproject.svc.cluster.local:8080]
2021-07-22 12:29:16,200 ERROR [io.git.mic.min.asy.cli.KeycloakConnector] (main) OAuth token cannot be retrieved for Keycloak server, check microcks.serviceaccount configuration
2021-07-22 12:29:16,201 ERROR [io.git.mic.min.asy.AsyncMinionApp] (main) Cannot authenticate to Keycloak server and thus enable to call Microcks APIto get Async APIs to mocks...: io.github.microcks.minion.async.client.ConnectorException: OAuth token cannot be retrieved for Microcks. Check microcks.serviceaccount.
	at io.github.microcks.minion.async.client.KeycloakConnector.connectAndGetOAuthToken(KeycloakConnector.java:103)
	at io.github.microcks.minion.async.client.KeycloakConnector_ClientProxy.connectAndGetOAuthToken(KeycloakConnector_ClientProxy.zig:157)

This is the configuration I used:

apiVersion: microcks.github.io/v1alpha1
kind: MicrocksInstall
metadata:
  name: bm
  labels:
    app: mercury
spec:
  features:
    async: 
      enabled: true
      defaultBinding: KAFKA
      defaultFrequency: 10
      kafka:
        install: false
        url: my-kafka-kafka-bootstrap:9092
  keycloak:
    install: false
    realm: mercury
    url: keycloak-myproject.apps.example.com
    privateUrl: http://keycloak-discovery.myproject.svc.cluster.local:8080
    serviceAccount: microcks-serviceaccount
    serviceAccountCredentials: super-secret
  microcks:
    replicas: 1
  mongodb:
    install: true
    persistent: true
    volumeSize: 2Gi
  name: bm
  postman:
    replicas: 1
  version: 1.3.0

The microcks-serviceaccount client exists in the realm and seems to be similar to the one provisioned by microcks itself (which is failing at the same point).

When accessing the pod for debug I can manually retrieve the token:

$ oc debug bm-async-minion-7558586b49-f66fw
$ export KC_URL=http://keycloak-discovery.myproject.svc.cluster.local:8080
$ curl -XPOST -H "Content-Type: application/x-www-form-urlencoded" -H "Authorization: Basic bWljcm9ja3Mtc2VydmljZWFjY291bnQ6c3VwZXItc2VjcmV0" -d "grant_type=client_credentials" "$KC_URL/auth/realms/mercury/protocol/openid-connect/token"
{# JsonToken}

I have also tried with the https://keycloak.myproject.svc.cluster.local:8443 but I guess the problem in that case is that it doesn't trust the self-signed certificate.

Any hints of what can be missing?
Besides that would it be possible to add more information to this stacktrace about what the server is responding for better troubleshooting?

Thanks in advance

SoapUI 5.6.0 version upgrade required adjusting the log levels to avoid to many logs. Changes are coming from microcks/microcks#405.

As a consequence of microcks/microcks#380, we should also add a specific installation flag for Operator that allows tuning the log level. For convenience, the default should be set to INFO or WARN and allow to put some DEBUG or TRACE verbosity level.

Microcks operator 1.3.0 on OCP 4.6.34 failed to reconcile a Microcks install that was installed at the 1.2.1 level after the operator was upgraded. The reconcile error is:

An unhandled exception occurred while running the lookup plugin 'file'. Error was a <class 'ansible.errors.AnsibleError'>, original message: could not locate file in lookup: microcks-grpc.crt

The Microcks CR spec is:

spec:
  keycloak:
    install: true
    persistent: true
    volumeSize: 1Gi
  microcks:
    replicas: 1
  mongodb:
    install: true
    persistent: true
    volumeSize: 2Gi
  name: mas-iot-microcksinstall
  postman:
    replicas: 1
  version: 1.2.1
status:
  conditions:
    - lastTransitionTime: '2021-07-28T07:58:35Z'
      message: Running reconciliation
      reason: Running
      status: 'False'
      type: Running
    - ansibleResult:
        changed: 0
        completion: '2021-07-28T07:59:16.821622'
        failures: 1
        ok: 27
        skipped: 19
      lastTransitionTime: '2021-07-28T07:59:17Z'
      message: >-
        An unhandled exception occurred while running the lookup plugin 'file'.
        Error was a <class 'ansible.errors.AnsibleError'>, original message:
        could not locate file in lookup: microcks-grpc.crt
      reason: Failed
      status: 'True'
      type: Failure

Also creating a new MicrocksInstall instance using the same updated operator also fails with the same error.