pubsub/Kafka component not connecting to Kafka cluster with SSL listeners about dapr HOT 3 CLOSED

In order to perform some more troubleshooting, I've now added the same Kafka cluster to the dapr configuration using the plaintext listeners:

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: kafkasafe2
spec:
  type: pubsub.kafka
  version: v1
  metadata:
  - name: brokers # Required. Kafka broker connection setting
    value: "kafka1:9192,kafka2:9192,kafka3:9192"
  - name: clientID
    value: "dapr-pub"
  - name: authType # Required
    value: "none"
  - name: disableTls
    value: "false"

If I filter the logs by the clustername, I get very different behaviors:

Cluster with SSL listeners:

time="2024-01-23T09:15:38.14784818Z" level=debug msg="Found component: kafkacluster (pubsub.kafka/v1)" app_id=kafkasafe-pub instance=0074ac53f3fc scope=dapr.runtime type=log ver=1.12.4
time="2024-01-23T09:15:38.149786801Z" level=debug msg="Loading component: kafkacluster (pubsub.kafka/v1)" app_id=kafkasafe-pub instance=0074ac53f3fc scope=dapr.runtime type=log ver=1.12.4
time="2024-01-23T09:15:38.150066896Z" level=warning msg="Component kafkacluster references a secret store that isn't loaded: secretsFile" app_id=kafkasafe-pub instance=0074ac53f3fc scope=dapr.runtime type=log ver=1.12.4

Cluster with plaintext listeners:

ime="2024-01-23T09:15:38.149807555Z" level=debug msg="Found component: kafkasafe2 (pubsub.kafka/v1)" app_id=kafkasafe-pub instance=0074ac53f3fc scope=dapr.runtime type=log ver=1.12.4
time="2024-01-23T09:15:38.150695755Z" level=debug msg="Loading component: kafkasafe2 (pubsub.kafka/v1)" app_id=kafkasafe-pub instance=0074ac53f3fc scope=dapr.runtime type=log ver=1.12.4
time="2024-01-23T09:15:38.151418666Z" level=debug msg="ConsumerGroup='kafkasafe-pub', ClientID='dapr-pub', saslMechanism=''" app_id=kafkasafe-pub component="kafkasafe2 (pubsub.kafka/v1)" instance=0074ac53f3fc scope=dapr.contrib type=log ver=1.12.4
time="2024-01-23T09:15:38.151474607Z" level=debug msg="Found brokers: [kafka1:9192 kafka2:9192 kafka3:9192]" app_id=kafkasafe-pub component="kafkasafe2 (pubsub.kafka/v1)" instance=0074ac53f3fc scope=dapr.contrib type=log ver=1.12.4
time="2024-01-23T09:15:38.151684424Z" level=debug msg="No authentication configured." app_id=kafkasafe-pub component="kafkasafe2 (pubsub.kafka/v1)" instance=0074ac53f3fc scope=dapr.contrib type=log ver=1.12.4
time="2024-01-23T09:15:38.151927919Z" level=debug msg="Initializing new client" app_id=kafkasafe-pub component="kafkasafe2 (pubsub.kafka/v1)" instance=0074ac53f3fc scope=dapr.contrib type=log ver=1.12.4
time="2024-01-23T09:15:38.152331791Z" level=debug msg="client/metadata fetching metadata for all topics from broker kafka3:9192\n" app_id=kafkasafe-pub component="kafkasafe2 (pubsub.kafka/v1)" instance=0074ac53f3fc scope=dapr.contrib type=log ver=1.12.4
time="2024-01-23T09:15:38.15609553Z" level=debug msg="Connected to broker at kafka3:9192 (unregistered)\n" app_id=kafkasafe-pub component="kafkasafe2 (pubsub.kafka/v1)" instance=0074ac53f3fc scope=dapr.contrib type=log ver=1.12.4
time="2024-01-23T09:15:38.158709974Z" level=debug msg="client/brokers registered new broker #1 at kafka1:9192" app_id=kafkasafe-pub component="kafkasafe2 (pubsub.kafka/v1)" instance=0074ac53f3fc scope=dapr.contrib type=log ver=1.12.4
time="2024-01-23T09:15:38.158769583Z" level=debug msg="client/brokers registered new broker #2 at kafka2:9192" app_id=kafkasafe-pub component="kafkasafe2 (pubsub.kafka/v1)" instance=0074ac53f3fc scope=dapr.contrib type=log ver=1.12.4
time="2024-01-23T09:15:38.158967282Z" level=debug msg="client/brokers registered new broker #3 at kafka3:9192" app_id=kafkasafe-pub component="kafkasafe2 (pubsub.kafka/v1)" instance=0074ac53f3fc scope=dapr.contrib type=log ver=1.12.4
time="2024-01-23T09:15:38.159314071Z" level=debug msg="Successfully initialized new client" app_id=kafkasafe-pub component="kafkasafe2 (pubsub.kafka/v1)" instance=0074ac53f3fc scope=dapr.contrib type=log ver=1.12.4
time="2024-01-23T09:15:38.159613852Z" level=debug msg="Kafka message bus initialization complete" app_id=kafkasafe-pub component="kafkasafe2 (pubsub.kafka/v1)" instance=0074ac53f3fc scope=dapr.contrib type=log ver=1.12.4
time="2024-01-23T09:15:38.159944936Z" level=info msg="Component loaded: kafkasafe2 (pubsub.kafka/v1)" app_id=kafkasafe-pub instance=0074ac53f3fc scope=dapr.runtime type=log ver=1.12.4

It looks like the component kafkacluster is not being initialized... but still I don't get any errors in the logs stating that or suggesting what the problem could be, just the warning I've put in bold.

Any ideas?

from dapr.

The name of the secret store component that you are defining is kafka-tls

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: kafka-tls
spec:
  type: secretstores.local.file
...

The error time="2024-01-23T09:15:38.150066896Z" level=warning msg="Component kafkacluster references a secret store that isn't loaded: secretsFile" app_id=kafkasafe-pub instance=0074ac53f3fc scope=dapr.runtime type=log ver=1.12.4
says you are trying to access a secret store named secretsFile as configured in the field

auth:
  secretStore: secretsFile

can you try by changing it to

auth:
  secretStore: kafka-tls

from dapr.

Hello @mukundansundar ,

thanks for the help.

That was indeed one of the issues. I would have expected the application to fail if it wasn't able to access the secrets (as I was referencing a wrong secret store)... If Dapr is not able to access the secrets, it should explicitly say it and fail, no?

Now it's finally working with this configuration:

If it can help anyone, I'll post the solution.

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: kafkacluster
spec:
  type: pubsub.kafka
  version: v1
  metadata:
  - name: brokers # Required. Kafka broker connection setting
    value: "kafka1:9199,kafka2:9199,kafka3:9199"
  - name: clientID
    value: "dapr-pub"
  - name: authType # Required
    value: "mtls"
  - name: disableTls
    value: "false"
  - name: caCert
    secretKeyRef:
      name: certificates
      key: caCert
  - name: clientCert
    secretKeyRef:
      name: certificates
      key: clientCert
  - name: clientKey
    secretKeyRef:
      name: certificates
      key: clientKey
auth:
  secretStore: kafka-tls
---
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: kafka-tls
spec:
  type: secretstores.local.file
  version: v1
  metadata:
  - name: secretsFile
    value: dapr/certificates/cer-store.json
  - name: multiValued
    value: "true"
---

the json file should be structured like this:

Please make sure the json contains the \n instead of the newlines

{
"certificates": {
"caCert": "-----BEGIN CERTIFICATE-----\nMIIGwDCCBKigAwIBAgIBATANBgkqhkiG9w0BAQsFADCB7zELMAkGA1UEBhMCQkUx**\nETAPlw==\n**-----END CERTIFICATE-----",
"clientCert":"-----BEGIN CERTIFICATE-----\nMIIFVDCCAzygAwIBAgIIE3qYAfC9/gkwDQYJKoZIhvcNAQELBQAwgeUxCzAJBgNV**\nBAYTAk
...
mIhU4OzAUyPusPUG7YM+Hms8Dbcj\nnCSkw+mjA71j0cJTwh1wqCtPlHLhFuIm\n**-----END CERTIFICATE-----",
"clientKey":"-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCypudP1g3y5PZ2**\nKKCfFsQPUt
...
OE6Js=\n**-----END PRIVATE KEY-----"
}
}

from dapr.