[Question] Why intermittently cannot find SpoolDir? about demo-scene HOT 3 CLOSED

How much memory have you allocated to Docker? What's the docker-compose logs kafka show as the reason for it exiting?

For general questions about the capabilities of ksqlDB head to http://cnfl.io/slack and the #ksqldb channel.

from demo-scene.

Hi Robin

Is confluent-hub install jcustenborder/kafka-connect-spooldir:2.0.43 a one time install?

Not sure why this is needed every time Mac machine starts up

does not return anything until installation of this connector happens next time?

even after installing connecting

curl -s localhost:8083/connector-plugins|jq '.[].class'|egrep 'SpoolDir

is not returning anything from time to time?

from demo-scene.

OK

now it is returning:

within the docker-compose.yml does this mean user needs to install the kafka-connect-spooldir manually on the host machine or inside docker then? Since neither of these were set:

/usr/share/java,/usr/share/confluent-hub-components

Hence spooldir was not available after docker-compose starts?

  CONNECT_PLUGIN_PATH: '/usr/share/java,/usr/share/confluent-hub-components/'
command: 
  - bash 
  - -c 
  - |
    echo "Installing connector plugins"
    confluent-hub install --no-prompt confluentinc/kafka-connect-jdbc:5.5.0
    confluent-hub install --no-prompt jcustenborder/kafka-connect-spooldir:2.0.43

EDIT: Realized didn't put correct path into docker-compose CONNECT_PLUGIN_PATH

so now:

  CONNECT_PLUGIN_PATH: '/usr/share/java,/usr/local/Cellar/confluent-platform/5.5.1/libexec/share/confluent-hub-components'

docker-compose up kafka-connect

But why

curl -s localhost:8083/connector-plugins|jq '.[].class'|egrep 'SpoolDir'
➜ csv-to-kafka git:(master) ✗

sometimes It returns and at other times SpoolDir not present?

Do I need to change
image: confluentinc/cp-kafka-connect-base:5.5.0

to a local image from Dockerfile as shown in video to work?

Also:

kafka-connect | [main] WARN org.apache.kafka.clients.ClientUtils - Couldn't resolve server kafka:29092 from bootstrap.servers as DNS resolution failed for kafka
kafka-connect | [main] ERROR io.confluent.admin.utils.cli.KafkaReadyCommand - Error while running kafka-ready.

from demo-scene.