Welcome to the Kafka Toolkit. This toolkit enables SPL applications to integrate with Apache Kafka.

This version of the toolkit currently supports: Apache Kafka 0.10.2, 0.11, 1.0, 1.1, 2.0, and 2.1. For the older Kafka versions 0.9 and 0.10.0, please use the Kafka operators from the com.ibm.streamsx.messaging toolkit.

For building applications that connect to the IBM Cloud Event Streams service, it is recommended that the com.ibm.streamsx.messagehub toolkit be used. This toolkit provides functionality on top of the Kafka toolkit to simplify the configuration needed to connect to the cloud service.

To migrate applications using the old Kafka operators in the com.ibm.streamsx.messaging toolkit, refer to the Migration Document.

• Toolkit documentation can be found here: SPLDoc
• Standard use cases for Consumers and Partitions can be found on the overview page of the user documentation
• Toolkit reference for latest and older versions

• Toolkit overview
• Improve performance when consuming from Kafka
• Other kafka related content on Streamsdev

cd com.ibm.streamsx.kafka
../gradlew build

./gradlew spldoc

NOTE: SPLDocs will be generated in the docs/spldoc directory.

./gradlew release

NOTE: The release will be available in the build/release/output folder.

cd tests/KafkaTests
./setup.sh -b <list_of_bootstrap_servers>
../../gradlew test

NOTE 1: setup.sh will add a domain-level app config called "kafka-tests", as well as create a properties file containing the bootstrap.servers property.

NOTE 2: Tests will run using the local domain specified by the STREAMS_DOMAIN_ID env var. All tests run in Distributed mode.

Create them by hand before or make sure that they are automatically created with the required number of partitions.

Hint: The test case kafkaStartPositionTest requires that the position topic is re-created before every test run.

Each sample contains a build.gradle file and a Makefile. The samples can be built/compiled by running ../../gradlew build or make from the sample directory.