rgeorgeoff / kafka-connector-skeleton Goto Github PK

This is a simple Kafka connector (formerly called a "copycat connector"). This repo is intended to be a simple skeleton, that you can copy/paste to get started with developing your own Kafka connector.

This repo contains a fork of the connect-file plugin from Kafka trunk. The code itself has not changed. However, I renamed the package, so that there is no chance that you will accidentally get the connect-file plugin from trunk.

This things specific to this repo are the build files, as well as the instructions in this README that tell you how to get started.

$ git clone https://github.com/wushujames/kafka-connector-skeleton connect-file
$ (cd connect-file && ./gradlew build)

1. Run zookeeper somehow.

   Here are instructions on how to run it using docker.
   $ docker run -d --name zookeeper -p 2181:2181 confluent/zookeeper
   

2. Download and run kafka 0.9.0 locally by following the instructions at http://kafka.apache.org/documentation.html#quickstart

   $ curl -o kafka_2.11-0.9.0.0.tgz http://www.us.apache.org/dist/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgz
   $ tar xvfz kafka_2.11-0.9.0.0.tgz
   $ cd kafka_2.11-0.9.0.0
   $ ./bin/kafka-server-start.sh config/server.properties
   

3. Run your connect-file-source plugin

   $ export CLASSPATH=/path/to/connect-file/build/libs/connect-file.jar
   $ kafka_2.11-0.9.0.0/bin/connect-standalone.sh connect-file/connect-standalone.properties  connect-file/connect-file-source.properties
   

4. Write stuff to test.txt (that is the that this connector will read from, as configured in connect-file-source.properties)

   $ echo `date` >> test.txt
   

5. Read the data out from the kafka topic named 'test' (that is the that this connector will write to, as configured in connect-file-source.properties)

   $ kafka_2.11-0.9.0.0/bin/kafka-console-consumer.sh  --zookeeper localhost:2181 --topic test
   {"schema":{"type":"string","optional":false},"payload":"Thu Oct 15 23:03:15 PDT 2015"}
   

6. Run your connect-file-sink plugin

   $ export CLASSPATH=/path/to/connect-file/build/libs/connect-file.jar
   $ kafka_2.11-0.9.0.0/bin/connect-standalone.sh connect-file/connect-standalone.properties  connect-file/connect-file-sink.properties
   

7. Check that the file-sink plugin has written the data to the file

   $ cat test.sink.txt
   Thu Oct 15 23:03:15 PDT 2015