APIs that help in writing data from Spark and Spark Streaming to Kafka. To use this in a Spark application, use Cloudera mvn repos: https://repository.cloudera.com/artifactory/cloudera-repos and https://repository.cloudera.com/artifactory/libs-snapshot-local for SNAPSHOTs.

Latest release artifact info, on maven:

  repository: https://repository.cloudera.com/artifactory/cloudera-repos
  groupId: org.cloudera.spark.streaming.kafka
  artifactId: spark-kafka-writer
  version: 0.1.0

To write to Kafka from a Spark app in Scala:

Import this object in this form:

  import org.cloudera.spark.streaming.kafka.KafkaWriter._

Once this is done, the writeToKafka method can be called on the DStream object in this form:

  dstream.writeToKafka(producerConfig, serializerFunc)

where the producerConfig is a properties object that defines the configuration that needs to be passed to the Kafka producer, and serializerFunc is a function that takes the each element from the DStream and converts it into the format that has to be used to write to Kafka (the output of this method is passed as is to the Kafka producer).

To write an RDD to Kafka, there is an equivalent method, which can be used in the exact same way as above.

  rdd.writeToKafka(producerConfig, serializerFunc)

To write a Java DStream to Kafka:

  JavaDStreamKafkaWriter<String> writer = JavaDStreamKafkaWriterFactory.fromJavaDStream(instream);
  writer.writeToKafka(producerConf, new ProcessingFunc());

Equivalently, a Java RDD can be written to Kafka in the following format:

  JavaRDDKafkaWriter<String> writer = JavaRDDKafkaWriterFactory.fromJavaRDD(inrdd);
  writer.writeToKafka(producerConf, new ProcessingFunc());

ProcessingFunc must be an implementation of the org.apache.spark.api.java.function.Function class, and the producerConf is a Properties object containing the properties to be passed to Kafka Producer.

The current released version is 0.1.0, built against Apache Spark 1.3.0-cdh5.4.8 and
Cloudera's distribution of Apache Kafka 0.8.2.0-kafka-1.3.2