1. Overview
2. Module Description - What the module does and why it is useful
3. Setup - The basics of getting started with kafka
   • What kafka affects
   • Setup requirements
   • Beginning with kafka
4. Usage - Configuration options and additional functionality
5. Reference - An under-the-hood peek at what the module is doing and how
6. Limitations - OS compatibility, etc.
7. Development - Guide for contributing to the module

The kafka module for managing the installation and configuration of Apache Kafka

The Kafka module for managing the installation and configuration of Apache Kafka: it's brokers, producers and consumers.

Installs the Kafka package and creates a new service.

To successfully install Kafka using this module you need to have Apache Zookeeper already running at localhost:2181. You can specify another Zookeeper host:port configuration using the config hash of the kafka:broker class.

The default configuration installs Kafka 0.9.0.1 binaries with Scala 2.11

  class { 'kafka': }

If you want a Kafka broker server that connects to zookeeper listening on port 2181:

  class { 'kafka::broker':
    config => { 'broker.id' => '0', 'zookeeper.connect' => 'localhost:2181' }
  }

You can specify different Kafka binaries packages versions to install. Please take a look at the different Scala and Kafka versions combinations at the Apache Kafka Website

We first install the binary package with:

  class { 'kafka':
    version => '0.9.0.1',
    scala_version => '2.11'
  }

Then we set a minimal Kafka broker configuration with:

  class { 'kafka::broker':
    config => { 'broker.id' => '0', 'zookeeper.connect' => 'localhost:2181' }
  }

One of the primary classes of the kafka module. This class will install the kafka binaries

Parameters within kafka:

The version of kafka that should be installed.

The scala version what kafka was built with.

The directory to install kafka to.

The url where the kafka is downloaded from.

Install java if it's not already installed.

The directory to install kafka.

Package name, when installing kafka from a package.

Package version (or 'present', 'absent', 'latest'), when installing kafka from a package.

Create kafka group with this ID

Create kafka user with this ID

User to install kafka as. Defaults to the kafka user.

Group to install kafka as. Defaults to the kafka group.

Directory for kafka config files. Defaults to /opt/kafka/config.

Directory for kafka log files. Defaults to /var/log/kafka.

One of the primary classes of the kafka module. This class will install a kafka broker.

Parameters within kafka::broker:

The version of kafka that should be installed.

The scala version what kafka was built with.

The directory to install kafka to.

The url where the kafka is downloaded from.

A hash of the configuration options. All values are used in the server.properties file directly.

Install java if it's not already installed.

Sets the ensure state of the broker service to stopped or running.

Install the init.d service.

Whether the configuration files should trigger a service restart

The directory to install kafka.

One of the primary classes of the kafka module. This class will install a kafka consumer.

Parameters within kafka::consumer:

The version of kafka that should be installed.

The scala version that kafka was built with.

The directory to install kafka to.

The url where the kafka is downloaded from.

Install java if it's not already installed.

The directory to install kafka.

One of the primary classes of the kafka module. This class will install a kafka mirror.

Parameters within kafka::mirror:

The version of kafka that should be installed.

The scala version that kafka was built with.

The directory to install kafka to.

The url where the kafka is downloaded from.

Install java if it's not already installed.

The directory to install kafka.

One of the primary classes of the kafka module. This class will install a kafka producer.

Parameters within kafka::producer:

The version of kafka that should be installed.

The scala version that kafka was built with.

The directory to install kafka to.

The url where the kafka is downloaded from.

Install java if it's not already installed.

The directory to install kafka.

Defined type that creates Kafka topics.

• kafka: Guides the basic installation of kafka binaries
• kafka::broker: Guides the basic installation of a kafka broker
• kafka::consumer: Guides the basic installation of a kafka consumer
• kafka::mirror: Guides the basic installation of a kafka mirror
• kafka::producer: Guides the basic installation of a kafka producer

• [kafka::broker::config] Manages all the default configuration of the kafka broker application
• [kafka::broker::install] Manages the installation of the kafka packages
• [kafka::broker::service] Manages the kafka server service
• [kafka::consumer::config] Manages all the default configuration of the kafka consumer application
• [kafka::consumer::install] Manages the installation of the kafka packages
• [kafka::consumer::service] Manages the kafka server service
• [kafka::mirror::config] Manages all the default configuration of the kafka mirror application
• [kafka::mirror::install] Manages the installation of the kafka packages
• [kafka::mirror::service] Manages the kafka server service
• [kafka::producer::config] Manages all the default configuration of the kafka producer application
• [kafka::producer::install] Manages the installation of the kafka packages
• [kafka::producer::service] Manages the kafka server service

• kafka::topic Create kafka topics

This module is tested on the following platforms:

• CentOS 5
• CentOS 6
• Ubuntu 12.04
• Ubuntu 14.04

It is tested with the OSS version of Puppet (>= 4.7.0) only.

Please read CONTRIBUTING.md for full details on contributing to this project.