Chef Cookbook that downloads the crowdmob/kafka-s3-consumer repository from GitHub, builds, and runs it using monit. Confirmed working with Amazon OpsWorks.

This cookbook depends on the following:

• deploy: the base amazon deploy recipe at https://github.com/aws/opsworks-cookbooks/tree/master/deploy
• golang: the installation of go recipe at https://github.com/crowdmob/chef-golang
• monit: the monit package to ensure your server is running, and tries to restart it if not at https://github.com/crowdmob/chef-monit

Of course, it also depends on a Kafka server running. There's a helpful Kafka chef recipe if you don't have a Kafka server running somewhere already

• kafka: an installation of kafka recipe at https://github.com/crowdmob/chef-kafka

At this time, the golang cookbook mentioned doesn't dynamically choose the right binary at runtime, based on CPU. That means that it assumes a 64 bit ec2 instance, which is a large instance or better.

To deploy your app, you'll have to make sure 2 of the recipes in this cookbook are run.

1. kafka-s3-consumer::install should run during the setup phase of your node in OpsWorks
2. kafka-s3-consumer::run should run after kafka-s3-consumer::install, possibly in the Configure phase of your node in OpsWorks.

This cookbook relies on a databag, which you should set in Amazon OpsWorks as your Stack's "Custom Chef JSON", with the following parameters:

{
  "service_realm": "production",
  "deploy": {
    "YOUR_APPLICATION_NAME": {
      ...
      "env": {
        "AWS_ACCESS_KEY_ID": "YOUR_AWS_ACCESS_KEY_CREDENTIALS",
        "AWS_SECRET_ACCESS_KEY": "YOUR_AWS_SECRET_KEY_CREDENTIALS",
        "AWS_REGION": "us-east-1",
        ...
      },
      "kafka": {
        "topics": [ "YOUR_TOPIC_1", "YOUR_TOPIC_2" ],
        "max_message_size": 4096
      }
    }
  }
}

Here's a little more about the ones you have to fill in:

• YOUR_APPLICATION_NAME is what you named your app, in the "Apps" section of OpsWorks
• YOUR_AWS_ACCESS_KEY_CREDENTIALS should be gotten from Amazon AWS
• YOUR_AWS_SECRET_KEY_CREDENTIALS should also be gotten from Amazon AWS
• YOUR_TOPIC_1 and YOUR_TOPIC_2 is an array of kafka topics the consumer for this app should consume

This cookbook builds and runs a go webapp in the following way:

• The consumer.go source file from https://github.com/crowdmob/kafka-s3-consumer is built using go get . followed by go build -o /usr/local/kafka-s3-consumer consumer.go. That results in an executable of at /usr/local/kafka-s3-consumer
• A consumer.properties file is created using your databag and output at /etc/kafka-s3-consumer/kafka_s3_consumer_APPNAME.properties
• A kafka-s3-consumer-APPNAME-daemon shell script is created and placed in /usr/local/, which handles start and restart commands, by calling /usr/local/kafka-s3-consumer -c /etc/kafka-s3-consumer/kafka_s3_consumer_APPNAME.properties and outputting logs to /var/log/kafka-s3-consumer/kafka_s3_consumer_APPNAME.out
• A kafka_s3_consumer_APPNAME.monitrc monit script is created, which utilizes the kafka-s3-consumer-APPNAME-daemon script for startup and shutdown, and is placed in /etc/monit.d or /etc/monit/conf.d, depending on your OS (defined in the monit cookbook)
• monit is restarted, which incorporates the the new files.

Author:: Matthew Moore

Copyright:: 2013, CrowdMob Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.