acrosa / kafka-rb Goto Github PK

Kafka 0.8 makes backwards-incompatible changes to the protocol.

I'm planning to start tackling the changes necessary to support 0.8. This includes supporting the metadata api and making a "smart" producer that has behavior equivalent to the scala producer.

Has anyone started down this path yet? I don't want to duplicate work.

My plan is to have the gem no longer support 0.7's protocol and bump the gems major? version so that the 0.x series can continue to be released with 0.7 support as long as people are still running the 0.7 series of the server. Does this seem reasonable?

Hey, could you please point to following libraries as they are those that are actively maintained:

• ruby-kafka - A Ruby client library for Apache Kafka, a distributed log and message bus. The focus of this library will be operational simplicity, with good logging and metrics that can make debugging issues easier.

• Racecar - A simple framework that integrates with Ruby on Rails to provide a seamless way to write, test, configure, and run Kafka consumers. It comes with sensible defaults and conventions.

• Karafka - Framework used to simplify Apache Kafka based Ruby and Rails applications development. Karafka provides higher abstraction layers, including Capistrano, Docker and Heroku support.

• Phobos - Micro framework and library for applications dealing with Apache Kafka. It wraps common behaviors needed by consumers and producers in an easy and convenient API.

I just wanted to suggest that you change the name of the send method in Producer. I'm just starting out with kafka so I don't know if naming it producer.push would be a good idea or perhaps producer <<. Naming a method "send" is problematic in Ruby because send is how you send arbitrary method calls to an object. Keeping a functioning send method around is pretty important, especially if someone using the library wanted to do some metaprogramming.

i will send you a pull request with specs and bugfix

Is it possible to just send the payload string to kafka broker instead of the whole Kafka::Message object ?
I've tried both 0.0.12 and 0.0.15 of kafka-rb.

My consumer shows this:

#<Kafka::Message:0x00000002392fb8 @magic=0, @payload="this is a log message", @checksum=348211236, @compression=0>
#<Kafka::Message:0x00000002392dd8 @magic=0, @payload="this is another log", @checksum=2885865363, @compression=0>
#<Kafka::Message:0x000000023929f0 @magic=0, @payload="and another", @checksum=3444544879, @compression=0>

... but I just want this, well, eventually a compressed version of this:

this is a log message
this is another log
and another

If I use the librdkafka_example (C program) the producer does send just the payload to kafka,
but I would prefer to use ruby so I can select random partitions to send to.

Ideas ?

Thanks,
Chris

Hi everybody,

I run into a problem where topic processing needs to start from very specific message. I can locate that message by iterating from the very beginning but it looks like that block offset is only available to me from in Consumer class. Any pointers or technic that can used in order to achieve this effect? Thanks.

When a message is encoded, it is assumed that the payload string can quietly be transcoded to binary (ASCII_8BIT). For strings containing characters with incompatible codepoints, this fails with the error "Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8". This example should clarify:

producer = Kafka::Producer.new(:host => "kafka1.companybook.no")
message = Kafka::Message.new("ümlaut")
producer.send(message) # raises Encoding::CompatabilityError

This is due to the way the Producer#encode concatenates the message payload with the binary-encoded magic and checksum.