Git Product home page Git Product logo

dekaf's Introduction

Dekaf

A set of tools for working with Kafka in elixir.

Installation

def deps do
  [
    {:dekaf, "~> 0.1.0"}
  ]
end

Write a consumer

defmodule ThingConsumer do
  use Dekaf.Consumer

  def start_link(opts=%{context: %{consumer_id: consumer_id}}) do
    config = %{
      topics: opts.source, 
      client_id: consumer_id, 
      consumer_group_id: opts.name, 
      client_config: [
        {:bootstrap_servers, Application.get_env(:your_app, :bootstrap_servers)},
        {:statistics_interval_ms, 1000}, 
        {:stats_callback, __MODULE__}
      ], 
      topic_config: [
        {:auto_offset_reset, :smallest}
      ]
    }
    Dekaf.Consumer.start_link(__MODULE__, config)
  end

  def handle_message({:erlkaf_msg, topic, partition, offset, id, msg}, state) do
    ...
  end
end

Note: {:stats_callback, __MODULE__} is optional but if it is set the cluster status will be polled and written to ETS. This will make it easy to know when a rebalance event happens or which broker is the leader for a given partition.

Write a producer

defmodule Producer do
  use Dekaf.Producer

  def start_link(opts=%{context: %{producer_id: producer_id}}) do
    config = %{
      producer_id: producer_id, 
      topic: opts.mailbox,
      config: [
        {:bootstrap_servers, Application.get_env(:your_app, :bootstrap_servers)},
        {:statistics_interval_ms, 1000}, 
        {:stats_callback, __MODULE__}
      ],
    }
    Dekaf.Producer.start_link(__MODULE__, config)
  end
end

TODO:

  • Move the nif from :erlkaf into this project as an elixir nif.
  • Add seek and other methods provided by the kafka lib

dekaf's People

Contributors

jessiahr avatar

Stargazers

Ronnie Miller avatar Topher avatar  avatar

Watchers

James Cloos avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.