Git Product home page Git Product logo

pony-kafka's Introduction

Pony-Kafka

Welcome to Pony Kafka.

This is a pure kafka client written in Pony. The goal is to eventually reach feature parity with the official kafka client.

Why

The main reason this exists is because the alternatives weren't necessarily going to be clean/easy to integrate into Pony and its normal "async everything" manner of working.

Building

You need ponyc to compile pony-kafka. This is currently tested with ponyc version master.

You also need the following (in addition to what is needed for Pony itself):

  • LZ4
  • Snappy
  • Zlib

For Ubuntu 16.04 or newer you can run:

sudo apt-get install libsnappy-dev liblz4-dev zlib1g-dev

For older Ubuntu you can run:

sudo apt-get install libsnappy-dev
cd /tmp
wget -O liblz4-1.7.5.tar.gz https://github.com/lz4/lz4/archive/v1.7.5.tar.gz
tar zxvf liblz4-1.7.5.tar.gz
cd lz4-1.7.5
sudo make install

For OSX you can run:

brew install snappy lz4

You can then build the pony-kafka tests by running:

ponyc pony-kafka

or the example performance application by running:

ponyc examples/performance

Current status

This is currently alpha quality software that still needs more work before it is production ready.

A quick summary of features:

Feature Description Status
Basic Consumer Ability to connect to kafka brokers and consume messages Implemented
Group Consumer Ability to do high level consumer failover like official kafka client Not Implemented
Producer Ability to connect to kafka brokers and produce messages Implemented
Leader Failover Ability to correctly recover from/react to kafka leader failover Partially Implemented
Compression Ability to use LZ4/Snappy/Zlib compression for message sets Implemented
Message Format V2 Ability to use message set format version 2 Not Implemented
Idempotence/Transaction Ability to use idempotence/transactions Not Implemented
Metrics Ability to collect metrics and provide reports of metrics periodically Not Implemented
Security Ability to use SSL/SASL/etc to secure connection to Kafka brokers Not Implemented
Message Interceptors Ability to intercept messages before produce and after consume to be able to modify them or extract metadata from them for monitoring and other purposes Not Implemented
Producer Batching Ability to batch produce requests for efficiency Implemented
Producer Rate Limiting Ability to limit number of outstanding produce requests Implemented
Throttling Ability to tell producers of data to slow down due to network congestion Implemented
Message Delivery Reports Report back to producers once Kafka has confirm message has been successfully stored Implemented
Logging Logging of what is happening/errors Partially Implemented
Error Handling Ability to gracefully handle errors (retry if possible; fail fast if not) Partially Implemented
Documentation Comprehensive documentation for developers using Pony Kafka and developers enhancing Pony Kafka Not Implemented
Testing Comprehensive test suite to confirm everything is working as expected Not Implemented

pony-kafka's People

Contributors

chaz6 avatar dipinhora avatar jonbrwn avatar jtfmumm avatar seantallen avatar

Watchers

 avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.