Disclaimer: We are working on a new release that treats Keys and Values of Kafka Messages just as the official Java Version does it. Please be patient. It will also allow to define partitions and keys during pipes to output topics.

npm install --save kafka-streams

const {KafkaStreams} = require("kafka-streams");

const config = require("./config.json");
const factory = new KafkaStreams(config);

const kstream = factory.getKStream("input-topic");
const ktable = factory.getKTable(..);

kstream.merge(ktable).filter(..).map(..).reduce(..).to("output-topic");

• Quick Start
• API Info
• Documentation
• Operator descriptions
• Examples
• Native Client | SSL, SASL, Kerberos

• node-kafka-connect
• node-schema-registry
• node-kafka-rest-ui

• Prerequisites
• Aim of this Library
• Description
• Port Progress Overview
• Operator Implementations
• Additional Operators
• Stream Action Implementations
• Join Operators Status
• Window Operations
• FAQ - More

• kafka broker should be version >= 0.9.x, suggested: >= 0.10.x
• nodejs should be version >= 6.10, suggested: >= 8.6.x

• this is not a 1:1 port of the official JAVA kafka-streams
• the goal of this project is to give at least the same options to a nodejs developer that kafka-streams provides for JVM developers
• stream-state processing, table representation, joins, aggregate etc. I am aiming for the easiest api access possible checkout the word count example

kafka-streams 🐙 equivalent for nodejs ✨🐢🚀✨ build on super fast 🔥 observables using most.js 🤘

ships with sinek 🙏 for backpressure

comes with js and native Kafka client, for more performance and SSL, SASL and Kerberos features

the lib also comes with a few window operations that are more similar to Apache Flink, yet they still feel natural in this api :squirrel:

overwriteable local-storage solution allows for any kind of datastore e.g. RocksDB, Redis, Postgres..

async (Promises) and sync stream operators e.g. stream$.map() or stream$.asyncMap()

super easy API

the lib is based on sinek, which is based on kafka-node's ConsumerGroups

• core structure
• KStream base - stream as a changelog
• KTable base - stream as a database
• KStream & KTable cloning
• complex stream join structure
• advanced joins see
• windows (for joins) see
• flink like window operations
• word-count example
• more examples
• local-storage for etl actions
• local-storage factory (one per action)
• KStorage example for any DB that supports atomic actions
• backing-up local-storage via kafka
• kafka client implementation
• KTable replay to Kafka (produce)
• stream for topic message production only
• sinek implementation
• backpressure mode for KafkaClient
• auto-json payloads (read-map/write-map)
• auto producer partition and keyed-message handling
• documentation
• API description
• higher join & combine examples
• embed native client librdkafka for more performance
• SSL
• SASL
• Kerberos

• map
• asyncMap
• constant
• scan
• timestamp
• tap
• filter
• skipRepeats
• skipRepeatsWith
• slice
• take
• skip
• takeWhile
• skipWhile
• until
• since
• reduce
• chainReduce
• forEach (observe)
• chainForEach
• drain
• _zip
• _merge
• _join
• _combine
• _sample
• throttle
• debounce
• delay
• multicast
• A description of the operators can be found here
• Missing an operator? Feel free to open an issue 👮

• mapStringToArray
• mapArrayToKV
• mapStringToKV
• mapParse
• mapStringify
• atThroughput
• mapWrapKafkaPayload
• mapToFormat
• mapFromFormat
• Want more? Feel free to open an issue 👮

• countByKey
• sumByKey
• min
• max
• Want more? Feel free to open an issue 👮

Operation description

• merge
• outerJoin
• innerJoin
• leftJoin

• merge
• outerJoin
• innerJoin
• leftJoin

• merge
• outerJoin
• innerJoin
• leftJoin

• window

Yes.

Probably, yes. 😄

Forks or Stars give motivation