jlgc77 / normalizer Goto Github PK

Normalizer is a stream processing engine based on Kafka Streams. Normalizer does maps (stateless and statefull), flatmaps and filters by us. You only need to define a JSON stream where you specify the process logic and how the message are transformed. It allows you to normalize data streams from different sources to convert them to the same data schema! Normalizer offers us: scalability, fault tolerance, back-pressure, KV states (RocksDB) and full Kafka integration ... Try it now!!

It reads json messages from Apache Kafka and writes them back normalized to Kafka.

Some of its abilities are:

• Add fields.
• Select or simplify fields.
• Replace values.
• Join two or more values.
• Select max/min from an array of values.
• Classify numeric values
• Split one value into multiple dimensions
• Convert time formats.
• Convert field types.
• Calculate arithmetic values
• Rename fields.
• Filtering
• Transform one message into several
• Easy scalable (automatic discovering and load balancing).
• Check docs for more!

You can find the docs on the Docs Website

You can get started with Normalizer with this tutorial.

To build this project you can use maven tool.

If you want to build the JAR of the project you can do:

mvn clean package

If you want to check if it passes all the test:

mvn test

If you want to build the distribution tar.gz:

mvn clean package -P dist

If you want to build the docker image, make sure that you have the docker service running:

mvn clean package -P docker

1. Fork it
2. Create your feature branch: git checkout -b my-new-feature
3. Commit your changes: git commit -am 'Add some feature'
4. Push to the branch: git push origin my-new-feature
5. Create a new Pull Request