gunnarmorling / awesome-opensource-data-engineering Goto Github PK

https://www.knime.com/

https://heron.incubator.apache.org/

its retired and no longer active, its good idea to remove it

I suggest adding the following:

1. Meltano by GitLab
2. dbt by Fishtown Analytics
3. Singer by StitchData (Talend)
4. Airbyte
5. ScyllaDB

https://gethue.com/

https://www.dremio.com/

Both offer just as much (if not more) features than Spark has. They should be grouped with Spark

I know this may come as a surprise for many people but Apache Kafka is not a messaging infrastructure. It is simpler to label it as such because it has been known this way since the project was born many years ago; but the reality is that today Kafka is more than that. Kafka is a distributed streaming platform.

A streaming platform has three key capabilities:

• Publish and subscribe to streams of records, similar to a message queue or enterprise messaging system.

• Store streams of records in a fault-tolerant durable way.

• Process streams of records as they occur.

Reference: https://kafka.apache.org/documentation/#gettingStarted

That means that Kafka should belong to the categories "Integration", "Messaging Infrastructure", and "Stream Processing" at the same time. Knowing that this would look odd on this list of open-source projects, I would suggest creating a new category called "Distributed Streaming Platforms" and put Kafka on it.

-- @riferrei