Demonstrating Data Offloading for Digital Twins with Kafka - Project for 521290S Distributed Systems, Spring 2023, @unioulu

Clone the repository with its submodules

git clone --recurse-submodules https://github.com/husmen/Kafka4Edge.git
cd Kafka4Edge
# If cloned already, update submodules
git submodule update --init

Run the main and edge clusters: NOTE: ALL COMMANDS ARE FOR LINUX DISTRIBUTION

docker-compose -f docker-compose-main.yml up -d
docker-compose -f docker-compose-edge.yml up -d --build --force-recreate

Make sure kafka-connect has started by checking logs (it'll take 30/40 seconds)

docker logs kafka-connect

After kafka-connect has started, run the following command to configure mongoDB as the sink for our kafka cluster

curl -d @config/kafka_connect/connect-mongodb-sink.json -H "Content-Type: application/json" -X POST http://localhost:8083/connectors

Then check:

• Kafka Producer Web API on http://localhost:8000/docs.
• MongoDB Client on http://localhost:3000.
• Grafana on http://localhost:4000.

When opening grafana:

• Enter admin as username and password
• Go to Configurations -> Data-Source -> Add Data Source, then select prometheus
• Enter http://kafka4edge_prometheus_1:9090 in the URL, Click Save & Test

To bring everything down:

docker-compose -f docker-compose-edge.yml down
docker-compose -f docker-compose-main.yml down

• Kafka cluster configuration
• Kafka edge node configuration, to join main cluster
• Simple Producer
• Simple Consumer
• MQTT Source
• MongoDB Sink
• Prometheus Monitoring
• Grafana Visualization
• Multithreaded Producer(s), to simulate multi sensor nodes and larger data loads
• Benchmarking, more info
• Attach to ditto
• Distributed database