Simple Data Flow with Strimzi
Simple POC for running different programs on a Apache Kafka dataflow, through Strimzi.
Install Stimzi
Install Strimzi into your Openshift cluster. Could be done with Strimzi-APB...
Install the apps
We have three apps that simulate a flow of data through a cluster of Apache Kafka. Location of the Kafka cluster as well as the different names for the Kafka topics are injected into the application using Environment Variables.
On the shell, go to your Openshift project, containing the Strimzi installation, like:
oc project myproject
The Producer (JAX-RS Application with WF-Swarm)
The producer is a simple JAX-RS / CDI app, using kafka-cdi, and produces messages to a injected Kafka topic, based on some random data, after a RESTful endpoint is accessed. This could be an API gateway, for instance...
Get it up and running:
mvn clean fabric8:deploy
This kicks in the deployment to the current Openshift project, by using a S2i build.
A vertx Consumer
The above producer writes to two different topic. The verx consumers reads from one of these topics. Install the app like:
mvn clean fabric8:deploy -Popenshift
A FatJAR Consumer
A normal FatJAR is a different consumer, reading Records from a different topic. Run the app like:
mvn clean fabric8:deploy -Popenshift
See it in Action
Do oc get pods, this might give you something like:
NAME READY STATUS RESTARTS AGE
my-cluster-kafka-0 1/1 Running 0 24m
my-cluster-topic-controller-527886806-2lh6w 1/1 Running 0 23m
my-cluster-zookeeper-0 1/1 Running 0 27m
producer-1-wd7s2 1/1 Running 0 21m
producer-s2i-1-build 0/1 Completed 0 21m
raw-app-1-hmjmh 1/1 Running 0 19m
raw-app-s2i-1-build 0/1 Completed 0 19m
strimzi-cluster-controller-3119394011-tpvcx 1/1 Running 0 27m
vertx-kafka-consumer-1-b7f8x 1/1 Running 0 17m
vertx-kafka-consumer-s2i-1-build 0/1 Completed 0 18m
show the vertx logs:
Run oc logs -f vertx-kafka-consumer-1-b7f8x -c vertx
Show the FatJAR logs:
Run oc logs -f raw-app-1-hmjmh -c java-exec
Send some data using cURL
Do oc get route to see the route of the producer. Output is like:
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
producer producer-myproject.192.168.37.1.nip.io producer 8080 None
raw-app raw-app-myproject.192.168.37.1.nip.io raw-app 8080 None
vertx-kafka-consumer vertx-kafka-consumer-myproject.192.168.37.1.nip.io vertx-kafka-consumer 8080 None
Now run something like the below to genrate some traffic and see it flow through the system:
watch -n 0.25 curl http://producer-myproject.192.168.37.1.nip.io/hello
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent