Git Product home page Git Product logo

camel-saga-quickstart's Introduction

Camel Saga with Kafka Streams

This quickstart demonstrates how to use the new saga feature of Camel 2.21 as well as using Kafka and Kafka Streams for storing and querying events.

Saga Kafka Demo

  • Saga pattern, compensating events, eventually consistent
  • Synchronous MSA, REST
  • CQRS, event driven async MSA
  • May receive out of order cancels
  • Topic partition auto by id
  • Long running action, narayana transaction manager
  • MSA springboot, fuse camel
  • Kafka + Kafka Streams - strimzi.io
  • Angular + node.js UI

Saga UI

Kafka as a Service running on Kubernetes and OpenShift

  • Install Kafka
  • Create Kafka Cluster

https://github.com/strimzi/strimzi#openshift

Kafka Architecture

Kafka Cluster Operator

Create Kafka Topics

Kafka Topic Operator

oc create -n strimzi -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: flights
  labels:
    strimzi.io/kind: topic
    strimzi.io/cluster: my-cluster
data:
  name: flights
  partitions: "1"
  replicas: "1"
  config: |-
    {
      "retention.bytes": "1073741824",
      "retention.ms": "86400000",
      "segment.bytes": "1073741824"
    }
EOF

oc create -n strimzi -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: trains
  labels:
    strimzi.io/kind: topic
    strimzi.io/cluster: my-cluster
data:
  name: trains
  partitions: "1"
  replicas: "1"
  config: |-
    {
      "retention.bytes": "1073741824",
      "retention.ms": "86400000",
      "segment.bytes": "1073741824"
    }
EOF

oc create -n strimzi -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: payments
  labels:
    strimzi.io/kind: topic
    strimzi.io/cluster: my-cluster
data:
  name: payments
  partitions: "1"
  replicas: "1"
  config: |-
    {
      "retention.bytes": "1073741824",
      "retention.ms": "86400000",
      "segment.bytes": "1073741824"
    }
EOF


oc create -n strimzi -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: trips
  labels:
    strimzi.io/kind: topic
    strimzi.io/cluster: my-cluster
data:
  name: trips
  partitions: "1"
  replicas: "1"
  config: |-
    {
      "retention.bytes": "1073741824",
      "retention.ms": "86400000",
      "segment.bytes": "1073741824"
    }
EOF
Kafka commands
oc exec -it my-cluster-zookeeper-0 -n strimzi -- bin/kafka-topics.sh --list --bootstrap-server=my-cluster-kafka-bootstrap:9092
oc exec -it my-cluster-zookeeper-0 -c zookeeper -n strimzi -- bin/kafka-topics.sh --zookeeper localhost:21810 --list

oc exec -it my-cluster-zookeeper-0 -n strimzi -- bin/kafka-console-consumer.sh --bootstrap-server=my-cluster-kafka-bootstrap:9092 --from-beginning \
    --topic flights \
    --formatter kafka.tools.DefaultMessageFormatter \
    --property print.key=true \
    --property print.value=true 

oc exec -it my-cluster-zookeeper-0 -n strimzi -- bin/kafka-console-consumer.sh --bootstrap-server=my-cluster-kafka-bootstrap:9092 --from-beginning \
    --topic trains \
    --formatter kafka.tools.DefaultMessageFormatter \
    --property print.key=true \
    --property print.value=true 

oc exec -it my-cluster-zookeeper-0 -n strimzi -- bin/kafka-console-consumer.sh --bootstrap-server=my-cluster-kafka-bootstrap:9092 --from-beginning \
    --topic payments \
    --formatter kafka.tools.DefaultMessageFormatter \
    --property print.key=true \
    --property print.value=true

oc exec -it my-cluster-zookeeper-0 -n strimzi -- bin/kafka-console-consumer.sh --bootstrap-server=my-cluster-kafka-bootstrap:9092 --from-beginning \
    --topic trips \
    --formatter kafka.tools.DefaultMessageFormatter \
    --property print.key=true \
    --property print.value=true

Deploy LRA Co-Ordinator

The upstream image may have issues. Try create a command in the template:

oc new-project saga --display-name='Saga Kafa LRA' --description='Saga Kafa LRA'
oc create -f lra-coordinator-template.yaml
oc new-app -lapp=lra-coordinator --template=lra-coordinator -e LOG_LEVEL=TRACE

Can deploy a manual build as well if required:

git clone [email protected]:eformat/lra-service.git
cd ~/git/lra-service/lra-coordinator
mvn clean package fabric8:deploy
oc create -f - <<EOF
{
  "apiVersion": "v1",
  "kind": "PersistentVolumeClaim",
  "metadata": {
    "name": "lra-data"
  },
  "spec": {
    "accessModes": [ "ReadWriteOnce" ],
    "resources": {
     "requests": {
        "storage": "1Gi"
      }
    }
  }
}
EOF
oc set volume dc/lra-coordinator --add --overwrite -t persistentVolumeClaim --claim-name=lra-data --name=lra-data --mount-path=/deployments/data

Deploy Core Saga Applications

oc project saga
cd ~/git/camel-saga-quickstart
mvn clean fabric8:deploy

Deploy UI Applications

UI Server
cd ~/git/camel-saga-quickstart/realtime_ui/server
oc new-build --binary --name=ui-server -l app=ui-server -i nodejs:8
oc start-build ui-server --from-dir=. --follow
-- -e KAFKA_URL
oc new-app ui-server
oc expose svc ui-server
UI Client
cd ~/git/camel-saga-quickstart/realtime_ui/client
oc new-build --binary --name=ui-client -l app=ui-client -i nodejs:8
oc start-build ui-client --from-dir=. --follow
oc new-app ui-client
oc expose svc ui-client
Istio

Deploy Istio to your cluster. Then Istio'ize your apps:

-- apps
oc apply -f <(istioctl kube-inject -f camel-saga-app/target/classes/META-INF/fabric8/openshift/camel-saga-app-deploymentconfig.yml)
oc apply -f <(istioctl kube-inject -f camel-saga-flight-service/target/classes/META-INF/fabric8/openshift/camel-saga-flight-service-deploymentconfig.yml)
oc apply -f <(istioctl kube-inject -f camel-saga-train-service/target/classes/META-INF/fabric8/openshift/camel-saga-train-service-deploymentconfig.yml)
oc apply -f <(istioctl kube-inject -f camel-saga-payment-service/target/classes/META-INF/fabric8/openshift/camel-saga-payment-service-deploymentconfig.yml)

-- lra build from source
oc apply -f <(istioctl kube-inject -f ~/git/lra-service/lra-coordinator/target/classes/META-INF/fabric8/openshift/lra-coordinator-deploymentconfig.yml)

-- lra built from template
oc create -f lra-coordinator-template.yaml
oc new-app -lapp=lra-coordinator --template=lra-coordinator -e LOG_LEVEL=TRACE -o yaml > /tmp/lra-coordinator.yml
oc apply -f <(istioctl kube-inject -f /tmp/lra-coordinator.yml) -n saga

TODO

  • Cassandra CDC
  • Istio (chaos, fallback, canary, slow consumers)
  • Full HA replicas + partitions
  • Compensating actions must succeed
  • CQRS blocking read instead if sliding window
  • Payload to json instead of strings
  • Single replica dtx manager (narayana) is not H/A in OCP - scaling still to be supported

References

camel-saga-quickstart's People

Contributors

eformat avatar nicolaferraro avatar

Stargazers

Tom Donohue avatar

Watchers

avatar James Cloos avatar avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.