outbox-pattern's Introduction

Implementing the Outbox Pattern

Blogs:

Dzone
Medium

Prerequisites

Building and Installing required tools.

1: Build the `custom-debezium-connect` component.

cd custom-debezium-connect
mvn clean install
docker build -t custom-debezium-connect .

2: Run the Docker-Compose under the project folder to install all the needed tools.

docker-compose up -d

Setting up Kafka topics and Kafka Connectors.

- Setting up needed Kafka Topics

docker exec -t kafka /usr/bin/kafka-topics \
      --create --bootstrap-server :9092 \
      --topic student_email_changed \
      --partitions 1 \
      --replication-factor 1
      
docker exec -t kafka /usr/bin/kafka-topics \
      --create --bootstrap-server :9092 \
      --topic student_enrolled \
      --partitions 1 \
      --replication-factor 1

- Linking the Debezium Kafka Connect With the Outbox Table

curl -X POST \
  http://localhost:8083/connectors/ \
  -H 'content-type: application/json' \
  -d '{
   "name": "student-outbox-connector",
   "config": {
      "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
      "tasks.max": "1",
      "database.hostname": "postgres",
      "database.port": "5432",
      "database.user": "user",
      "database.password": "password",
      "database.dbname": "studentdb",
      "database.server.name": "pg-outbox-server",
      "tombstones.on.delete": "false",
      "table.whitelist": "public.outbox",
      "transforms": "outbox",
      "transforms.outbox.type": "com.sohan.transform.CustomTransformation"
   }
}'

- Consuming Kafka Topics

docker exec -t kafka /usr/bin/kafka-console-consumer \
      --bootstrap-server :9092 \
      --from-beginning \
      --topic student_enrolled

docker exec -t kafka /usr/bin/kafka-console-consumer \
      --bootstrap-server :9092 \
      --from-beginning \
      --topic student_email_changed

Starting the `student-microservice`

mvn spring-boot:run

Service endpoints.

Create Student

curl -X POST \
  'http://localhost:8080/students/~/enroll' \
  -H 'content-type: application/json' \
  -d '{ 
    "name": "Megan Clark",
    "email": "[email protected]",
    "address": "Toronto, ON"
}'

Update Student

curl -X PUT \  
  'http://localhost:8080/students/1/update-email' \  
  -H 'content-type: application/json' \
  -d '{"email": "[email protected]"}'

outbox-pattern's People

Contributors

Stargazers

Watchers

outbox-pattern's Issues

Sending messages to kafka reliably

Problem, how do I know that messages will come to kafka? Even if debezium-connect can be temporary down.

Example
I have running all services. I create students in student app

HTTP POST http://localhost:8080/students/~/enroll

{ 
    "name": "Yan-1",
    "email": "[email protected]",
    "address": "Toronto, ON"
}

The message comes to kafka

{"schema":null,"payload":"{\"studentId\":4,\"name\":\"Yan-1\",\"email\":\"[email protected]\",\"address\":\"Toronto, ON\"}"}

Now, I stopped debezium connect. docker-compose stop debezium-connect.
Imagine a real world situation when some containers can go down.

Then I create a user.

HTTP POST http://localhost:8080/students/~/enroll

{ 
    "name": "Yan-2 ebezium is down",
    "email": "[email protected]",
    "address": "Toronto, ON"
}

The student record is created in the database. However, there is not kafka message.

Performance Benchmark

Hi, thanks for this repo. I am curious about the performance. How much of a difference in the time can we expect to see in our messages being sent to Kafka with debezium; is it negligible?

Recommend Projects