Git Product home page Git Product logo

pulsar_workshop's Introduction


1. Overview

The goal of this GitHub repository is to demonstrate with concrete code examples of how Apache Pulsar can be used as a powerful and unified platform for common messaging and streaming processing use cases.

This repository is composed of a series of demo scenarios (or simply demos). Each demo is a self-contained unit that covers a complete, end-to-end messaging/streaming processing use case using a specific message processing protocol that is supported by Pulsar.

The majority of the demos are organized under the subfolder of scenarios, and they're then further grouped into following categories based on the underlying API and/or messaging protocols being used.

  • Native Pulsar API and protocol (native-pulsar)
  • JMS specification with Starlight for JMS (S4J) API (jms-s4j)
  • Kafka protocol with Starlight for Kafka (S4J) protocol handler (kafka-s4k)
  • AMQP protocol with Starlight for RabbitMQ (S4R) protocol handler (rabbitmq-s4r)

There are also a subset of the demos that are organized under another subfolder of spring-scenarios. These demos are written using the Java Spring Boot framework for Pulsar. Because this framework requires JDK17 as the minimum version requirement. This is different from the rest of the demos that run at JDK11. (Please see Build. Programs for more details)

1.1. Software Requirement

Running the scenario require the following software to be installed on your local computer:

  1. JDK 11 (for scenarios) and JDK17 (for spring-scenarios)
  2. curl utility

1.2. Build Demo Programs

Each demo has its own set of programs to showcase how message publishing/producing/sending or subscribing/consuming/receiving works in that particular demo. In order to run these demos, we first need to build their programs Please refer to the document of Build Scenario Programs for more details.

1.3. Deploy and Run the Demos

Each of the demos included in this repository has 3 standard bash scripts that can be found under the _bash sub-folder of each demo.

  • deploy.sh is used to deploy the Pulsar resources (tenant, namespace, topic, function, etc.) as needed in the demo.
  • runProducer.sh is used to publish/send/produce messages to one Pulsar topic
  • runConsumer.sh is used to subscribe/receive/consume messages from one Pulsar topic

We first need to run deploy.sh in order to deploy the Pulsar resources (tenant, namespace, topic, function, etc.) that are needed for a particular demo. The deploy.sh has the standard input command line input parameters; but they will deploy different sets of Pulsar resources that are unique to each demo. For more details, please see the document of Deploy Demos.

After that, we can either run runProducer.sh to publish/produce/send messages or run runConsumer.sh to subscribe/consume/receive messages. These 2 scripts take some common command line input parameters but may be slightly different from demo to demo. For the general description of running a demo, please see the document of Run Demos. The demo specific execution steps will be explained in details under each demo.

2. Demo List

2.1. Demos under scenarios subfolder

API Demo Name Description
Native Pulsar native-pulsar/message-enrichment Demonstrate how an IoT sensor reading raw data can be published and enriched in Pulsar before consumed by a consumer.
Native Pulsar native-pulsar/message-enrichment-avro Similar to the message-enrichment scenario. The only difference is this scenario demonstrates the message processing with AVRO/JSON schema instead of as pure text.
Native Pulsar native-pulsar/message-redelivery Demonstrate how message redelivery is handled in Pulsar.
JMS + S4J jms-s4j/p2p-basic Demonstrate basic JMS message sending and receiving with a JMS Queue using S4J API and Pulsar
JMS + S4J jms-s4j/pubsub-basic Demonstrate basic JMS message publishing and subscribing with a JMS Topic using S4J API for Pulsar
Kafka + S4K kafka-s4k/pubsub-basic Demonstrate basic Kafka message producing and consuming with a Kafka Topic with Pulsar's S4K protocol handler
RabbitMQ + S4R rabbitmq-s4r/pubsub-queue Demonstrate RabbitMQ message producing and consuming with a RabbitMQ Queue using S4R API and Pulsar
RabbitMQ + S4R rabbitmq-s4r/fanout-exchange Demonstrate RabbitMQ message producing and consuming with a RabbitMQ Fanout Exchange using S4R API and Pulsar
RabbitMQ + S4R rabbitmq-s4r/topic-exchange Demonstrate RabbitMQ message producing and consuming with a RabbitMQ Topic Exchange and Routing Key using S4R API and Pulsar
RabbitMQ + S4R rabbitmq-s4r/direct-exchange Demonstrate RabbitMQ message producing and consuming with a RabbitMQ Direct Exchange and Routing Key using S4R API and Pulsar

2.2. Demos under spring-scenarios subfolder

Spring Boot Demo Name Description
spb-pulsar-pubsub-basic Demonstrate basic message publishing and consuming using Spring boot for Pulsar framework

pulsar_workshop's People

Contributors

chrisjohnson16 avatar devinbost avatar patbstream avatar yabinmeng avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.