Git Product home page Git Product logo

dsps's Introduction

DSPS - Durable & Simple PubSub

DSPS Banner

MIT License Server Test Codecov Go Report Card DockerHub saiya/dsps npm version


Intro

DSPS is a PubSub server that provides following advantages:

  • Durable message handling
    • No misfire, supports buffering & resending & deduplication
  • Simple messaging interface
    • Just curl is enough to communicate with the DSPS server
    • Supports JWT to control access simply
    • Also supports outgoing webhook

DSPS system diagram

Non goal

Note that DSPS does not aim to provide followings:

  • Very low latency message passing
    • DSPS suppose milliseconds latency tolerant use-case
  • Too massive message flow rate comparing to your storage spec
    • DSPS temporary stores messages to resend message
  • Warehouse to keep long-living message
    • DSPS aim to provide message passing, not archiving message

3 minutes to getting started with DSPS

# Download & run DSPS server
docker run --rm -p 3099:3000/tcp saiya/dsps:latest

#
# ... Open another terminal window to run following tutorial ...
#

CHANNEL="my-channel"
SUBSCRIBER="my-subscriber"

# Create a HTTP polling subscriber.
curl -X PUT "http://localhost:3099/channel/${CHANNEL}/subscription/polling/${SUBSCRIBER}"

# Publish message to the channel.
curl -X PUT -H "Content-Type: application/json" \
  -d '{ "hello": "Hi!" }' \
  "http://localhost:3099/channel/${CHANNEL}/message/my-first-message"

# Receive messages with HTTP long-polling.
# In this example, this API immediately returns
# because the subscriber already have been received a message.
curl -X GET "http://localhost:3099/channel/${CHANNEL}/subscription/polling/${SUBSCRIBER}?timeout=30s&max=64"

ACK_HANDLE="<< set string returned in the above API response >>"

# Cleanup received messages from the subscriber.
curl -i -X DELETE \
  "http://localhost:3099/channel/${CHANNEL}/subscription/polling/${SUBSCRIBER}/message?ackHandle=${ACK_HANDLE}"

Tips: see server interface documentation for more API interface detail.

Message resending - you have to DELETE received messages

You may notice that your receive same messages every time you GET the subscriber endpoint. Because DSPS resend messages until you explicitly delete it to prevent message loss due to network/client error.

The way to delete message depends on the subscriber type. For example, HTTP polling subscriber (used in above example) supports HTTP DELETE method to remove messages from the subscriber.

More Documentations

dsps's People

Contributors

dependabot[bot] avatar ma2gedev avatar saiya avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

dsps's Issues

Implement outgoing Webhook

  • Also change webhook config section from array to map with user-defined ID.
  • Document that no guarantee of ordering & high durability and advice to use subscription API if such needed.

Storage multiplexer should early return on long-polling

If there are two or more storages under multiplexer, one (or more) storage may lost data.

In such circumstance, if there are unread message on subscriber...

  1. The storage that lost data blocks long-polling until timeout
  2. Other storages that kept data returns unread messages immediately

In this case, multiplexer should return messages immediately.

Support prometheus for monitoring

  • Export golang standard expvar values (-> memstat)
  • Also export some information
    • Some runtime stat such as running goroutines
    • Cumulative count of each API error code (each variable represents code, value is map from endpoint type to count)
    • Cumulative count of each HTTP status codes (each variable represents HTTP status, value is map from endpoint type to count)
    • Current count of in-flight requests
    • Current count of in-flight Redis subscribers
    • Cumulative count of storage fatal errors (-> domain.IsStorageNonFatalError(error) == true)
  • Write document about monitoring

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.