Git Product home page Git Product logo

selium's Introduction

Selium

Crates.io MPL2 licensed Build Status Audit Status

Selium is an extremely developer friendly, composable messaging platform with zero build time configuration.

Getting Started

Hello World

First, create a new Cargo project:

$ cargo new --bin hello-selium
$ cd hello-selium
$ cargo add futures
$ cargo add -F std selium
$ cargo add -F macros,rt tokio

Copy the following code into hello-world/src/main.rs:

use futures::{SinkExt, StreamExt};
use selium::{prelude::*, std::codecs::StringCodec};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let connection = selium::client()
        .with_certificate_authority("certs/client/ca.der")? // your Selium cert authority
        .with_cert_and_key(
            "certs/client/localhost.der",
            "certs/client/localhost.key.der",
        )? // your client certificates
        .connect("127.0.0.1:7001") // your Selium server's address
        .await?;

    let mut publisher = connection
        .publisher("/some/topic") // choose a topic to group similar messages together
        .with_encoder(StringCodec) // allows you to exchange string messages between clients
        .open() // opens a new stream for sending data
        .await?;

    let mut subscriber = connection
        .subscriber("/some/topic") // subscribe to the publisher's topic
        .with_decoder(StringCodec) // use the same codec as the publisher
        .open() // opens a new stream for receiving data
        .await?;

    // Send a message and close the publisher
    publisher.send("Hello, world!".into()).await?;
    publisher.finish().await?;

    // Receive the message
    if let Some(Ok(message)) = subscriber.next().await {
        println!("Received message: {message}");
    }

    Ok(())
}

For the purpose of testing, developing Selium, or otherwise jumping straight into the action, you can use the selium-tools CLI to generate a set of self-signed certificates for use with mTLS.

To do so, install the selium-tools binary, and then run the gen-certs command. By default, gen-certs will output the certs to certs/client/* and certs/server/*. You can override these paths using the -s and -c arguments respectively.

$ cargo install selium-tools
$ cargo run --bin selium-tools gen-certs

Next, open a new terminal window and start a new Selium server, providing the certificates generated in the previous step.

$ cargo install selium-tools
$ cargo install selium-server
$ cargo run --bin selium-server -- \
  --bind-addr=127.0.0.1:7001 \
  --ca certs/server/ca.der \
  --cert certs/server/localhost.der \
  --key certs/server/localhost.key.der

Finally, in our original terminal window, run the client:

$ cargo run

Running Benchmarks

Included in the repository is a benchmarks binary containing end-to-end benchmarks for the publisher/subscriber clients.

These benchmarks measure the performance of both encoding/decoding message payloads on the client, as well the responsiveness of the Selium server.

To run the benchmarks with the default options, execute the following commands:

$ cd benchmarks
$ cargo run --release

This will run the benchmarks with default values provided for the benchmark configuration arguments, which should produce a summary similar to the following:

$ cargo run --release

Benchmark Results
---------------------
Number of Messages: 1,000,000
Number of Streams: 10
Message Size (Bytes): 32

| Duration             | Total Transferred    | Avg. Throughput      | Avg. Latency         |
| 1.3476 Secs          | 30.52 MB             | 22.65 MB/s           | 1347.56 ns           |

If the default configuration is not sufficient, execute the following command to see a list of benchmark arguments.

$ cargo run -- --help

Next Steps

Selium is a brokered messaging platform, meaning that it has a client and a server component. Check out the client and server crates for more details.

We also have the user guide that includes all of this information and much more. Our Getting Started chapters will step you through the process of setting up a secure, working Selium platform in 5 minutes or less.

Contributing to Selium

We'd love your help! If there's a feature you want, raise an issue first to avoid disappointment. While we're happy to merge contributions that are in line with our roadmap, your feature may not quite fit. Best to check first.

selium's People

Contributors

adenh93 avatar dependabot[bot] avatar ltuch avatar petehayes102 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

selium's Issues

Internal pipelines

Allow clients to specify pipelines that are not linked to external sockets. For example, a client could setup a WASM module that publishes to a given topic. This does not require the client to maintain an open socket with the Selium server once headers are exchanged.

Encryption

As a user, I want all of my traffic to be encrypted between the client and the server, so that it cannot be read or manipulated by malicious third parties.

Acceptance criteria:

  1. Curve25519 is used as the encryption algorithm
  2. All packets sent between clients and brokers are encrypted
  3. Certificates are used to authenticate users against a namespace
  4. End-to-end encryption is not a goal for this feature

Compression

Piggy back on existing pluggable encoder architecture to deliver a few sane compression formats.

Subscriber groups

As a consumer, I want to round robin messages between an arbitrary number of consumers, so that I only receive an equal percentage of messages.

A group could operate like a pre-shared key that developers distribute to all members of a consumer group beforehand.

RUSTSEC-2020-0071: Potential segfault in the time crate

Potential segfault in the time crate

Details
Package time
Version 0.1.45
URL time-rs/time#293
Date 2020-11-18
Patched versions >=0.2.23
Unaffected versions =0.2.0,=0.2.1,=0.2.2,=0.2.3,=0.2.4,=0.2.5,=0.2.6

Impact

Unix-like operating systems may segfault due to dereferencing a dangling pointer in specific circumstances. This requires an environment variable to be set in a different thread than the affected functions. This may occur without the user's knowledge, notably in a third-party library.

The affected functions from time 0.2.7 through 0.2.22 are:

  • time::UtcOffset::local_offset_at
  • time::UtcOffset::try_local_offset_at
  • time::UtcOffset::current_local_offset
  • time::UtcOffset::try_current_local_offset
  • time::OffsetDateTime::now_local
  • time::OffsetDateTime::try_now_local

The affected functions in time 0.1 (all versions) are:

  • at
  • at_utc
  • now

Non-Unix targets (including Windows and wasm) are unaffected.

Patches

Pending a proper fix, the internal method that determines the local offset has been modified to always return None on the affected operating systems. This has the effect of returning an Err on the try_* methods and UTC on the non-try_* methods.

Users and library authors with time in their dependency tree should perform cargo update, which will pull in the updated, unaffected code.

Users of time 0.1 do not have a patch and should upgrade to an unaffected version: time 0.2.23 or greater or the 0.3 series.

Workarounds

A possible workaround for crates affected through the transitive dependency in chrono, is to avoid using the default oldtime feature dependency of the chrono crate by disabling its default-features and manually specifying the required features instead.

Examples:

Cargo.toml:

chrono = { version = &quot;0.4&quot;, default-features = false, features = [&quot;serde&quot;] }
chrono = { version = &quot;0.4.22&quot;, default-features = false, features = [&quot;clock&quot;] }

Commandline:

cargo add chrono --no-default-features -F clock

Sources:

See advisory page for additional details.

Messaging patterns

As a user, I want to compose multiple messaging patterns for interacting with other clients, so that different use cases can be handled.

Initial patterns for beta:

  • Publish/subscribe
  • Request/reply
  • Push/pull (request without the reply)

Message chunking

For messages larger than the maximum frame size, split into chunks for transmission.

Overhaul Client API to make full use of multiplexing

We want to make full use of QUIC's multiplexing by overhauling the client API to allow multiple Subscriber and Publisher streams to be opened against a single connection.

This will require a significant change to the API, as an example:

let connection = selium::client()
    .keep_alive(Duration::from_secs(5))?
    .with_certificate_authority("certs/ca.crt")?
    .connect("127.0.0.1:7001")
    .await?;

let mut stocks_pub = connection.publisher("/acmeco/stocks")
    .map("/acmeco/forge_numbers.wasm")
    .retain(Duration::from_secs(600))?
    .with_encoder(BincodeCodec::default())
    .open()
    .await?;

tokio::spawn({
    let mut stocks_pub = stocks_pub.clone().await.unwrap();
    async move {
        stocks_pub.send(StockEvent::new("APPL", 9.98)).await.unwrap();
        stocks_pub.finish().await.unwrap();
    }
});

stocks_pub.send(StockEvent::new("MSFT", 12.75)).await?;
stocks_pub.finish().await?;

let mut greetings_sub = connection.subscriber("/acmeco/greetings")
    .with_decoder(StringDecoder)
    .open()
    .await?;

while let Some(Ok(message)) = greetings_sub.next().await {
    println!("{message}");
}

greetings_sub.finish().await?;

Establish selium-std crate

Add selium-std crate, containing standard offerings for compression, client codecs, etc.

The library should be an optional dependency, enabled by the feature std. It will then be available to end-users through the selium::std::* namespace, e.g selium::std::compression::zstd.

The reasoning for selium-std being optional is that some users/organizations might prefer to use their own compression/encoding implementations, so they can trim down their binary size + compile times by not including the selium-std dependency.

Key/value tables

As a publisher, I want to opt-in to retaining messages as key/value pairs, so that they can be queried by consumers.

If a consumer subscribes to the key/value topic, they would receive new and updated pairs only. If a consumer requests a single key from a topic (i.e. request/reply), the topic will send just that key, or None if it doesn't exist.

Update benchmarks

The e2e benchmarks will need to be updated to add optional flags for compression, batching, chunking, etc.

Authorisation

As a user, I want to prevent unauthorised users from publishing or subscribing to my namespace, so that my data is secure.

Acceptance criteria:

  1. Only authorised users can publish or subscribe to any given namespace
  2. The authentication mechanism is secured so it cannot be sniffed

Message retention

As a user, I want to optionally retain messages for an arbitrary period, so that they can be replayed at a later time.

Acceptance criteria:

  1. Either a producer or consumer can opt in to message retention for the given topic
  2. Retention can be on the basis of memory size (e.g. "a 100mb buffer")
  3. Or, retention can be on the basis of time, with optional memory limits (e.g. "2 days of retention, up to 10gb buffer")

To do:

  • Standardise trait for file pointers

Add stream extensions to server

The server should support the following stream manipulations, powered by user-defined WASM:

  • Map
  • Filter
  • Join (i.e. join 2 or more topics)
  • Split (i.e. split a topic into two)
  • Fold

Integration tests

  1. Validate that multiple publishers and subscribers can produce and consume from the same topic
  2. Validate that subscribers do not get messages that they are not subscribed to

Adjust visibility of modules in Client library

All types publicly exported from modules are currently visible to library users. This is not ideal, as there are publicly visible types that are strictly related to implementation details, and are not important to the end user.

We can resolve this by adding the pub(crate) scope modifier to any types/functions/etc that do not need full user visibility, but still require public visibility to other modules in the selium crate.

Implement tree to track clients

  • Instead of adding clients directly to a topic, add publishers and subscribers to respective trees.
  • Implement a method for deduplicating the tree (i.e. if there are connections A => B and A => C, dedupe these into A => B + C).
  • Implement a method for spawning any unspawned futures.

New sink + stream types will need to be implemented to allow each stream to have multiple downstreams, and each sink to have multiple drains. topic.rs can be the basis for both of these structures.

Message replay

As a consumer, I want to optionally receive a stream of historical messages, so that I don't miss anything.

Acceptance criteria:

  1. A consumer must opt in to message replay
  2. The consumer can choose to replay all messages in the queue
  3. The consumer can choose to replay all messages from the given timestamp
  4. Or, the consumer can choose to replay all messages from the given message number

Update documentation

All documentation will need to be brought up to date to reflect all of the new features added in this milestone.

Delivery guarantees

As a user, I want to opt in to either "at least once", or "at most once" delivery, so that I have assurance that messages make it to their destinations.

Acceptance criteria:

  1. Both producers and consumers can opt in to either delivery guarantee
  2. A producer and a consumer may have opposing delivery guarantees for the same topic
  3. The default delivery guarantee is "at most once"

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.