suborbital / grav Goto Github PK

This issue provides visibility into Renovate updates and their statuses. Learn more

Rate Limited

These updates are currently rate limited. Click on a checkbox below to force their creation now.

• fix(deps): update module github.com/gorilla/websocket to v1.5.0
• fix(deps): update module github.com/nats-io/nats.go to v1.14.0
• fix(deps): update module github.com/twmb/franz-go to v1.4.2
• chore(deps): update actions/checkout action to v3
• chore(deps): update actions/setup-go action to v3

• Check this box to trigger a request for Renovate to run again on this repository

Libp2p is a networking stack and library for p2p protocols. It supports several transport, security, pubsub, and routing protocols. Grav could really benefit from supporting libp2p as a transport as it offers a lot of nice features from scratch that could help Grav scale in certain decentralized environments.

Leveraging a libp2p transport in Grav could also add service mesh discovery (#32) and a pubsub transport (#15) by design.

Despite all this, for the best of my knowledge libp2p has not been extensively deployed in cloud environments (traditionally less aggressive than p2p networks). Thus, my plan is to build a basic go-libp2p transport plugin for grav, evaluate how it performs compared to the existing transports, and understand if it is worth it supporting additional integration and embedding with libp2p protocols.

This could maybe also open the door to this? :)

It would be nice to add more complex Pod type that can handle multiple message types. Pod would be like Actor

When receiving a message via a transport, the transport should be able to ack it and show it's been sent to the local bus.

As a bonus, it would be great if a MessageTicket could have some kind of method to know acks have taken place? How this would work is unclear.

Grav should be able to use AWS SQS as a transport.

There should be a pod.Disconnect() that essentially "closes" it and signals to the bus that it should be removed on the next revolution of the ring.

Grav should be able to use AWS EventBridge as a transport.

Grav should be able to use Kafka as a transport.

Under high load, something like this can fail to find a reply:

pod.Send(jobMsg).WaitUntil(grav.Timeout(5), func(msg grav.Message) error { ... })

It's unclear why, as another connected pod can indeed find said reply.

Something to collect multiple pods, like in cases where you need to watch multiple message types and need to keep references to multiple pods.

Something like:

pods := g.PodGroup()

pods.New().On(...)
pods.New().OnType(...)

It would still be safe since pods retains the pod object so you don't have to if you don't want to.

Documentation is absent for these topics:

• Usage / Advanced
• HTTP Transport
• Websocket Transport
• Local Network Discovery

Other TODOs:

From Sending Messages:

Messages have a few other properties that can be taken advantage of, which will be discussed in a future section.

• Document other properties of Messages

Grav should be able to use websockets as a transport.

When using a non-default Message type, it's essential that incoming messages can be unmarshalled into the correct struct. User should be able to provide a function that, given the Message's Type(), returns an empty struct.

Something like UseMsgGenerator(func(msgTypeString) Message)

Grav should be able to use GCP PubSub as a transport.

Hey!

I have been digging into a memory leak with our product that uses Grav. During my search, I noticed ever-increasing memory consumption by github.com/suborbital/grav/grav.(*messageFilter).FilterUUID. Some further digging, has led to the following theory:

In pod.Send() p.FilterUUID(msg.UUID(), false) is called. In FilterUUID(), the UUID of the sent message is added to the UUIDMap map. This UUID seems to never be removed from the map again, leading to an increase in memory consumption every time pod.Send() is called.

I am happy to open an MR to address this, but my understanding of the internals of Grav are not super strong. So some guidance is certainly welcome!

When the failed message array for a podConnection gets to a certain (configurable) size, that pod should be removed from the ring.

Pods should also be able to disconnect themselves.

Kafka supports listing multiple bootstrap servers, the grav transport should support this as well.

https://pkg.go.dev/github.com/twmb/franz-go/pkg/kgo#SeedBrokers is a variadic function which will take n number strings.

Given this fact it should be as simple as introducing https://pkg.go.dev/strings#Split to https://github.com/suborbital/grav/blob/main/transport/kafka/transport.go#L62

Grav should be able to use RabbitMQ as a transport.

Hi Grav team. Is there any documentation or description of how messaging internally (in process) interacts with messaging using transports? For example do we just get one big bus or can we set filtering so that only a subset of messages move across the mesh/bridge?

I'm reading the source but I can't get a feel of how flexible the overall architecture of a Grav mesh might be, how topic hygiene, security, authentication and authorisation might be introduced, and so on. Apologies for the very general questions it would be great to have a clearer picture!

Grav should be able to discover nodes on the local network using multicast/broadcast.

I started playing around with this project a little bit and was wondering if there was any examples on how to setup a multi Grav instanced network?

I noticed the docs in this area was still under active work, so just wanted to reach out to see if there happened to be any working examples or anything I could play around with to get my head around some of this.

Thanks!

The ability to define static peers and have Discovery ensure ongoing connections to said peers would be useful.

Links to master should be changed to main.

Still an enthusiastic Go semi-novice but I was surprised that Grav offers both a non-blocking and a blocking message receive. I would have thought that the paradigm in Go would have been to offer only the blocking receive and to run that in a goroutine and then sync over a channel to avoid callbacks :)

Would love to know why both are offered and where you've used the async receive...

The HTTP transport uses a single pod for all incoming and outgoing messages, but more sophisticated transport plugins should use a seperate pod per connection. Both situations need to be supported.

Hi @cohix, thanks so much for this lovely clean flexible project. Is there anything conceptually that would prevent the use of MQTT as a bridge for Grav?

Allow nodes to discover one another by querying a service mesh's control plane

Having a MsgData interface would make it easier to create/send messages in one step, for example:

type MsgData interface {
    Data() []byte
    MsgType() string
}

Could then allow something like:

pod.SendData(structThatImplementsMsgData)

Grav should be able to use NATS as a transport.

Either the podConnection or the Pod itself should allow auto-filtering of messages via configurable filters.

This should be designed such that ResponseTo messages are easily implemented later using a list of expected responseTo UUIDs

With the transition to SE2, we'll no longer maintain Grav docs on the main docs site, and will instead move them to this repo.

Grav should be able to discover other nodes in a cloud provider using the provider's APIs.

The bus should buffer the last x number of messages (configurable) such that when a new pod connects, it can optionally request to have the latest message replayed.