Is there are roadmap available (how about Azure Event Grid)? about messagesilo HOT 7 CLOSED

Yes @Expecho , that's exactly what I thought of too, If you can channel the messages into a normal Service Bus Queue/Topic, then it will work (As soon as Message Silo also supports Event Grid)

from messagesilo.

Hi @Expecho!

Thanks for your interest!

There is no roadmap, as I develop the tool in my free time, but the plan was, to include what the community wanted.

I still have to look into how well Azure Event Grid fits into the picture, but it could be a good next candidate!

If I make any progress with this, I will definitely let you know!

from messagesilo.

Hello @Expecho !

I investigated the possibilities here, and publishing messages to the Azure Event Grid Topics and Domains from Message Silo not a big deal because these are MQTT queues.

The problem is: Event Grid has only one option for dead-lettering, as you mentioned the Storage Account (wich not a dead-letter-queue), and unfortunetly Message Silo is designed for consume messages from queues.

So I think, in the short term, its not an option to fully support this scenario.
But publishing to Event Grid Topics/Domains (as a Target) could be implemented.

from messagesilo.

Hi @berkid89 thanks for the response. I could vision a scenario in which a blob triggered function puts the dead letter blobs as messages in a queue so Message Silo can pick it up from there to republish the message. In that case Message Silo being able to publish to Event Grid Topics/Domains (as a Target) would be a great addition.

from messagesilo.

Hello @Expecho!

I introducted the Event Gird Target type:
https://github.com/MessageSilo/MessageSilo/wiki/02.-Entities-&-Concepts#target

So, now you are able to use Message Silo in your scenario, if you can copy the Storage file contents to a Queue. I tested with these kind of example deadletter messages:

[{
  "id": "13453454359",
  "eventType": "recordInserted",
  "subject": "myapp/vehicles/motorcycles",
  "eventTime": "2017-08-10T21:03:07+00:00",
  "data": {
    "make": "Ducati",
    "model": "Monster"
  },
  "dataVersion": "2.0"
}]

In your case the Message Silo configuration look something like this with an Inline enricher:

apiVersion: v1
kind: Connection
name: myQueue
type: Azure_Queue
queueName: example_queue
connectionString: "Endpoint=sb://..."
receiveMode: ReceiveAndDelete
enrichers:
 - myEnricher
target: myTarget
---
apiVersion: v1
kind: Enricher
name: myEnricher
type: Inline
function: "(x) => {
    return x.map(p => {
      if (p.data.make === 'Ducati') {
        return {
          ...p,
          data: {
            ...p.data,
            model: 'Forza'
          }
        };
      }
      return p;
    });
  }"
---
apiVersion: v1
kind: Target
name: myTarget
type: Azure_EventGrid
endpoint: "https://xy.westeurope-1.eventgrid.azure.net/api/events?api-version=2018-01-01"
accessKey: "XGvb..."

2 hints:

• This configuration deletes the messages automatically from the Queue, so make sure you have the massages in the Storage Account as well, if something goes wrong and need to resend them again.

• If you automated the dead-letter channeling from the Storage Account to the Queue, and the fixed messages still landing in the deadletter blob, it can cause an endless loop of messages, so if it happens stop your Message Silo configurations, with a siloctl delete, or trigger the "dead-letter file -> queue" process manually or schedule it (daily for example) to avoid this case.

I hope it helps :)

from messagesilo.

I'm closing because the Event Grid integration is basically done.
If you have comments or suggestions, please open a new ticket.

from messagesilo.

@berkid89 great, we are getting things ready to put it to work. I will let you know if anything comes up. Thanks for the great work!

from messagesilo.