Queues (for communications and other) about govflow HOT 5 CLOSED

@noamoss in our context, queuing systems are used to manage asynchronous jobs that do not need (for various types of "need" - performance, application semantics, etc.) to be dispatched and returned in the request/response cycle of a web app.

Dispatching emails is a classic example of this. Dispatching them in a request/response cycle means the web app has more processing to do before it can respond back to the client (especially true in languages like python).

So, what is commonly done is to dispatch a "job" (a job might be a function and its arguments), put it "somewhere", and call it "later", allowing the response to return without calling the job. This design has additional benefits like the ability to retry the job if there is a failure, to set dependencies across jobs etc.

In GovFlow, because Node is an event-driven web server (it is asynchronous by design), we do the "dispatch" part using some node functionality, but, we don't get the benefit of the ability to re-run jobs, inspect job health, etc.

A concrete example is, if we try to dispatch an email and the receiving server is temporarily down, we don't know and we can't try again later.

A job queue will allow us to manage such tasks, which run run the background of the web server, more robustly.

from govflow.

@noamoss @amirozer I looked into it last week and it seems this is probably the best solution for this in node:

https://github.com/OptimalBits/bull

@amirozer if you have other choices that are in use, let me know.

from govflow.

@pwalsh not sure I understand the core of this issue, can you elaborate a bit?

from govflow.

So, SendGrid has a send_at functionality - and it turns out that Twilio does too. This basically solves our whole issue:

• we dont need to manage our own queue or schedule - they do it
• we just dispatch message tasks to those backends as we normally do
• we do need to update our communication model like in the previous comment to ensure we are tracking what got send properly (enough), and save some rendered message data for potential debugging purposes

from govflow.

This is now done in ba32772 by using the functionality of our service providers Twilio and SendGrid.

from govflow.