govflow / govflow Goto Github PK
View Code? Open in Web Editor NEWAn open, modular work order and workflow management system for local government and residents.
Home Page: https://govflow.org
License: MIT License
An open, modular work order and workflow management system for local government and residents.
Home Page: https://govflow.org
License: MIT License
We have started with a standard webform but the goal is the support many inbound request channels.
We have a need now to support email, so, email is next!
We already depend on SendGrid for transactional email, so, we will build this feature using SendGrid's inbound parse API:
https://docs.sendgrid.com/for-developers/parsing-email/setting-up-the-inbound-parse-webhook
At a high level, this feature will require the following:
As per @amirozer
For example based on the current implementation which is just a simple, linear workflow:
inbox [open] | todo [open] | doing [open]| blocked [stalled] | done [closed] | invalid [closed] | moved [closed]
Repositories are a data access abstraction and specifically for us we are using them to provide access to the core entities of the system. There should be basically a 1:1 relation between a repository and an entity in our domain.
Services are for logic performs over multiple data entities, or external interactions, for example dispatch methods for email and sms.
Originally, communications dispatch methods were implemented on the communications repository. In 7f8a41b they are refactored into a service, and services are provided to the app via our registry which uses IoC. So, like Repositories, Services are also pluggable.
This is not in a release yet, I'm just recording the refactor here for reference.
cc @amirozer
We have a number of state changes to requests that do not have data about the change persisted.
For example: change status, change assignee.
We should have a record of every change made to a request, including what the change was, and who made the change.
We can do this in a simple manner by "abusing" the RequestComment data and writing additional events. This maps well with how we display such data in the UI we have.
For these new "automated" comments we will add as records of change, we can write them to a request comment using one or several templates.
As per #5 and specifically #5 (comment) we currently rely on the communication backends (Twilio and SendGrid) to manage unsubscribes, and Gov Flow is not aware of a given user's subscription status.
Next step around this feature would be one of:
I prefer the first option (less coupling with the service provider), but either should be fine.
Needs a bit more scoping to see what is best and when we do it.
Currently we just call sequelize sync to ensure the tables if they don't exist.
Umzug, the sequelize tool for programmatic migrations, is installed and loosely setup, but not working at present.
Unfortunately with sequelize we need to write up/down migrations by hand and maintain them in addition to model definitions.
And, because we support custom models we also need to support user provided migrations.
This is all possible with the tools we have, and we will implement it after a few more model changes are made to the current alpha codebase
In #26 ( implemented in #29 ) we added basic support for departments in a jurisdiction, being the ability to associate a service request with a department. As noted in #26 (comment) there are several ways this feature can develop based on input we have from current users and potential users.
The current implementation optionally allows assigning a request to a department. This assignment has no impact or relation to the person assigned to the request at present. The department itself does have a primary contact name and email as metadata, but this is not a StaffUser object, and it just for display purposes, not for any use in the assignment of requests.
We have a new potential user with key user scenarios that we can support by iterating on the initial base.
Our current understanding of the user scenarios is as follows:
Supporting this in the GovFlow API server requires:
ENFORCE_ASSIGNMENT_VIA_DEPARTMENT
ENFORCE_ASSIGNMENT_VIA_DEPARTMENT
is true
.Probable client changes required:
ENFORCE_ASSIGNMENT_VIA_DEPARTMENT
is true
require assigning a department before assigning a staff user, and, if department changes, clear user assignmentTBD
https://wiki.open311.org/GeoReport_v2/
https://github.com/rjsf-team/react-jsonschema-form
We will have lots of form configuration and this will allow us to hard code as little as possible.
Currently, repositories (data access abstraction) are pluggable, supporting user-provided plugins. Using the same internal infrastructure, next candidates to make pluggable are:
Other areas like input processing stacks for future iterations.
We have a clear set of use cases where Gov Flow users definitely want to use it as the backend for a general workflow management system, and not just as the workflow management system for public submitted 311 enquiries.
We need to consider how we model services for this purpose ( #30 ), and how this impacts the way data is read and analyzed.
Key use case now:
Internal requests - gov staff make requests of people from other gov departments, and the request types may be different to those that are publicly available.
When using Govflow as a package, the .txt
template files are missing, and messages cannot be sent
The Need
The Solution
Broken off from #66
Original in #66 we wrote:
When a Service Request is made via SMS, and, when a Service Request is submitted with Phone Number information, allow two-way communication around the Service Request with the Submitter.
Still deciding: Issue a dedicated Phone Number per Service Request - provides identical UX and data management flows to the current email integration, OR, have a "single" Phone Number, and a (more complex) disambiguation process to ensure two way communication is routed correctly to a given existing Service Request (will write about this a bit further below)
We since decided to implement a happy path where a Jurisdiction has a single Phone Number for Service Requests, and GovFlow manages a disambiguation flow to ensure incoming messages are understood correctly. Even if multiple phone numbers are in use for a Jurisdiction, via the creation of InboundMap
instances, we still run this disambiguation process (within the context of the matching InboundMap for a Number).
So, that disambiguation process is:
Jurisdiction
due to a matching InboundMap
, and optionally with deeper context (eg, a department) based on the InboundMap
configurationInboundMap
context, then, create a new Service Request (we disambiguated the incoming message without user involvement)InboundMap
context, then, create a new comment on the existing Service Request (we disambiguated the incoming message without user involvement)
To support the manual disambiguation flow, we need a new model, let's call it a MessageDisambiguation
. Here, we persist user messages and system responses, and probably references to existing objects like Service Requests, until the message from the user has been disambiguated. When it is disambiguated, we take the data in the MessageDisambiguation
object and create either a new Service Request or a new Service Request comment from it.
We now have clear user use cases for submitting requests by email.
As a first iteration, we will use an inbound email processing service provided by SendGrid, which we use for transactional messaging:
https://docs.sendgrid.com/for-developers/parsing-email/setting-up-the-inbound-parse-webhook
email addresses of users of Gov Flow can be configured to forward to email addresses we designate, and these email addresses will take and process the emails (in the simplest way) and then be ingested into Gov Flow and specifically to the request inbox where they can be further processed manually.
We've seen that some existing systems being used for 311-type requests allow "assigning" or "associating" a request with a government department, for example, "Public Works".
We want to model this, starting with a really simple use case, being, the ability to associate a ServiceRequest with a department.
This initial implementation will support the following scenario:
For each jurisdiction, we require:
The initial implementation essentially annotates a service request with a department from a closed list of departments for a given jurisdiction. This annotation can be used for user interface.
Implementing this requires the following changes to the code:
All of these would require additional logic and relations not present in the initial implementation.
We could use updatedAt, but, something might update after closed, so better that we add a closeDate
field and set it when the request state moves to a status that is of the closed type.
Users should be limited to accessing their associated jurisdiction's data, unless they have the access-all-jurisdictions
permission.
Currently, the Open 311 repositories for Service and Service Request are decoupled from the "main" Service and ServiceRequest repositories. Given they have similar logic, it might make sense to have the Open311 repositories depend on their "main" equivalents, rather than having their own logic and querying the database directly.
Not an urgent refactor but possibly good in the scenarios that the datastore for Open311, Service, or Service Repository is pluggable and outside the core system.
cc @amirozer
When a citizen submits a service request, sometimes she would like to be notified of the status of the request (let's say specifically, when it is resolved, but in general a notification could be sent based on a number of events in a service request lifecycle).
To support this, we can verify the citizen's contact channel (email or phone for sms) is valid. We can do this via Twilio/SendGrid ( see #5 ).
We can do this for:
Ok, I've been looking at the auth use case we have in our managed hosting environment (where authentication is handled outside the scope of GovFlow, and we just push a middleware in GovFlow to check that each request is authorized), and seeing what we can learn from that in terms of how to implement pluggable auth in GovFlow.
I think it is pretty simple, and breaks down to:
openid-connect
plugin by default, configured for Auth0 integration (ref.)Auth will have its own plugin type: AuthPlugin
AuthPlugin.login
: callable | null | undefined. If undefined, use the default implementation (passport.authenticate('openidconnect')
), if null, disable the login, if callable, use that as the passport.authenticate middleware on the login routeAuthPlugin.logout
: callable | null | undefined. If undefined, use the default implementation (req.logout
), if null, disable the login, if callable, use that as the passport.authenticate middleware on the logout routeAuthPlugin.verify
: callable | null | undefined. If undefined, use the default implementation (passport.authenticate('session')
), if null, disable the login, if callable, use that as the passport.authenticate middleware on all routes that require authenticationAt the moment, we attach additional settings, models and repositories to concrete implementations of repositories by adding a property after they are bound to the IOC container, and suppressing typescript errors. It works but it is neither good typescript nor good dependency injection.
Rather, we can get type safety and add these dependencies properly using dependency injection, either with factories for concrete implementations or by injecting constant values (models and settings are immutable by the time they are assigned to repositories).
Here is a description of an approach https://stackoverflow.com/questions/37439798/inversifyjs-injecting-literal-constructor-parameters (I like the "injecting the literal as a constant value" approach as it is simple)
Integrate with Twilio's API for subscription status events and record/track them as ChannelStatus
records - we will have an SmsStatusRepository
facade over Channel Status
just like we do for email
A bunch of related docs from Twilio:
Track delivery status and description and list of possible statuses
SMS Webhooks
Twilio opt-out keywords
We need to message with the public, and also with admin users, via email and optionally via SMS.
We will do this by integration with SendGrid and Twilio. Feature flags will prevent certain functionality if SendGrid and/or Twilio SMS is not configured.
We don't have a queue for dispatching tasks. Currently, the only area we really could use it is in dispatching email and sms messages via API to the backend providers. The initial implementation of that uses Node's event emitters to dispatch these jobs, but there is no management of such jobs like retry etc.
It would be good if we could have a simple queue framework that uses Postgres, to reduce the out of the box dependencies for the code base, but probably we'll go for a rabbit mq or redid backed queue framework.
Just like we currently support both inbound email for service requests, and, two-way communication around service requests, we are now implementing the same feature set and UX (to the extent reasonable) via SMS.
This integration is possible due to the extensive APIs that Twilio provides for SMS and phone number management (similar to how we leverage SendGrid, also a Twilio product, for email). Therefore, SMS support requires a direct dependency on Twilio - we are not designing an abstraction for potential use with other SMS backends.
At a high level, we plan to expose the following features:
ChannelStatus
model.Some links to docs on the inbound email, and two-way email, implementation:
We are currently proposing the following key modules for this system. Dropping this here for general knowledge, and we'll update as we go along.
People who submit service requests also need to optionally be able to submit images, to show the thing they are requesting to be addressed.
There are many ways to provide image support in the application: different ways to use cloud object storage, or upload to a filesystem, etc.
An ideal solution probably looks something like:
This pattern, or a variation of it, is pretty straight forward for file system storage, or AWS S3, and also I think minIO. Need to check if this pattern translates to Azure Blob Storage as we do want to support that (but we can also assume a minIO gateway over Azure Blob Storage as an option).
After some discussion with @amirozer it looks like the simplest, first solution will be:
This solution works probably works in the short term as the current client runs in a privileged environment with access to non-public image upload APIs.
The Client
object represents a jurisdiction (we may in future break this into a client can have many jurisdictions, for the county > many cities use case).
Clients
have one or many StaffUser
models, where each represents a user with admin rights (at this stage, we have no permission system for more complex rules). And, each StaffUser
only belongs to one Client
.
A StaffUser
logs in, and once verified, we therefore know both the user
and client
objects for each request (putting aside now if we store this in a session, with some other type of token like JWT, or, we pass the username and password in the header for each request to verify identity and therefore the user
and the client
for the request).
Currently, we are running an API that talks to an internal frontend, and that frontend can query other systems to get both the client
and the user
objects. So, the issue the for the near term is, what is the most direct way to do the integration, and we can build on this after to contain proper auth logic in this app.
Suggestion:
X-Zen311-Client
X-Zen311-Username
The user is already logged in so we do not need to auth again on the API, as we trust the execution environment for the short term.
From these headers, we query the database and set req.user
, which will have the full user object, including the client accessible at req.user.client
. This will be done in middleware here: https://github.com/pwalsh/zen311/blob/main/src/core/accounts/middlewares.ts#L3
The above approach should enable us to get working with the current frontend.
After this, once we are moving, we properly implement, via passport.js:
@amirozer WDYT?
It is not idiomatic TypeScript to prefix interfaces with I
(See here and here, an in general, any of the types from @types
), and so on creation of this codebase, I didn't carry over the usage of I
prefixed interfaces in general from the internal codebase it is based on.
As documented, I did retain I
prefixes for a special meaning - as a way to indicate interfaces that can be overridden by plugins.
Some recent additions used the I
prefix for non-pluggable interfaces. Let's discuss, so that the codebase remains consistent.
When building a project using 0.0.18-alpha, I'm getting the following error:
#8 16.85 npm ERR! syscall chmod
#8 16.85 npm ERR! path /app/node_modules/@govflow/govflow/cli/migrate
#8 16.85 npm ERR! errno -2
#8 16.86 npm ERR! enoent ENOENT: no such file or directory, chmod '/app/node_modules/@govflow/govflow/cli/migrate'
#8 16.86 npm ERR! enoent This is related to npm not being able to find a file.
#8 16.86 npm ERR! enoent
*Not sure whether this is an issue with the Govflow project/package or there is some adaptation required on my project to support this
What we did have:
Until 0.0.15-alpha
services were modeled in a parent-child hierarchy, the idea being that:
There were tests at the API endpoint and repository levels that showed serialization to/from open311 group to this parent-child data model was working (for some definition of working).
I came to this model from:
What we now have:
This was changed in 0.0.15-alpha
back to a model that directly follows the Open311 approach of a "flat" container group. I don't really know why - perhaps there was an edge case, or some behavior in the existing client that was not identical to the test cases we had.
Whatever the case, I think this highlights that we need to discuss and design the Service entity a bit to ensure we model it in a way that supports Open311 but is not restricted by it.
** Some thoughts:**
@amirozer @idoivri be great to get your thoughts here, and/or discuss this further in person.
In the core codebase, there are a bunch of commands in package.json
and also in the Makefile
- primarily based on my preference of using Makefiles as documentation. This approach is fine for developing the Gov Flow codebase itself.
But, the way these commands are implemented are useless for the primary use case of having Gov Flow as a dependency of a codebase.
We need a CLI that is exposed when Gov Flow is in your node_modules
, and also can be used when developing Gov Flow. Specifically, right now we need to be able to conveniently run database migrations.
In development I commonly run make initdb && make migrate && make test
. We could probably expose:
govflow initdb
govflow migrate
govflow serve # maybe ....
Spam is very likely to occur very early, and I’d suggest we integrate a form of human validation from the beginning for all anonymous users (so, all users as we are not having logged in users from the beginning). That can be one of:
The third option is the more complex, but, it also plays into our value proposition of facilitating communication between the city and citizens, and, when the requester is anonymous, that is the only way we know the details of the requester are valid (so we meet two needs - verify humanness and verify the human is contactable).
Two-way SMS requires some manual disambiguation message flows between the server and the submitter. See dfc6214 and https://github.com/govflow/govflow/blob/main/docs/two-way-email-and-sms.md
With the way this works now, if the submitter simply never responds to a disambiguation message, then, her original message never becomes a service request in any way (the request message IS persisted) in the MessageDisambiguationModel, though, with other data about the state of the disambiguation process.
Two ways (non-exclusive) that we could enhance this current implementation are:
The second option would involve a large amount of work at the UI level, and leverage the existing data flows that support the submitter disambiguation process.
The first option would be simpler and just require us to keep track of the # of times we sent a reminder.
We want to allow UIs to have editable service for a given request, as the public user who submitted a request may have miss-assigned, for example.
We need to add a new action endpoint, like with editing status or assignee, for editing service.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.