Git Product home page Git Product logo

tc-notifications's Introduction

TOPCODER NOTIFICATIONS

Description

This repository hosts the API and processors for enabling notifications in various topcoder apps. Currently it is limited to provide this facility to the Connect application. Theoretcially, it is a generic framework and application which could be used for sending and consuming notificaitons by any other topcoder app. In very simple words to send notifications using this application:

  1. Send an event to bus
  2. Listen that event in tc-notifications
  3. There is a config in tc-notifications for each event it want to listen and we can specify rules who are the target users to whom we should send notifications for this event
  4. By default it saves all notifications which are generated after parsing the rules specified in step 3 and these are the treated web notifications which we show in the app directly
  5. Then there is option to add notification handlers where we get all these notifications one by one and we can process them further for more channels e.g. we send notification emails for each of notification generated
  6. When one wants to show the notifications, we use the notifications api (hosted inside the tc-notifications itself as separate service) to fetch notifications, mark notifications read or unread etc.

tc-notifications (as a standard nodejs app) provides generic framework around notifications, it does not have config (used in step 3 of previous list) for specific apps. So, to have config for specific apps, we have to start the notification consumers per app e.g. for connect, we have a folder connect which hosts start script to start notification consumers with connect specific configuration (events-config.js). It also adds email notification service which sends emails for notifications as per notification settings (coming from common framework laid by tc-notifications) for the user.

Steps needed to enable other apps to use the notifications are:

  1. Have a separate start up script (in a new folder at the root of the repo) for the concerned app. I would call this as notification consumer/processor.
  2. Write events-config.js (name is not important, we have to read this file in the start up script written in step 1) for app specific notifications
  3. Write additional notification services (eg. if you want to send email or slack or any other notification) and add them to startup script
  4. Specify a node script in package.json to launch the start up script written in step 1
  5. Either add deployment for this new notification consumer/processor in existing deployment script (if you want to host the processor as separate service in the same ECS cluster) or write a new script if you want to keep the deployment separate.

Dependencies

Configuration

Notification server

Configuration for the notification server is at config/default.js. The following parameters can be set in config files or in env variables:

  • General
    • LOG_LEVEL: the log level
    • PORT: the notification server port
    • DATABASE_URL: URI to PostgreSQL database
    • DATABASE_OPTIONS: database connection options
  • JWT authentication
    • AUTH_SECRET: TC auth secret
    • VALID_ISSUERS: TC auth valid issuers
    • JWKS_URI: TC auth JWKS URI (need only for local deployment)
  • KAFKA
    • KAFKA_URL: comma separated Kafka hosts
    • KAFKA_GROUP_ID: Kafka consumer group id
    • KAFKA_CLIENT_CERT: Kafka connection certificate, optional; if not provided, then SSL connection is not used, direct insecure connection is used; if provided, it can be either path to certificate file or certificate content
    • KAFKA_CLIENT_CERT_KEY: Kafka connection private key, optional; if not provided, then SSL connection is not used, direct insecure connection is used; if provided, it can be either path to private key file or private key content
  • Topcoder API
    • TC_API_V3_BASE_URL: the TopCoder API V3 base URL
    • TC_API_V4_BASE_URL: the TopCoder API V4 base URL
    • TC_API_V5_BASE_URL: the TopCoder API V5 base URL
  • Notifications API
    • API_CONTEXT_PATH: path to serve API on
  • Machine to machine auth0 token
    • AUTH0_URL: auth0 URL
    • AUTH0_AUDIENCE: auth0 audience
    • TOKEN_CACHE_TIME: time period of the cached token
    • AUTH0_CLIENT_ID: auth0 client id
    • AUTH0_CLIENT_SECRET: auth0 client secret
    • AUTH0_PROXY_SERVER_URL: auth0 proxy server URL
  • Consumer handlers
    • KAFKA_CONSUMER_HANDLERS: mapping from consumer topic to handlers
  • Email notification
    • ENV: used to construct email category
    • ENABLE_EMAILS: whether to enable email notifications
    • ENABLE_DEV_MODE: whether to enable dev mode
    • DEV_MODE_EMAIL: recipient email used in dev mode
    • DEFAULT_REPLY_EMAIL: default reply email

Connect notification server

Configuration for the connect notification server is at connect/config.js. The following parameters can be set in config files or in env variables:

  • Topcoder API
    • TC_API_V3_BASE_URL: the TopCoder API V3 base URL
    • TC_API_V4_BASE_URL: the TopCoder API V4 base URL
    • MESSAGE_API_BASE_URL: the TopCoder message service API base URL
  • Topcder specific
    Also it has probably temporary variables of TopCoder role ids for 'Connect Manager', 'Connect Copilot' and 'administrator':
    • CONNECT_MANAGER_ROLE_ID: 8,
    • CONNECT_COPILOT_ROLE_ID: 4,
    • ADMINISTRATOR_ROLE_ID: 1
      Provided values are for development backend. For production backend they may be different. These variables are currently being used to retrieve above role members using API V3 /roles endpoint. As soon as this endpoint is replaced with more suitable one, these variables has to be removed if no need anymore.
    • TCWEBSERVICE_ID - id of the BOT user which creates post with various events in discussions
  • Machine to machine auth0 token
    • AUTH0_URL: auth0 URL
    • AUTH0_AUDIENCE: auth0 audience
    • TOKEN_CACHE_TIME: time period of the cached token
    • AUTH0_CLIENT_ID: auth0 client id
    • AUTH0_CLIENT_SECRET: auth0 client secret
  • Email notification service
    • ENV: environment variable (used to generate reply emails)
    • AUTH_SECRET: auth secret (used to sign reply emails)
    • ENABLE_EMAILS: if email service has to be enabled
    • ENABLE_DEV_MODE: send all emails to the DEV_MODE_EMAIL email address
    • DEV_MODE_EMAIL: address to send all email when ENABLE_DEV_MODE is enabled
    • MENTION_EMAIL: recipient email used for notifications.action.email.connect.project.post.mention event
    • REPLY_EMAIL_PREFIX: prefix of the genereated reply email address
    • REPLY_EMAIL_DOMAIN: email domain
    • DEFAULT_REPLY_EMAIL: default reply to email address, for example [email protected]
  • Slack api
    • SLACK_URL: slack api url to post messages
    • SLACK_BOT_TOKEN: slack bot user OAuth token
    • SLACK_NOTIFY: slack notification switch, set to 'true' to enable slack notifications.

Note that the above two configuration are separate because the common notification server config will be deployed to a NPM package, the connect notification server will use that NPM package, the connection notification server should only use API exposed by the index.js.

JWT Token Generation

JWT token can be generated using the script test/token.js, its usage: node test/token {user-id}. Then use the generated token to manage the user's notifications.

In the Postman bus API, the Post Connect event will create a Kafka event of project id 1936; In the Postman notification server API, the TC API - get project will get details of project id 1936, we can see the project has one member of user id 305384; so we can run node test/token 305384 to generate a token to manage notifications of the user of id 305384.

The generated token is already configured in the Postman notification server API environment TOKEN variable. You may reuse it during review.

Local deployment

  • for local development environment you can set variables as following:
    • AUTH_SECRET,VALID_ISSUERS can get from tc-project-service config
    • PORT=4000 because connect-app call this port by default
    • TC_API_V4_BASE_URL=https://api.topcoder-dev.com/v4
    • TC_API_V3_BASE_URL=https://api.topcoder-dev.com/v3
    • KAFKA_URL, KAFKA_CLIENT_CERT and KAFKA_CLIENT_CERT_KEY get from tc-bus-api readme
  • if you are willing to use notifications API which is hosted by the notifications server locally, you will need to use some patched tc-core-library-js module, which skips verification of user token. Because we don't know Topcoder AUTH_SECRET locally. So you can install this fork:
    npm i https://github.com/maxceem/tc-core-library-js/tree/skip-validation
    
    WARNING do not push package.json with this dependency as it skips users token validation.
  • start local PostgreSQL db, create an empty database, update the config/default.js DATABASE_URL param to point to the db
  • install dependencies npm i
  • run code lint check npm run lint
  • init DB npm run reset:db
  • start connect notification server npm start
  • the app is running at http://localhost:4000, it also starts Kafka consumer to listen for events and save unroll-ed notifications to db

Heroku deployment

  • git init
  • git add .
  • git commit -m 'message'
  • heroku login
  • heroku create [application-name] // choose a name, or leave it empty to use generated one
  • heroku addons:create heroku-postgresql:hobby-dev
  • note that you may need to wait for several minutes before the PostgreSQL database is ready
  • optionally, to set some environment variables in heroku, run command like: heroku config:set KAFKA_CLIENT_CERT=path/to/certificate/file heroku config:set KAFKA_CLIENT_CERT_KEY=path/to/private/key/file heroku config:set KAFKA_GROUP_ID=some-group etc.
  • git push heroku master // push code to Heroku

Verification

  • start the app following above sections
  • note that if you use the Heroku app, the app may be in sleep after some long idle time, you need to call any Postman test, e.g. the listNotifications test, so that the app wakes up, during wake up, the Heroku PostgreSQL database will be cleared and re-initialized
  • in Postman, using the bus API collection and environment, run the POST /events / Post event - XXX tests, you may run it multiple times to create multiple events in Kafka, then you may watch the console output in the app, it should show info about handling the events
  • in Postman, using the notification server API collection and environment, run the tests

Swagger

Swagger API definition is provided at docs/swagger_api.yaml, you may check it at http://editor.swagger.io.

tc-notifications's People

Contributors

akumar1503 avatar architectt1 avatar eisbilir avatar gondzo avatar gunasekar-k avatar maxceem avatar mfikria avatar mtwomey avatar nkumar-topcoder avatar passimm avatar prakashdurlabhji avatar rashmi73 avatar rishirajsahu avatar sachin-maheshwari avatar yoution avatar zjuasmn avatar zsudraco avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

tc-notifications's Issues

Add confirm prompt for "npm run reset:db"

A command npm run reset:db to recreate DB schema has been added, so we don't need to uncomment code here or here during local development to create a DB schema.

Though there is a risk of running this command in the production environment. To make it safer we can do the next thing:

  1. When we run this command npm run reset:db we should show a prompt like Are you sure you want to recreate the DB "<DB_NAME>" schema on "<DB_HOST>" by user "<DB_USER_WITH_STARS>" and password "<DB_PASSWORD_WITH_STARS>"? This would REMOVE all existent data if any.

    • A user should type yes and press Enter to confirm. Typing anything else should cancel the command.
    • <DB_USER_WITH_STARS> should show the user, but only first 2 and last 2 characters visible, the rest are replaced by *. For example, instead of coder show co*er.
    • <DB_PASSOWRD_WITH_STARS> should show the password, but only first 1 and last 1 characters visible, the rest are replaced by *. For example, instead of mysecretpassword show m**************d.
  2. We also have to be able to run the command npm run reset:db using scripts automatically without the participation of a human. For such cases, we have to support an optional argument --yes, so the command could be run like npm run reset:db -- --yes. If the command run this way:

    • don't ask for confirmation

[$60] Database migrations

We need to create migrations for the database.
Initial db state should be the one before we added version column to notifications table, and the first migration should be adding the version column. We'll use this when releasing the next version to prod

Bundling support per event group

@gondzo we might need the support of bundling the notification emails per event group i.e. user should be able to configure separate bundling settings for Posts and replies, Team Changes etc. I think adding one column (may be named as Bundled) just like web and email which would again be a switch to determine if user wants to enable the bundling for that particular event group. OR just replicate the bundled settings row for each event group.

It needs more discussion before getting started.

fyi @acshields @vic-topcoder

Unit testing

This project doesn't have unit testing setup at the moment.

It would be good to have unit tests for this project because:

  • This project is not easy to fully setup up locally, so having unit tests would help to validate fixes for small issues without the necessity of the full local project setup.
  • The project is growing and adding more code may lead to breaking existent code (regression issues). And unit tests may help there.

Consumer stuck in unknown condition

Observed two times that service was up and running but there is no increment in corresponding DB row. That is only possible if consumer stop consuming the messages. Restarting the instance is work around solution.
We need to investigate the root cause.

Support for all email events

Right now, email notifications are enabled only for post events (topic created/updated, post created/updated). The scope here is to add support for the remaining events. Full list of supported events is available at

There are two modes of sending email notifications - immediate (as it happens, one by one) or bundled. Both are handled in https://github.com/topcoder-platform/tc-notifications/blob/dev/connect/notificationServices/email.js and the logic for the other events should be added there as well.

Individual notifications should always send projectId, project name, relevant user id, file name and url (only for File uploads), link name and url (only for New project links).
Bundle notifications should contain the same data as individual notifications, but they should be grouped by projectId and grouped by notification type - the grouping can be reused from the frontend app - https://github.com/appirio-tech/connect-app/blob/e77f7d0a7daf4efbc1301f943fd1c6a8394f0211/src/routes/settings/routes/notifications/components/NotificationSettingsForm.jsx#L21
Our email server does not support loops in email templates, so bundle emails will send the full html to the email service.

Phase updates

At the moment we send out phase updates to all users even though customers don't see draft phases in connect. We would need some logic to check which users should get the updates.

It would be great if we could share such checks with connect-app since we will have more and more such cases in the future. I'm not sure if having a small common package for permission checks would be helpful here ? any ideas @vikasrohit ?

[$60] Notifications from CoderBot are not created correctly

One of the notifications is being created in a wrong way when new event joan-26673.notifications.connect.project.post.created arrives. And the next INSERT query is performed:

INSERT INTO "Notifications" ("id","userId","type","contents","read","createdAt","updatedAt") VALUES (DEFAULT,'CoderBot','notifications.connect.project.post.created.v2','{"topicId":"1926","postId":6854,"postContent":"<p>test</p>\n\n<p>​</p>","userId":40152856,"projectId":"2372","toTopicStarter":true,"projectName":"test","userHandle":"pshah_manager","userFullName":"Parth Manager"}',false,'2018-01-06 09:26:35.973 +00:00','2018-01-06 09:26:35.973 +00:00') RETURNING *

It contains 'CoderBot' value for the column userId. But is has to contain a numeric userId instead. Because of this the query fails.

The whole log when event joan-26673.notifications.connect.project.post.created arrives is below.

info: Handle Kafka event message; Topic: joan-26673.notifications.connect.project.post.created; Partition: 0; Offset: 153; Message: {"topicId":"1926","postId":6854,"postContent":"<p>test</p>\n\n<p>​</p>","userId":40152856,"projectId":"2372"}.
debug: ENTER listNotifications
debug: input arguments
debug: { query: { read: 'false', limit: '1000' }, userId: 40152922 }
Executing (default): SELECT "id", "userId", "topic", "deliveryMethod", "value" FROM "NotificationSettings" AS "NotificationSetting" WHERE "NotificationSetting"."userId" = 40152922;
Executing (default): SELECT count(*) AS "count" FROM "Notifications" AS "Notification" WHERE "Notification"."userId" = 40152922 AND "Notification"."read" = false;
Executing (default): SELECT "id", "userId", "type", "contents", "read", "createdAt", "updatedAt" FROM "Notifications" AS "Notification" WHERE "Notification"."userId" = 40152922 AND "Notification"."read" = false ORDER BY "Notification"."createdAt" DESC LIMIT 1000 OFFSET 0;
debug: EXIT listNotifications
debug: output arguments
debug: output arguments
debug: { items: [], offset: 0, limit: 1000, totalCount: 0 }
debug: ENTER listNotifications
debug: input arguments
debug: { query: { read: 'false', limit: '1000' }, userId: 40153455 }
Executing (default): SELECT "id", "userId", "topic", "deliveryMethod", "value" FROM "NotificationSettings" AS "NotificationSetting" WHERE "NotificationSetting"."userId" = 40153455;
Executing (default): SELECT count(*) AS "count" FROM "Notifications" AS "Notification" WHERE "Notification"."userId" = 40153455 AND "Notification"."read" = false;
Executing (default): SELECT "id", "userId", "type", "contents", "read", "createdAt", "updatedAt" FROM "Notifications" AS "Notification" WHERE "Notification"."userId" = 40153455 AND "Notification"."read" = false ORDER BY "Notification"."createdAt" DESC LIMIT 1000 OFFSET 0;
debug: EXIT listNotifications
debug: output arguments
debug: output arguments
debug: { items: [], offset: 0, limit: 1000, totalCount: 0 }
Executing (default): INSERT INTO "Notifications" ("id","userId","type","contents","read","createdAt","updatedAt") VALUES (DEFAULT,'CoderBot','notifications.connect.project.post.created.v2','{"topicId":"1926","postId":6854,"postContent":"<p>test</p>\n\n<p>​</p>","userId":40152856,"projectId":"2372","toTopicStarter":true,"projectName":"test","userHandle":"pshah_manager","userFullName":"Parth Manager"}',false,'2018-01-06 09:26:35.973 +00:00','2018-01-06 09:26:35.973 +00:00') RETURNING *;
Executing (default): INSERT INTO "Notifications" ("id","userId","type","contents","read","createdAt","updatedAt") VALUES (DEFAULT,'40152856','notifications.connect.project.post.created.v2','{"topicId":"1926","postId":6854,"postContent":"<p>test</p>\n\n<p>​</p>","userId":40152856,"projectId":"2372","projectRole":"owner","projectName":"test","userHandle":"pshah_manager","userFullName":"Parth Manager"}',false,'2018-01-06 09:26:35.974 +00:00','2018-01-06 09:26:35.974 +00:00') RETURNING *;
error:  SequelizeDatabaseError: invalid input syntax for integer: "CoderBot"
    at Query.formatError (/Users/maks/dev/topcoder/72_tc_connect_bug_bash_06/tc-notifications/node_modules/sequelize/lib/dialects/postgres/query.js:356:16)
    at query.catch.err (/Users/maks/dev/topcoder/72_tc_connect_bug_bash_06/tc-notifications/node_modules/sequelize/lib/dialects/postgres/query.js:86:18)
    at tryCatcher (/Users/maks/dev/topcoder/72_tc_connect_bug_bash_06/tc-notifications/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/Users/maks/dev/topcoder/72_tc_connect_bug_bash_06/tc-notifications/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/Users/maks/dev/topcoder/72_tc_connect_bug_bash_06/tc-notifications/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/Users/maks/dev/topcoder/72_tc_connect_bug_bash_06/tc-notifications/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/Users/maks/dev/topcoder/72_tc_connect_bug_bash_06/tc-notifications/node_modules/bluebird/js/release/promise.js:689:18)
    at Async._drainQueue (/Users/maks/dev/topcoder/72_tc_connect_bug_bash_06/tc-notifications/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/Users/maks/dev/topcoder/72_tc_connect_bug_bash_06/tc-notifications/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues (/Users/maks/dev/topcoder/72_tc_connect_bug_bash_06/tc-notifications/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:789:20)
    at tryOnImmediate (timers.js:751:5)
    at processImmediate [as _immediateCallback] (timers.js:722:5)

Topic missing in Kafka

When starting tc-notifications and connecting to dev Kafka, there is a warning message that one topic doesn't exist - notifications.connect.project.post.mention

@sachin-maheshwari @vikasrohit can you check dev/prod Kafka and create it if necessary?

Improve Universal Notifications Payload

Background

At the moment to send email or web notification we require to userId and/or email address as per the Kafka Message payload format https://gist.github.com/maxceem/bb2ad68d2f8a790f01e3326f9f8015eb#file-notification-action-create-js-L9-L48

Though sometimes service which sends notification doesn't know userId/email and have just a handle for example. We don't want to ask each service to implement functionality to retrieve userId/email. Instead of this, we would allow passing only of the user identifier like userId, userUUID, email or handle and tc-notificatoin would have to get any details about the user necessary for sending notifications by itself.

Task

  1. For email the recipients array should allow passing the next object:

    { // only ONE of these fields is required (any of them can be passed)
       userId: 1212748193,
       userUUID: "12341234-1234-sdfsa-123fa-sadf",
       email: "[email protected]",
       handle: "maxceem"
    }
    • to send email tc-notification needs to know email so we have to get email if it's not passed by userId or userUUID or handle. If we could not get email - log error as we cannot send notification in such a case.
    • we also should getuserId which is needed to check user settings. So we have to use handle, email or userUUID to get userId. If user is not found, then still send notification to the email, this would be needed in case if user is not registered yet. But if there is some other error while getting userId (not NOT FOUND), then don't send email and log error as we failed to get userId.
    • The same logic should be supported for cc of the email. Thoug as we don't need to check setting for the user, then userId for cc is not need to retrieve.
  2. For web notification:

    • replace userId field with recipients wich would also be an array. And we would should send this web notification to each member in the recipients array.
      • recipients array should allow the same objects as for email:
         {  // only ONE of these fields is required (any of them can be passed)
            userId: 1212748193,
            userUUID: "12341234-1234-sdfsa-123fa-sadf",
            email: "[email protected]",
            handle: "maxceem"
         }
    • to send web notification we need userId, so we have to get it by userUUID or email of handle depend on what is provided.
    • if we could not get userId for any reason - log error as we cannot create web notification without userId.

API Endpoints

We have 2 kind of user ids in Topcoder which are provided by different services, though we keeping user in sync between them.

  • userId is a numeric user id like 13123123
  • userUUID is string UUID user id like 1323423421-asdfas-123asdf-123

For example, TC-Notifications services use only userId, while TaaS API always uses userUUID.

Get userId

  1. Get userId by userUUID - https://github.com/topcoder-platform/taas-apis/blob/feature/notifications-scheduler/src/common/helper.js#L1065-L1095 ( enrich can be false )
  2. Get userId by email - https://github.com/topcoder-platform/tc-project-service/blob/feature/new-milestone-concept/src/util.js#L909-L947
  3. Get userId by handle - https://github.com/topcoder-platform/tc-project-service/blob/feature/new-milestone-concept/src/util.js#L584-L608

Get email

  1. Get email by userUUID - https://github.com/topcoder-platform/taas-apis/blob/feature/notifications-scheduler/src/common/helper.js#L1065-L1095 ( enrich must be true to return emails )
  2. Get email by userId - https://github.com/topcoder-platform/tc-project-service/blob/feature/new-milestone-concept/src/util.js#L559-L579
  3. Get email by handle - https://github.com/topcoder-platform/tc-project-service/blob/feature/new-milestone-concept/src/util.js#L584-L608

API Notes

  • For most cases, we have endpoints that can get several users at once. So we can organize code in such a way that we would call such endpoints only once for all users in recipients array. (theoretically, we call it only once per Kafka message for all items in the payload, but it might be too many users at once, so calling it per notification object inside the payload array should be fine).
  • When getting something by userUUID we would need to create a separate M2M token specially for u-bahn like in TaaS API https://github.com/topcoder-platform/taas-apis/blob/feature/notifications-scheduler/src/common/helper.js#L889-L894

Unsubscribe link

In the current emails we have two links to unsubscribe - one at sendgrid and one at connect
image

@vikasrohit can we remove the sendgrid link? I'm not sure where in sendgrid is it configured

Escape ElasticSearch special characters when sending search request to member api

As of now, when whenever we making query for user via handle, we are not escaping any special character present in the handle which is causing troubles for fetching the user details for some of the handles e.g. any handle that has [ or ] in it which in turn causes 500 error from the member api.
Purpose of this ticket is to update the https://github.com/topcoder-platform/tc-notifications/blob/dev/connect/service.js#L178 and https://github.com/topcoder-platform/tc-notifications/blob/dev/src/common/tcApiHelper.js#L86 to escape the special character identified for the ElasticSearch.

fyi @maxceem let me know if we can handle this this week.

Figure out what to do with "old" notificaitons

We don't want them building up in the DB "forever".

Some thoughts:

  1. When a Connect project closes, delete all associated notificaitons
  2. Delete unacknowledged notifications after some long period of time (like 6 months or something...)

Bundling support for email notifications and email settings

Add support for bundling emails end email settings.

Notifications settings endpoints should be updated to save preferences for emails (same as what we are doing for web preferences now). Additionally, there should be a bundle setting for emails with options: no bundling, 10 minutes, 1 hour, 24 hours, weekly (these should be configurable levels).
When a notification is created, email should not be sent right away, but we should first check email settings and send it immediately if email bundling is off. Otherwise, the email for that notification will be sent later.

All bundle intervals should be handled via scheduled jobs. At each interval check for new notifications that require emails to be sent and send bundle email containing all notifications created since last sending the email.

@vikasrohit should we create a new email event in kafka and new template in sendgrid for this one?

Empty emails

I still occasionally receive emails with only project name with no actual notifications inside, ex
https://www.screencast.com/t/bTInb62Adh

@vikasrohit can you take a look at the email logs to check which events were sent there?

btw, maybe it would be a good idea to list all notification types covered by email templates and when sending an email log a warning or error if we run into a notification that doesn't have a template?

Separate task deployment for API and event bus consumers

This is to make the two portions decoupled and make consumers fault tolerant. As of now, if due to a runtime error consumers dies, ECS does not know that and never starts a new task for it. It causes missed notifications till we restart the process again.

Mentions are rendered as links in emails

User mentions are rendered as links in notification emails and they point to and invalid link like http://users/gondzo. Maybe it would be better to update draftjs mention plugin in connect app to not create links in the first place. Alternative is to remove the links in tc-notifications only.

Update links to post in email notifications

In the last release 2.4.13 we updated URLs for posts inside Connect App.

We should now update links inside post email notifications:

image

So far we are redirecting old links to the new links client-side so no functionality is broken.

Email templates support

Two tasks are in scope of this issue:

  • add email support for all event types (we support only topic/post events now)
  • add email templates support for all emails (this affects how we send the emails, so it's a good idea to complete this task first)

Emails for all event types

Right now, email notifications are enabled only for post events (topic created/updated, post created/updated). The scope here is to add support for the remaining events. Full list of supported events is available at

There are two modes of sending email notifications - immediate (as it happens, one by one) or bundled. Both are handled in https://github.com/topcoder-platform/tc-notifications/blob/dev/connect/notificationServices/email.js and the logic for the other events should be added there as well. Note that constructing email logic will be changes in the next task (send raw html instead of data placeholders)

Individual notifications should always display projectId, project name, relevant user id, file name and url (only for File uploads), link name and url (only for New project links).
Bundle notifications should contain the same data as individual notifications, but they should be grouped by projectId and grouped by notification type - the grouping can be reused from the frontend app - https://github.com/appirio-tech/connect-app/blob/e77f7d0a7daf4efbc1301f943fd1c6a8394f0211/src/routes/settings/routes/notifications/components/NotificationSettingsForm.jsx#L21
Also, multiple notifications of the same type should render just one notification. For example if we have two project.specificationModified events for the same project, we can display a notification like 'Project notification was updated by user1,user2' instead of two notifications 'Project notification was updated by user1' and 'Project notification was updated by user1'

Email templates support
Our current approach is to have email templates in email service (in Sendgrid actually) and we send only placeholder values data to the email service, but templating engine in email service doesn't support loops or conditional expressions, so we're changing the behavior - tc-notifications should generate the email html content and send it as only placeholder value to the email service.
Attached is the UI prototype for email templates. To get consistent email layout across all email clients, we need to inline all the css into html elements style tag. UI proto project has the css inliner support configured and it needs to be integrated into tc-notifications (see src/emails/one/one.template.html vs dist/one.template.html).
Email service template has the following content

<!DOCTYPE html>
<head>
    
</head>
<body style="background-color: #EBEBEB; font-family: Helvetica, sans-serif; margin: 0 auto !important;">
    <!-- container begings -->
    <table class="container" style="background-color: #EBEBEB; border-collapse: collapse; color: #151516; font-family: Helvetica, sans-serif; margin: 0 auto !important; padding: 0; width: 100%;">
        {{contents}}
    </table>
    <!-- container ends -->
    <!-- footer container begins -->
    <table class="footer-container" style="background-color: #EBEBEB; border-collapse: collapse; color: #151516; font-family: Helvetica, sans-serif; margin: 0 auto !important; padding: 0; width: 100%;">
        <tr class="empty-21" style="height: 20px; margin: 0; padding: 0;">
            <td style="margin: 0; padding: 0;">
                <table style="border-collapse: collapse; height: 20px; margin: 0; padding: 0;">
                    <tr style="margin: 0; padding: 0;"><td style="margin: 0; padding: 0;"></td></tr>
                </table>  
            </td>
        </tr>
        <!-- footer beings -->
        <tr class="footer" style="margin: 0; padding: 0;">
            <td class="footer-space" style="margin: 0; padding: 5px !important;"></td>
            <td class="footer-content" align="center" style="margin: 0; padding: 0; vertical-align: middle;">
                <table style="border-collapse: collapse; margin: 0; padding: 0; width: 400px;">
                    <tr style="margin: 0; padding: 0;"><td style="color: #808080; font-size: 11px; line-height: 15px; margin: 0; padding: 0; text-align: center; width: 400px;">Connect is the world’s first project management crowdsorcing platform utilizing the power of Topcoder communities</td></tr>
                    <tr class="empty-10" style="height: 10px; margin: 0; padding: 0;"><td style="color: #808080; font-size: 11px; line-height: 15px; margin: 0; padding: 0; text-align: center; width: 400px;"></td></tr>
                    <tr style="margin: 0; padding: 0;"><td style="color: #808080; font-size: 11px; line-height: 15px; margin: 0; padding: 0; text-align: center; width: 400px;">201 S Capitol Ave #1100</td></tr>
                    <tr style="margin: 0; padding: 0;"><td style="color: #808080; font-size: 11px; line-height: 15px; margin: 0; padding: 0; text-align: center; width: 400px;">Indianapolis, IN 46225 United States</td></tr>
                    <tr class="empty-10" style="height: 10px; margin: 0; padding: 0;"><td style="color: #808080; font-size: 11px; line-height: 15px; margin: 0; padding: 0; text-align: center; width: 400px;"></td></tr>
                    <tr style="margin: 0; padding: 0;"><td class="html-entities" style="color: #808080; font-size: 11px; line-height: 15px; margin: 0; padding: 0; text-align: center; width: 400px;">&#9679;&#9679;&#9679;</td></tr>
                    <tr class="empty-10" style="height: 10px; margin: 0; padding: 0;"><td style="color: #808080; font-size: 11px; line-height: 15px; margin: 0; padding: 0; text-align: center; width: 400px;"></td></tr>
                    <tr style="margin: 0; padding: 0;">
                        <td style="color: #808080; font-size: 11px; line-height: 15px; margin: 0; padding: 0; text-align: center; width: 400px;">You are receiving this email based on your Connect account preferences. To change which emails you receive from Connect, go to your <a href="#" style="color: #006DEA; font-size: 11px; line-height: 15px; text-align: center; text-decoration: none;">Notification Settings.</a>
                    </td></tr>
                </table>
            </td>
            <td class="footer-space" style="margin: 0; padding: 5px !important;"></td>
        </tr>
        <!-- footer ends -->
        <tr class="empty-20" style="height: 20px; margin: 0; padding: 0;">
            <td style="margin: 0; padding: 0;">
                <table style="border-collapse: collapse; height: 20px; margin: 0; padding: 0;">
                    <tr style="margin: 0; padding: 0;"><td style="margin: 0; padding: 0;"></td></tr>
                </table>  
            </td>
        </tr>
    </table>
    <!-- footer container ends -->
</body>

In tc notifications, we need to build only the content inside "body table.container", inline the css and send that as the contents placeholder value

Support "taas" platform

When retreiving notifications using endpoint GET https://api.topcoder-dev.com/v5/notifications/list?read=false&platform=connect&per_page=1000 we can filter notifications by the platform https://github.com/topcoder-platform/tc-notifications/blob/dev/src/services/NotificationService.js#L199-L206.

At the moment we only support 2 platforms: connect which retunrs only notification types which start from connect.notification. and community which returns all other notifications.

We need to support 3rd platfrom: taas which would return only notifications with type which stats from taas.notification.. Note, that community should be also updated to return all other notifications which are not connect and not taas.

Do NOT generate notifications for inactive users

This is with reference to #70. Although, we are handling it before sending emails, ideally we should not have generated notifications for inactive users. Purpose of this task is to stop generating notifications for inactive users. This would require us to query user details (status for now) when generating notifications.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.