Git Product home page Git Product logo

missing-children-society-canada / messaging Goto Github PK

View Code? Open in Web Editor NEW
5.0 11.0 6.0 784 KB

Data processing pipeline broken into microservices and run on Azure Functions. [Deprecated]

Home Page: https://mcsc.ca/

License: Apache License 2.0

JavaScript 100.00%
azure-functions azure node node-js azure-app-service azure-documentdb microsoft azure-storage service-bus azure-service-bus microservice event-sourcing

messaging's Introduction

Missing Children Society of Canada

Description

Data processing pipeline broken into microservices and run on Azure Functions. Message Enters Queue (from trigger) -> Filter checks to see if user is 'known' -> Structure is called to format messages for Topic -> Context is added during Augment -> Finally Data is Stored

  • Each Step the message is added to a queue for resiliance
  • System is designed as an Event Sourcing Model; with each step being additive.

Diagram Diagram

Filter

Ensures that only register users are tracked

Structure

Pulls out message property and adds them to brokered message so that Topic Subscriptions can filter on key values

Augment

Items

  • Facebook Media
  • Facebook Profile
  • Instagram Media
  • Instagram Profile
  • Twitter Media
  • Twitter Profile

Configuration (App Settings):

  • AzureWebJobsServiceBus: Service Bus connection string
  • DocumentDB: Cosmos DB connection string
  • DocDb_Host: Cosmos DB host
  • DocDb_AuthKey: Cosmos DB auth key
  • TwitterConsumerKey: Twitter consumer key
  • TwitterConsumerSecret: Twitter consumer secret
  • TwitterAccessTokenKey: Twitter access token key
  • TwitterAccessTokenSecret: Twitter access token secret
  • IG_TOKEN_VERIFY: Instagram verification token
  • APPINSIGHTS_INSTRUMENTATIONKEY: App Insights api key
  • SendGridAPIKey: SendGrid api key
  • NotifyEmailTo: email to
  • NotifyEmailFrom: email from
  • DashboardURL: dashboard url
  • DashboardProfileURL: profile page in dashboard

Store

All data is stored to Cosmos DB.

Notify

DESCRIPTION NEEDED

Notify Police

DESCRIPTION NEEDED

Validate Token

DESCRIPTION NEEDED

Quick start

messaging's People

Contributors

bgervin avatar bjeans avatar hashtagchris avatar joescars avatar m-gagne avatar nastassiar avatar tohling avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

messaging's Issues

Filter: Remove Cosmos DB Query

During the creation of the Filter Job, we needed to write a custom code to query and validate the user in the Cosmos DB; this can now be done in the binding. So this code should be removed from the 'actual' code and placed in the binding.

const query = 'SELECT * FROM c WHERE (c.twitter['$id'] = '' + message.userid + '' AND 'twitter' = '' + message.platform + '') OR (c.instagram['$id'] = '' + message.userid + '' AND 'instagram' = '' + message.platform + '') OR (c.facebook['$id'] = '' + message.userid + '' AND 'facebook' = '' + message.platform + '')';

By doing this, the user object should come in as a parameter for further validation.

Issues with historical data!

The historical data is not being pulled correctly here is the behavior I saw. This is all based on my newly registered profile that is registered with FB, twitter and instagram and has activity on each platform within the last 24 hours:

  1. When triggered from FB
  • The post itself is saved in the DB
  • NO historical data at all is saved in the DB
  • So in the portal nothing is shown under posts, but the triggered time is updated
  • Also the last platform used is wrong
  1. When triggered from Twitter
  • Nothing saved in the DB
  • Nothing happens
  • So the portal remains the same, no new posts and the triggered time is not updated
  1. When triggered from Instagram
  • The post is saved in the DB
  • FB Historical data is saved in the DB
  • So the portal shows posts for facebook and the last triggered time is updated correctly
  • Also the last platform used is wrong

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.