Git Product home page Git Product logo

aws-solutions / media-services-application-mapper Goto Github PK

View Code? Open in Web Editor NEW
84.0 27.0 27.0 55.29 MB

Media Services Application Mapper is a browser-based tool that allows operators to visualize the structure and logical connections among AWS Media Services and supporting services in the cloud. The tool can be used as a top-down resource monitoring tool when integrated with CloudWatch.

License: Apache License 2.0

Shell 2.62% Python 37.32% HTML 6.83% JavaScript 44.36% TypeScript 8.87%
medialive mediapackage media cloudfront s3 mediaconnect cloudwatch mediastore speke serverless

media-services-application-mapper's Introduction

πŸš€ Solution Landing Page | 🚧 Feature request | πŸ› Bug Report

Note: If you want to use the solution without building from source, navigate to Solution Landing Page

Table of contents

Solution Overview

  • AWS Media Services Application Mapper (MSAM) is a browser-based tool that allows operators to visualize the structure and logical connections among AWS Media Services and supporting services in the cloud.
  • MSAM can be used as a top-down resource monitoring tool when integrated with CloudWatch.
  • MSAM offers two different visualization options: Diagrams and Tiles.
  • MSAM can be configured to automatically display AWS Media Services alerts from AWS Elemental MediaLive channels and multiplex and AWS Elemental MediaConnect.

Go here for more information on MSAM's capabilities and features.

Installation Guide

Go here for more information on installing MSAM into your AWS account.

Architecture Diagram

You'll find the various architectural views for MSAM here.

Customizing the Solution

Prerequisites for Customization

  • Install/update to Python 3.x
  • Install the AWS Command Line Interface (CLI)
  • Create a Python virtual environment using requirements.txt and activate it
  • Configure the bucket name of your target Amazon S3 distribution bucket
export DIST_OUTPUT_BUCKET=my-bucket-name # bucket where customized code will reside
export SOLUTION_NAME=my-solution-name
export VERSION=my-version # version number for the customized code

Note: You would have to create an S3 bucket with the prefix 'my-bucket-name-<aws_region>'. aws_region is where you are testing the customized solution. Also, the assets in bucket should be publicly accessible.

Build

To build the distributable and prepare the CloudFormation templates:

cd deployment
chmod +x ./build-s3-dist.sh
./build-s3-dist.sh $DIST_OUTPUT_BUCKET $SOLUTION_NAME $VERSION

CloudFormation templates will be written to deployment/global-s3-assets.

Lambda binaries will be written to deployment/regional-s3-assets.

Unit Test

Execute this solution's unit tests manually with the following commands:

cd deployment
chmod +x run-unit-tests.sh
./run-unit-tests.sh

If no virtual environment is currently activate, this script will create and activate a temporary Python virtual environment capable of running the unit tests.

Deploy

Deploy the distributable to an Amazon S3 bucket in your account.

  1. From the deployment directory run the deploy.sh script.

Script usage:

./deploy.sh [-b BucketBasename] [-s SolutionName] [-v VersionString] [-r RegionsForDeploy] [-p AWSProfile] [-a ACLSettings(public-read|none)] [-t DeployType(dev|release)] 

Example usage:

 ./deploy.sh -b mybucket -s aws-media-services-application-mapper -v v1.8.0 -r "us-west-2 us-east-1 us-east-2" -p default -a public-read -t dev


All CloudFormation templates and lambda binaries will end up in:

s3://my-bucket-aws-region/solution-name/version/

If deploying with type release, CloudFormation templates will also be written to:

s3://my-bucket-aws-region/solution-name/latest/
  1. Get the link of the solution template uploaded to your Amazon S3 bucket.
s3://my-bucket-aws-region/solution-name/latest/aws-media-services-application-mapper-release.template

OR

s3://my-bucket-aws-region/solution-name/version/aws-media-services-application-mapper-timestamp.template
  1. Deploy the solution to your account by launching a new AWS CloudFormation stack using the link of the solution template in Amazon S3.

File structure

AWS Media Services Application Mapper consists of:

|- deployment
|   |- build-s3-dist.sh             [ Script for building distributables and preparing the CloudFormation templates ]
|   |- deploy.sh                    [ Script for deploying distributables and CloudFormation templates to user's S3 bucket ]
|   |- global-s3-assets             [ CloudFormation templates get written here during custom build ]
|   |- regional-s3-assets           [ Packaged code for Lambda get written here during custom build ]
|   |- run-unit-test.sh             [ Script for running unit tests ]
|- docs
|   |- ARCHITECTURE.md              [ 4+1 architectural views of MSAM ]
|   |- DEV_PROCESS.md               [ Describes the MSAM team's development process and environment ]
|   |- EXTENDING_MSAM.md            [ Instructions to extend MSAM with your own types ]
|   |- FEATURES.md                  [ Overview of solution features ]
|   |- INSTALL.md                   [ Installation guide for MSAM ]
|   |- MANAGED_INSTANCES.md         [ Using AWS Systems Manager and on-premise hardware ]
|   |- MEDIALIVE_WORKFLOW_WIZARD.md [ Monitoring worfklow created by MediaLive workflow wizard ]
|   |- RESOURCE_TAGS.md             [ Tagging resources for tile and diagram creation ]
|   |- REST_API.md                  [ Overview of the MSAM REST API and use ]
|   |- UNINSTALL.md                 [ Steps to remove MSAM from your AWS account ]
|   |- USAGE.md                     [ Getting started and usage tips for the browser tool ]
|   |- WORKSHOP.md                  [ Steps for a workshop presented at re:Invent 2019 ]
|   |- behavioral-views.drawio      [ diagrams.net source for behavioral view ]
|   |- deployment-view.drawio       [ diagrams.net source for deployment view ]
|   |- images                       [ Images used in documentation ]
|   |- logical-view.drawio          [ diagrams.net source for logical view ]
|   |- physical-view.drawio         [ diagrams.net source for physical view ]
|   |- use-cases.drawio             [ diagrams.net source for use case view ]
|- source
    |- events                       [ Source files for CloudWatch Event and Alarm handling ]
    |- html                         [ Source files for browser application ]
    |- msam                         [ Source files for the MSAM REST API and scheduled tasks ]
    |- tools                        [ Scripts used in the development of MSAM ]
    |- web-cloudformation           [ Source files for the web template and custom resources ]

License

See license here.

Collection of operational metrics

This solution collects anonymized operational metrics to help AWS improve the quality of features of the solution. For more information about this capability, please see the implementation guide.

Navigate

Navigate to Architecture | Workshop | Install | Usage | Uninstall | Rest API | Contributing

media-services-application-mapper's People

Contributors

amzn-gaod avatar clarkatamazon avatar dch90 avatar devonbleak avatar e-thoman avatar eggoynes avatar greglobrien avatar jimtharioamazon avatar joaquin6 avatar morjoan avatar roottool avatar sandimciin avatar tomgilman avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

media-services-application-mapper's Issues

Audit IAM permissions needed by custom resources

First off, MSAM is very impressive and it looks like it could be a great fit for our live-stream monitoring needs.

I have noticed that there are some fairly loose permissions on a couple of the stacks referenced in the install guide. This seems to be particularly true for the custom CloudFormation resources. I think it could be helpful to refine some of these permissions, especially since much of the code lives in S3 and can be difficult to audit.

The most concerning to me are the following:

  • The AdministratorAccess policy used by the MSAMCustomResourceRole in msam-browser-app-release.json
  • The AdministratorAccess policy used by the DefaultSettingsResourceRole in msam-dynamodb-release.json

Need a reaper on the back-end to remove old DynamoDB records

Items in the layout table should be removed once items in the content table expire. We can add a daily scheduled job to remove any items from layout that no longer exist in content.

Currently, there is no way to remove obsolete items from the layout table.

Frequent updates to informational layer make nodes flicker

Probably need a different way to update the informational layer so as not to trigger it too often, giving that flicker effect. I only see this on the local hosted app.
Code hosted on bucket doesn't seem to update information layer at all (might be addressed if informational layer is reworked).

Rework automated tests

Need to automate front-end and back-end tests
Invoke interactively from CLI
Invoke during automatic build

Integrate multi-diagram management code

Integrate classes and modules for managing multiple diagrams. Non-visible change; no UI changes. Global view diagram internally becomes a 'default' user defined diagram.

Diagram contents will come from Layout table!
Contents table will be used to feed upcoming inventory drawer.

Rework build system

  1. Nest existing CloudFormation templates into a single master template to simplify single-region installations
  2. Find solution to manual merge required between SAM and CF templates
  3. Add override parameters for base bucket name (currently rodeolabz)
  4. Add override parameters for deployment regions (currently all/most)

These are intended to simplify 3rd party contributions and build testing independent of core team.

Support discovery of on-premise equipment

One possibility is to use AWS Systems Manager as the cloud-side integration.

Install SSM agent on each on-premise host
Tag the host to help MSAM identify it ("MSAM-Type: Encoder", "MSAM-Region: us-west-2")
MSAM's inventory process queries SSM database for inventory with above tags

Change the front-end for specific rendering needs of on-premise nodes.

Support diagram definition by tag

Inventory elements with the tag "MSAM-Diagram=" will be used to define that diagram contents. That tag can occur multiple time per element so the element will appear on multiple diagrams.

Selection to render tiles as 'small' or 'regular'

Small tiles have only a name and status, and are intended for very large accounts with thousands of streaming channels.

Regular tiles are what we have today - name, resource counts, alarm and alert counts.

Allow a switch on the tile compartment between regular and small.

Tile-view Detailed Model button enhancement for multi-diagram

Today the Detailed Model button on a tile switches to the Global View, hides unrelated elements, and zooms in on the remaining model.

With multi-diagram:

  1. Detailed Model clicked
  2. Find all diagrams with the tile's elements
  3. Pop-up a list and let the user choose which diagram to switch to.
  4. If all elements of the tile are not found on any existing diagram, generate a new diagram with just those elements. Let the user choose to keep it or now.

Support user-defined tile tabs similar to diagrams

Tile tabs can be referenced groups of tiles with similar characteristics, like live or VOD, or category or rating.

Today we have a single tile tab, which may limit scaling out monitoring large number of channels.

Add documentation for H/A configuration

Add HA_CONFIG.md to repo. The document should describe how MSAM can be used from an alternate region if the primary region goes down completely. This should include the process to install a high-availability configuration of MSAM, which includes using global tables in DynamoDB.

Migrate from API keys to IAM authentication for REST

Internal user wants to have multiple access roles for browser connections.

Operator (change anything, diagrams, tiles, alarms, etc.)
Viewer (read only, no changes to back-end)

Process for adding new users changes (IAM users, or Cognito user pool)
Stored credentials on browser-side changes

Browser code needs to be aware of read-only role

Add links to specific tile tabs

It would be super handy if we can link to available tile tabs directly. This would allow for sharing content and landing on a specific tile tab page that you want to highlight. Currently, you can only link to the main MSAM page which takes you to the default Tiles tab.

Handle scaling for large inventories

Browser tool has problems over 2,000 nodes on the diagram. Need to perform some basic profiling and performance monitoring to understand the problem areas.

Support URL parameters for endpoint and API key

This helps for demo installations. Allow the user to provide an API endpoint and key as query parameters. Example:

../index.html?endpoint=https%3A%2F%2Fqsw0pljef6.execute-api.us-west-2.amazonaws.com%2Fmsam&key=PKICsVazoh7Ag9SrQ7RKm8cS2MJZCz9l6ef4hIng

This will override any previous settings and use this as the default. The user should get prompted, as usual, if the connection cannot be verified.

Support tile definition by service tag

Inventory elements with the tag "MSAM-Tile=" will be used to define that tile's inventory. That tag can occur multiple time per element so the element will appear on multiple tiles.

Support inventory and diagrams that come from multiple AWS accounts

How to display MediaLive/MediaPackage connections if they’re located in different AWS accounts?

Multiple REST endpoints can be supported. The internal model needs update to track which endpoint a particular node or edge originates.

Does any endpoint act as a 'master' or 'primary' endpoint, or is all everything distributed -- layout, settings, etc.

Generate a diagram dynamically for tile view detailed model button

Multiple diagrams makes it difficult to navigate 'back' to the original model. When the Detailed Model button is clicked on a tile:

  1. Find all diagrams with all elements of the tile and let the user choose which to show
  2. If not found, generate a new diagram dynamically with the name of the tile with the tile contents

Hide/eliminate the pre-layout resource "hairball"

Hide the pre-layout resource hairball. This is a default layout with physics enabled to pull all elements toward the center of the diagram. This should only show if no other layout information is present in the database.

Can we find a better "default" layout mode for new elements/first run of the tool?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.