Git Product home page Git Product logo

plaid's Introduction

APPS PDS Label Assistant for Interactive Design (PLAID)

The "APPS PLAID" is a web GUI to help create PDS4-compliant metadata labels. The core of "APPS PLAID" is a wizard that walks the user through a sequential process of creating a label. Behind the scenes, "APPS PLAID" interacts with a database for storing the user's information and progress through the wizard as well as the actual XML of the label.

"APPS PLAID" consists of a PHP-based application front end and a MariaDB persistence layer. It also uses SMTP to send registration emails to new users. The Docker container and composition environment is used to run and orchestrate the services in a repeatable fashion in both development and in operational deployment.

🏎 Quick Start

To get going quickly, run:

$ docker image build --tag plaid --file docker/Dockerfile .
$ env PLAID_IMAGE_OWNER= PLAID_VERSION=latest PLAID_PORT=8080 SMTP_HOST=smtp.jpl.nasa.gov \
        docker compose --file docker/docker-compose.yaml up

where $ is the command prompt.

Then, visit http://localhost:8080/ with a browser.

For a full set of instructions, see the development notes or the production instructions.

✋ Prerequsites

"APPS PLAID" requires the following in order to run regardless of if it's in development or production:

  • An SMTP server. "APPS PLAID" sends email and therefore needs a Simple Mail Transport Protocol server with which it can transmit emails. We don't go into setting up SMTP here, and in general it's best to consult your system administrator. You'll need to know:
    • Your SMTP host name and port.
    • The security protocol the server uses.
    • Any username and password (if required by the server) as well as the authentication method ; see the deployment instructions for full details.
  • Docker. APPS PLAID uses the Docker containerization and the Docker Compose orchestration system. Systems may come with Docker or you can install it yourself. If you're not usre, consult your system administrator. As of this writing, Docker Engine 20.10 and Docker Compose 2.1 or later are recommended.

🏃‍♀️ Developing and Deploying APPS PLAID

You can develop and deploy APPS PLAID using a Docker Composition (docker compose). This is the preferred configuration, but requires a front-end web server such as Nginx or Apache HTTPD to reverse-proxy to APPS PLAID.

💁‍♀️ Using APPS PLAID to make PDS4 Labels

This section describes the steps that go into making labels using APPS PLAID.

🎬 Preparation

Inside of APPS PLAID is the overall PDS4 JSON. This JSON is generated from the schema, which defines the structure and necessary content of a label. After the user selects the product type for their label, APPS PLAID starts creating a new JSON that is a subset of the overall PDS4 JSON. In this new JSON, objects are stored in a hierarchy relating to the structure of the label's XML. APPS PLAID uses the data stored in this new JSON to dynamically create the wizard's content and guide the user through the entire process of creating a PDS4-compliant label.

🧙 Wizard

After the initial set of steps has been dynamically created from the JSON for the selected product type, APPS PLAID handles a variety of tasks behind the scenes as the user progresses through the wizard.

First, it generates any subsequent steps according to the user's decisions. This is again accomplished by referencing the JSON formed after the user selects the product type. For example, if the user chooses to include an element that has sub-elements, a new step must be created to allow the user to choose which of the sub-elements to include in the label.

Second, it updates the XML based on the user's decisions and stores that XML in the database. In order to accurately place the new XML in the label, the path for the PDS4-attribute is stored in the ID of the HTML element. The path was stored in the ID when the step's content was dynamically created from the JSON data. This path is passed to the backend and used to locate where to place the new element in the XML. Prior to running the XPath query on the backend, the path is parsed so that it maps to the XML structure.

  • HTML ID: id="0/Identification_Area/1/logical_identifier"`
  • Corresponding XML path: Identification_Area/logical_identifier

Third, it captures the user's progress in the wizard and saves it to the database for future loading. This information is stored in a JSON that sits in a column of the label’s table. Upon reloading the wizard, this JSON is used to restore the user’s progress for the selected label.

🏁 Completion

Once the user has completed the process of constructing the label, APPS PLAID presents a preview and the option to export. The preview and file export are both accomplished by getting the XML from the database and either displaying it in the webpage or sending it as a downloadable file.

APPS PLAID Flow Diagram

APPS PLAID Flow

Database Diagram

DB Diagram

👥 Contributing

Within the NASA Planetary Data System, we value the health of our community as much as the code. Towards that end, we ask that you read and practice what's described in these documents:

  • Our contributor's guide delineates the kinds of contributions we accept.
  • Our code of conduct outlines the standards of behavior we practice and expect by everyone who participates with our software.

📃 License

The project is licensed under the Apache version 2 license.

plaid's People

Contributors

jeffyuhaoliu avatar jordanpadams avatar nutjob4life avatar pdsen-ci avatar stirlingalgermissen avatar tloubrieu-jpl avatar voxparcxls avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

plaid's Issues

Base image used by APPS PLAID is not supported

Vulnerability

"APPS PLAID" uses as its base Docker image php:5.6-apache-jessie.

Debian Linux jessie reached end-of-life on 2018-06-17 and end of long-term support on 2020-06-30. It has received no security updates since then.

PHP 5.6 reached end of life on 2019-01-01 and has no long-term support. It has received no security updates since then.

Upgrading the base image of "APPS PLAID" is strenuously recommended.

Software Version

main

As a PDS Operator, I want to update PLAID to the latest PDS4 Discipline PDS Data Dictionaries

https://pds.nasa.gov/datastandards/dictionaries/

Namespace Parent Class
proc proc:Processing_Information
sp sp:Spectral_Characteristics
img_surface img_surface:Surface_Imaging
nucspec nucspec:NucSpec_Observation_Properties
msn msn:Mission_Information
img img:Imaging
geom geom:Geometry
speclib speclib:Spectral_Library_Product
msss_cam_mh msss_cam_mh:MSSS_Camera_Mini_Header
disp disp:Display_Settings
cart cart:Cartography
msn_surface msn_surface:Surface_Mission_Information
rings rings:Ring_Moon_Systems
survey survey:Survey

Update PLAID to better manage dependencies

Current implementation

#15 will be implemented using a temporary solution where all discipline LDDs and the PDS common namespace are all in one uber JSON.

Proposed improvement

For a more refined solution, we would like to have each discipline LDD have its own JSON, and PLAID will parse that JSON for a new namespaces key to determine all dependent namespaces for the LDD, and load those other JSONs as appropriate.

Per this improvement to LDDTool, generated LDDs now contain this new key. See attached for the CART LDD: PDS4_CART_1G00_1950.JSON.txt

From this example you can see the new namespaces key:

[
  {
    "dataDictionary": {
      "Title": "PDS4 Data Dictionary" ,
      "Version": "1.16.0.0" ,
      "Date": "Tue Jan 19 19:36:45 PST 2021" ,
      "Description": "This document is a dump of the contents of the PDS4 Data Dictionary" ,
      "namespaces": ["pds:", "geom:", "cart:"] ,
      "classDictionary": [
...

PLAID should use this to then go through the other JSON files and parse each of those namespaces

Update available discipline LDDs

Current list of discipline LDDs:

./proc
./sp
./img_surface
./nucspec
./msn
./img
./geom
./speclib
./msss_cam_mh
./disp
./cart
./msn_surface
./rings
./survey

Improved User Guide

Current User Guide is not very helpful and trail and error is necessary to figure out how to use the software. Improvements and/or tutorials would better help onboard new users.

As a PDS Operator, I want to update PLAID to the latest PDS4 Information Model

Per January tag-up, here is the plan to fix these issues:

  • PDS4 IM versions 1.14.0.0 and earlier - LDDs are not organized. Grab all dictionaries, use the latest
  • PDS4 IM versions 1.15.0.0 and on - group by IM version. Once a user selects an IM version, we have a specific subset of discipline LDDs

@jeffyuhaoliu identified an issue with the JSON including all dependencies. near term solution: have @jshughes create on uber JSON with all LDDs in it. future solution: JSON output for each dLDD will contain all dependent LDDs

Revise Docker and deployment docs for open source access

📖 Additional Details

  • See PR here for initial version of docs: #29
  • Per this comment, we need to update the Docker deployment and procedures to use publicly available images

⚖️ Acceptance Criteria

Given a desire to deploy PLAID
When I perform the procedure described here (link TBD) to deploy using docker containers for dev purposes
Then I expect to be able to spin up PLAID and run in a dev environment

Given a desire to deploy PLAID
When I perform the procedure described here (link TBD) to deploy using docker containers for ops purposes (includes instructions for necessary HTTP proxying, file system mounting, etc)
Then I expect to be able to spin up PLAID and run in a dev environment

`call_user_func` executes unfiltered arguments from http client

Vulnerability

In 9 places in the PLAID source code, call_user_func and call_user_func_array appear. In at least 3 of those locations, the first argument (the name of the function to call) is passed in from the HTTP client; for example:

if(isset($_POST['Function'])){
    $DOC = readInXML(getLabelXML());
    call_user_func($_POST['Function'], $_POST['Data']);
}

No checking is performed before making this call. A client could construct a specially formulated POST payload to execute an arbitrary PHP function with an arbitrary argument, such as system with rm -rf /. As a proof of concept, I was able to execute phpinfo() as well as pcntl_exec().

Software Version

main as of 2022-01-04.

Have a release/tag workflow for PLAID

💪 Motivation

...so that we can publish and track release and tags.

📖 Additional Details

⚖️ Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details

Upgrade PLAID to use latest IM

Came across issues with trying to upgrade some LDDs. Need to test and debug where the issues are.

Determine if these customizations in PLAID will cause significant overhead in the future

All products are assumed to be Product_Observational by PLAID

🐛 Describe the bug

Whenever I try to create a product not Observational (e.g. Document, Context, etc.), the software tries to drop this into Product_Observational and it does not work.

Minimal support should include:
See https://pds.nasa.gov/datastandards/documents/im/v1/index_1G00.html#class_pds_observation_area for:

  • Product_Observational

See https://pds.nasa.gov/datastandards/documents/im/v1/index_1G00.html#class_pds_context_area for:

  • Product_Context
  • Product_Document
  • Product_Collection
  • Product_Bundle

Compatibility with PDS4 version 1.13.0.0

Hi there,

I've just taken PLAID for a spin at it seems to be a really nifty tool. However, I am wondering whether there are plans to update PLAID to support v1.13 of the information model? If not, could you provide a rough estimate of the work required to complete such work?

All the best,
Ariel Ladegaard

As a developer, I want to see exception data

💪 Motivation

So that I can more effectively containerize, open-sourcerize, and otherwise whip APPS PLAID into shape (especially with regard to how #30 is growing in scope), it would be great to see exception and error condition details.

📖 Additional Details

In eight different locations (and perhaps more) in the APPS PLAID code, there are lines like these:

} catch (PDOException $ed) {
    //print($ex->getMessage()); // Don't print any error message, for security reasons
    return true;
}

That makes debugging APPS PLAID about 43× more difficult.

I get that PHP is awful and sort of insecure by default but there probably is some way to at least write exception details to the server log without it also going to the HTTP client.

⚖️ Acceptance Criteria

Given an exceptional condition
When I perform viewing the server log via docker-compose logs plaid
Then I expect to see stack traces or other exception data

⚙️ Engineering Details

As an PDS Operator, I want a checklist for items to test in order to update to the latest PDS4 IM and LDDs

Motivation

... so that I can quickly spot check test that the new version worked.

Draft checklist

  • Verify new IM version appears on "create" page
  • Spot check 1+ changes to the new IM version appears in the PDS common portion of the workflow
  • Verify all new LDD versions appear on Discipline Dictonaries tab of the workflow
  • Spot check 1+ changes to 1+ LDDs appear in the LDD portion of the workflow

Prepare PLAID for production release on pds.nasa.gov

  • EN to test deployment of PLAID to Staging (/tools/plaid)
  • EN deploy to Production
    • deploy to production
    • export from PLAID.jpl.nasa.gov and import to production
    • redirect plaid.jpl.nasa.gov to pds.nasa.gov/tools/plaid

Develop procedure for deploying production version of PLAID

We need a procedure for how to deploy PLAID to plaid.jpl.nasa.gov OR just simply the installation and deployment instructions if they differ at all in production versus test regarding MySQL / Docker / etc. and we can forward plaid.jpl.nasa.gov to something like pds.nasa.gov/tools/plaid

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.