nasa-pds / doi-service Goto Github PK

Applicable requirements
🦄 #6
🦄 #11
🦄 #12

Do that when we draft the osti record

1. Organize code as a python package so that it can be tested and packaged with:
   $ python setup.py test
   $ python setup.py sdist bdist_wheel

2. use the libraries standard for PDS dev (e.g. requests instead of urllib)

3. break the code into sub-modules and files,

Associated requirements:
🦄 #16

Propose in comment in the OSTI labels tags which are not managed by the software but that a user could manually fill in.

For example. 'affiliation'.

Heavily relies upon the Status component

Test more "programmatic" way of documenting requirements, possibly link them to tests

Look at behavioral testing (behave package)

Applicable requirements
🦄 #7

Design Details

See https://github.com/NASA-PDS/pds-doi-service/blob/master/docs/design/pds-doi-service-srd.md#listretrieve-a-doi

Inputs

This component should enable the status query of:

• DOI (1..*) - should return 0..1 DOI summary objects
• LID (1..*) - should return 0..* DOI summary objects
• LIDVID (1..*) - should return 0..1 DOI summary objects
• Submitter (1..*) - should return 0..* DOI summary objects
• Discipline Node Abbreviation (1..*) - should return 0..* DOI summary objects
• Submission datetime range (start, end) (ISO-8601 format) - should return 0..* DOI summary objects

Outputs

Based upon the DOI summary objects returned from the database, the software should return the summaries in two possible formats (specified via the API):

• JSON
• CSV

Applicable requirements
🦄 #30

Purpose of this task is to develop an internal database for maintaining the data in the DOI Summary object.

See https://github.com/NASA-PDS/pds-doi-service/blob/master/docs/design/pds-doi-service-srd.md#listretrieve-a-doi for the data we want to maintain in the database

Validation of the DOI metadata will most likely involve some curation from PDS personnel, however, there are some way we could potentially validate specific values entered. We could check these individually, use something like Schematron to validate.

Below is Schematron we can look through for reference submitted from one our discipline node colleagues.

SBN-DataCite.sch.txt (rename to .sch)

Applicable requirements
🦄 #12
🦄 #13

See https://github.com/NASA-PDS/pds-doi-service/blob/master/docs/design/pds-doi-service-srd.md#listretrieve-a-doi for details

Design Details

• Input: DOI Metadata Object, force_flag (available in the event we want to overwrite a record)

• General steps for updates:

  1. Verify the input DOI Metadata is valid
  2. Verify DOI from DOI Metadata Object is in current system.
     a. Failure: Return error that DOI does not exist
     b. Success: continue
  3. Optionally, check DOI+Node:
     a. if force flag is true, verify DOI+Node from DOI Metadata Object match a currently existing DOI+Node combination in the system.
     i. Failure: Return error with expected DOI +Node combination
     ii. Success: Continue
     b. If force flag is false, continue.
  4. Insert data into database (NOTE: database should be keeping transaction record for DOIs)
  5. Submit updated metadata to OSTI.
  6. Return submission response to user
  7. Continue to verify update status with OSTI and notify user (see #27).

• Outputs:
  TBD

Flow Diagram

TBD

Applicable requirements
🦄 #8

Request made by Ron:
An issue came up a couple of days ago where the Data Provider noticed that a DOI record, which was added @ OSTI on 2020-02-27, wasn’t showing @ DOI.org.
The OSTI record still has a status of “pending”. I regularly check for “pending” records using the OSTI GUI. But, the OSTI record doesn’t show up as “pending” in the GUI.

The point is that something like a CRON job will need to be in place to ensure that the OSTI “pending” records are propagated to DOI.org so that the DOI record(s) are accessible.

Linked requirements:
🦄 #30
🦄 #15

• Develop Sphinx documentation for the repo
• Documentation should include the following sections:
  • Installation
  • Usage
    • How to reserve a DOI
    • How to release a DOI (draft then release)
    • How to update DOI metadata (query then release)
  • Development
  • Support

See https://github.com/NASA-PDS/pds-deep-archive and https://nasa-pds.github.io/pds-deep-archive/# for an example of the docs. See docs directory for how the structure the RST files: https://github.com/NASA-PDS/pds-deep-archive/tree/master/docs

Related to Tracking Service.

See parent Epic for more details

Should support OSTI originally, but provide the ability to switch to DataCite (or other interface) in the future.

OSTI: https://www.osti.gov/iad2/docs

Applicable requirements
🦄 #10
🦄 #6
🦄 #7
🦄 #8
🦄 #9

Applicable requirements
🦄 #35
🦄 #30
🦄 #5
🦄 #9
🦄 #7
🦄 #6

Acceptance criteria:
The pds-doi-service should propose a restful server to activate the management of DOIs are described in the requirements.

See parent Epic for details

Plug the OSTI schema updated by Ron in the code at the DOI validation step.

See comments for schema to use.

Applicable requirements
🦄 #9

Applicable requirements
🦄 #7

START

Once a DOI submission enters a pending state, this handler component is triggered to will iteratively query OSTI for the current status of the pending DOI submission(s). Note: All submissions should enter this state when submitted to OSTI, unless there is an immediate failure returned.

Inputs

• List of pending_submissions - this dictionary contains all the submitted objects
  • the identifiers for these objects may be:
    • DOI(s) - already reserved/released a some prior point
    • LIDVID(s) - first time attempting to reserve/release a DOI for this product

Capabilities

Once a status change has been identified for individual members of pending_submissions:

• update the database with the updated status
• return to iterating over the remaining pending_submissions (Note: be sure to maintain those submissions whose status has updated so we can report on them once all pending_submissions have been updated.)

Once all members of pending_submissions have changed status (or a timeout has occurred):

• send email notification to the Submitter and [email protected] / [email protected] with list of submitted_dois DOI summary metadata
• trigger Discipline Node report generation per the discipline node of the Submitter

Applicable Requirements:
🦄 #35

Applicable requirements
🦄 #14
🦄 #15

This will be a parent ticket to capture all of the anomalies we trace down that will eventually be broken down into sub tickets.

For the future

dataCite: https://support.datacite.org/docs/api

Received some inputs from Ron regarding some example inputs/outputs.

We need to properly design the REST API and JSON response format for the DOI service.

Applicable requirements
🦄 #5
🦄 #6
🦄 #7
🦄 #8
🦄 #9
🦄 #10
🦄 #11

Reporter Comments
Should we play around with Swagger here? Or may not be worth the effort just yet.

This web service should meet the following use cases:

1. Generate DOI Object from PDS4 Label
   a. A user posts a bundle bundle/collection/product PDS4 label to the service.
   b. The service translates the metadata from the PDS4 label into the DOI/IAD Record Object
   c. The service return the IAD record object

Example Record Objects: DOI_bundle.xml.txt, DOI_collection_data.xml.txt

2. User Submits DOI Object
   a. A user posts a DOI/IAD Record object
   b. The service verifies validity of the object
   c. The service posts the object to the Tracking Service
   i. Alternatively, the service could post the object to a file staging area or just include in email
   c. The service notifies Operator of DOI submission
   i. Alternatively, the service could send an email to Operator for manual vetting and submission.

3. Operator Reserves DOI Object
   a. The authenticated Operator submits a DOI object to the service for reserve DOI
   b. The service validates the DOI object
   c. The service submit the DOI object to the IAD Service and verifies return values
   d. The service posts Reserved DOI information to Tracking Service and forwards response to Operator

4. Operator Release DOI Object
   a. The authenticated Operator submits a DOI object to the service for release DOI
   b. The service validates the DOI object
   c. The service submit the DOI object to the IAD Service and verifies return values
   d. The service posts Released/Published DOI information to Tracking Service and forwards response to Operator

Other things to think about:

• Ability to query DOIs - should probably be done by Tracking Service, but for workaround, could probably use DataCite
• Accumulating volumes - allow for some easy way to version accumulating volumes
• Best effort DOIs - e.g. "LROC RDR data" - give them the latest DOI for that data
• Figure out how to track DOIs at a LID-level
  • How do we include in a label or documentation that DOIs change by release?
  • People would want to able to potentially find DOIs for the top level
  • Need to overhaul citing web page (https://docs.google.com/document/d/1t27bRPMjM-AkkuUHqHV20zeBG-d4Db3Mya3Ri5DrRac/edit?usp=sharing)

More details on the IAD API and Schema: https://www.osti.gov/iad2/docs#record-model-fields

requirement

• why does this title contain some funky date information there? We should just use the Bundle Title as-is (I checked the Insight label and this is not the title).
  https://github.com/NASA-PDS/pds-doi-service/blob/61c593bce85c2b0bb4bd6e912339212f62031b11/input/DOI_Release_20200727_from_draft.xml#L4

• by default (for now), this should always be Dataset
  https://github.com/NASA-PDS/pds-doi-service/blob/61c593bce85c2b0bb4bd6e912339212f62031b11/input/DOI_Release_20200727_from_draft.xml#L32

• this should be defaulted to NASA Planetary Data System
  https://github.com/NASA-PDS/pds-doi-service/blob/61c593bce85c2b0bb4bd6e912339212f62031b11/input/DOI_Release_20200727_from_draft.xml#L54

• let's camel case this - PDS Operator
  https://github.com/NASA-PDS/pds-doi-service/blob/61c593bce85c2b0bb4bd6e912339212f62031b11/input/DOI_Release_20200727_from_draft.xml#L55

• product_type_specific should use the following values:

  • PDS4 Product_Bundle - PDS4 Refereed Data Bundle
  • PDS4 Product_Collection - PDS4 Refereed Data Collection
    https://github.com/NASA-PDS/pds-doi-service/blob/61c593bce85c2b0bb4bd6e912339212f62031b11/input/DOI_Release_20200727_from_draft.xml#L33

• for the LIDVID related identifier:

  • identifier_type = URN
  • relation_type = IsIdenticalTo
    https://github.com/NASA-PDS/pds-doi-service/blob/61c593bce85c2b0bb4bd6e912339212f62031b11/input/DOI_Release_20200727_from_draft.xml#L36

• is there a reason this related_identifiers block repeats?
  https://github.com/NASA-PDS/pds-doi-service/blob/61c593bce85c2b0bb4bd6e912339212f62031b11/input/DOI_Release_20200727_from_draft.xml#L86

• check to see if this mandatory (test without it). if not, remove.
  https://github.com/NASA-PDS/pds-doi-service/blob/3ce958942c3a97572eee3958622aceac1a141e39/config/DOI_template_20200407-mustache.xml#L5

See detailed design documentation https://github.com/NASA-PDS-Incubator/pds-doi-service/blob/master/docs/design/pds-doi-service-srd.md

This only include this function as a standalone python command based on a pds-data-doi module.

Applicable requirements
🦄 #5
🦄 #15

🦄 #35