teselagen / fsml.org Goto Github PK

In regards to libFSML https://github.com/TeselaGen/fsml.org/milestone/3 , task indicates that we need to write the library in Python. However I think writing it in Deno and compiling it to WebAssembly (WASM) and then importing WASM into Python would provide us maximum flexibility because then we could consume it in the browser via JavaScript or on the server via JavaScript or Python or any other language that supports WASM (which is growing everyday).

1. Can we switch and write the libraries in Deno instead?

We need to start developing relationships with other developers who will be interested in both using and contributing to this schema and tool chain.

Daniel Bryce at SIFT.net has created a nice alpha for a directed graph style protocol editor for PAML (see screenshot). We should investigate tooling that fits better with our pretty-much pure JS development environment. @tgadam has suggested the following:

• https://js.cytoscape.org/#introduction
• https://github.com/uber/react-digraph
• https://reactflow.dev/
• https://github.com/projectstorm/react-diagrams

Let's check them out and pick one to move forward as a provisional choice by the next BioMADE meeting on Feb 25.

Dan's visualization of a 96-96 well plate stamp:

Implement CLI Command to Generate Manifest for PDF with Core Sections. Depends on #35

Develop appropriate tooling for reading, writing, and
manipulating files and data streams in FSML format. Written in Python and provides APIs for JavaScript,
Python and tools for binding standardized XML to Python, JSON objects. An additional objective in support of
the library is to develop an FSML Development Process that details the development process and community
organization procedures that are followed by the FSML community. Also in support of the library we will
develop a FSML Test Suite for conformance testing FSML.

We need to layout the command structure for the FSML CLI tool.

Here's what I have so far:

---
Global flags

--config/-c=[./config.json,filepath] #sets flags from config file
--stdin #reads inputs from stdin

---

help

version

create manifest 
  --type=[data,protocol]
  --parser=[...parser plugin name] 
  --format/-f=[json,yaml,toml] 
  --write/-w=[filepath] # required if pack set / by default all will be sent to stdout
  --pack=[zip,tar,tgz]
  --author=[[email protected]] # required if not set in defaults
  <filepattern...>

update manifest
  --type=[data,protocol] 
  --parser/-p=[...parser plugin name] 
  --format/-f=[json,yaml,toml] 
  --write/-w=[filepath] # required if pack set / by default all will be sent to stdout
  --pack/-z=[zip,tar,tgz]
  --patch="{}" #inline json
  --patch-file=[filepath] #patch json file
  --patch-type=[json,json-merge,lodash-merge]
  --author=[[email protected]] # required if not set in defaults
  <path>

fetch manifest
  --registry=<name> / --uri=<download uri>
  --format/-f=[json,yaml,toml] 
  --unpack=[true, false]
  --write/-w=[filepath] # required if pack set / by default all will be sent to stdout
  <id>

import manifest
  --from <registry name/alias> / --from-uri=<download uri>
  --to <registry name/alias>
  <id>

describe manifest
  --output/-o=[record,json (pretty print),yaml,toml]
  --write/-w=[filepath] # required if pack set / by default all will be sent to stdout
  --select=[comma separated list of property paths]
  --summary #only show summary
  --section=[supplemental section name] #only show that section
  <path>

score manifest
  --formula=[latest,vx.x.x]
  --dry-run (or --update)
  --author=[[email protected]]
  <path>

export manifest
  --exporter=[...export plugin name]
  --write/-w=[filepath] #by default all will be sent to stdout / maybe required by some exporters
  [...--exporter flags]
  <path>

validate manifest <path>

pack manifest 
    --pack=[zip,tar,tgz]
    <path>

unpack manifest <path>

defaults edit  # runs interactive default settings mode

default list # shows default flag values

defaults set <key> <value>

defaults reset <key> # resets key to installed default

defaults reset-all # resets all keys to installed default
          --confirm 

install plugin 
  --cache 
  <path, uri>

list plugin / list plugins
  --type=[parser,exporter]
  --filter=[glob] 
  --regex=[regex]
  --sort=[asc,desc]

upgrade plugin 
  --latest / --version=[latest] / --minor / --major / --patch #one of these flags must be provided
  --cache
  <name>

uninstall plugin <name>


### Plugins will be "installed" by saving a reference to the plugin via a url however
### the plugin won't be actually downloaded until it's used. The cache command will
### force the plugin to be downloaded locally and cached 
cache plugin <name>

cache plugins #cache all plugins

delete plugin cache # removes all plugins from local cache
(or clear plugin cache)

run <plugin name> ...args


add registry  
  --platform=[gh,git,s3,gcs,ftp,fsml-api,fs]
  --alias=[alternate name]
  <uri / path>

index registry <name>

remove registry <name>

search registry <name>

serve registry
  --port=[3090]
  --name=[fsml_<shortid>]
  --properties=[./server-properties.json]
  --middleware=[..list of plugins to use a middleware]
  --allow-submissions
  --temp-dir=[./tmp]
  --index-dir=[./index]
  <manifests location>

The current way of updating the Gatsby site is cumbersome. You have to pull the code locally, edit the markdown, run a build process, then preview it locally, commit it and then merge it into the main branch. Additionally there's no way to share a preview of the changes with anyone else.

The compiled website needs to go in the /docs folder on the main branch. That's where we're serving the content for GitHub pages.

Currently the /website folder contains the Gatsby source.

• Create GitHub Action to auto build markdown changes
• Create a way to preview new website changes

We've proposed a draft for the exchange protocols and now we need to expand upon that to also support exchanging data generated by those protocols such as execution traces and workflow runs.

We should take our existing Test Data Model and convert it into a standard.

Create a Proof of Concept with a trivial example that takes a Deno package, compiles it to WASM, and uses that WASM module in a Python library.

Phase I

• Create trivial Deno package that exports several functions and classes
• Convert the package to JavaScript using Deno to Node dnt
• Convert the JavaScript code to WASM using Javy (or something equilavent)
• Test running WASM package

Phase II

• Create a trivial Python app that uses wasmer to run the Deno wasm file
• Create a trivial Python library that imports and re-exports a function from the Deno wasm file
• Publish the Python WASM wrapper library
• Try to use the Python WASM wrapper library in a Python app

Phase III

• Create unit tests for the Python WASM wrapper library

Phase IV

• Create a Pre Publish GitHub Action to create the WASM, wrap it in Python, and run tests
• Create a Publish GitHub Action to publish the WASM Python wrapper

Related to #14

MVP rendering of a "static protocol" from JSON structure based on protocols.io

Protocols.io provides an obvious template for how to layout a protocol and associated metadata. This open-source "protocol viewer" can be developed in analogy with OVE, so it could be easily incorporated into TG as well as other tools, and we can solicit community support for its development.

The MVP is a read-only view of the protocol, but we will no doubt want to add the ability to edit soon thereafter.

Create a CLI project using Deno and Nx monorepo workspaces.

Deno - Deno is the "next" iteration of Node.js. It provides native TypeScript and es6 support. It's compatible with Node.js and supports isomorphic (runs on back and front end) code. Finally it has the advantage of being able to target all major OS's and compile to a single executable / CLI tool.

Nx - for monorepo support. Nx is fairly platform agnostic however it is missing tooling for nrwl/nx#5823 which we'll need to provide as plugins.

Example Deno CLI projects

• https://dev.to/unorthodev/build-a-simple-cli-tool-with-deno-1fmk
• https://startfunction.com/2020/07/03/how-to-create-deno-cli-tool/
• https://www.twilio.com/blog/use-deno-build-cli
• https://levelup.gitconnected.com/build-a-cli-tool-deno-by-example-79d39f25eb0a
• https://www.loginradius.com/blog/engineering/build-a-cli-tool-using-deno/

• Protocols.io is a source of relatively well-structured protocols, even though these protocols do not contain enough information to be considered fully fleshed out "workflows".
• Creating an interface in TG whereby users can import a protocol from protocols.io into TG would be a first step toward creating a fully fleshed-out workflow based on what traditionally constitutes a protocol. This is an end-run around starting from a raw text file or PDF since protocols.io is structured and has a fairly complete model for what metadata should be associated with a protocol.

This ticket covers the creation of an interface that allows TG users to load a protocol from protocols.io.

We want to have a predictable way to convert markdown to html. Showdown is one of the most popular packages that does this. We need to determine if it will work with Deno.

https://github.com/showdownjs/showdown

Here's the link to the Deno docs on how to use Node.js code in Deno https://deno.land/[email protected]/npm_nodejs

The screenshot below shows a 96-96 well plate stamp in Dan Bryce's PAML editor. We should pick a preferred JS-based graph tool (see ticket #2) and generate a simple mockup of how the tool should be laid out... perhaps in analogy with Node-RED.

Here is a screenshot from Dan's PAML editor to give us an idea of its look and feel.

Here is a screenshot from Node-RED to give us an idea about its look and feel and the needed functional elements.

It was identified that the manifest schema will need to cover fermentation data as well as protocol. Additionally we've identified upstream components such as en experiment definition and other associated information such as the equipment and execution environment needed to run a protocol that may also need be tracked by a manifest.

To accommodate this I propose adding a type field to the core schema as well as then creating a new set of type specific nested core settings for each type.

The utility scores will need to be nested under this type specific schema section since different types will have different utility scores.

Each utility score area needs to have the calculation method defined for it. Each score should be normalized to fall between 0-1 inclusive.

• #39
• #40
• #41
• #42
• #43
• #44
• #45
• #46
• #47
• #48
• #49
• #50
• #51
• #52
• #53

I propose using TypeBox with Deno to create the specification. Then compile that library to WASM for use in other languages such as Python.

The current draft specification is documented on the website fsml.org.

TypeBox - TypeBox is a type validation system built on TypeScript and JSON Schema. It provides the ability to have static (or transpile time) TypeScript validation of types as well as runtime JSON Schema validation of types. It can also generate a sinclairzx81/typebox#143 from the TypeScript types. It's available for both Node.js and Deno.

For protocols imported or indexed in our platform... users should have the option to enhance to such an extent that they are fully fleshed out workflows.

This is imagined as a way, for example, of converting protocols.io (or other) protocol "steps" into a set of placeholder tools in a TeselaGen workflow. The user would then convert each step to a proper tool or tools and link them as appropo.

Otherwise, for a simple pdf... it could just be a convenient side-by-side or standard browser tab view that could be torn off so the user could see the protocol steps while creating a workflow... (trivial example).

In any case, the workflow should be cross linked with the protocol.
i.e. in the protocol... a list of linked workflows (where this protocol is utilized).
i.e. in the workflow... a list of linked protocols (that this workflow is based on).

Implement e2e CI tests for CLI tool using GitHub Actions.

Objective 2.1 - Develop a draft standard for fermentation process and protocol data: This
objective includes the creation of a specification, which we provisionally call the Fermentation Science Markup
Language or FSML. The FSML specifications define the syntax and semantics of FSML. These documents will
become the definitive references for FSML. The objective will also include the development and support of a
community website for sharing FSML developments and software along the lines of SBML.

MVP Node-RED Flow for importing a complete protocol from protocols.io into TeselaGen.

1. The imported protocol will appear in the Protocols table in TeselaGen.
2. The imported protocol will be viewable in TeselaGen's (or anyone else's) implementation of the static protocol viewer.
3. The imported protocol will be exportable as JSON.

The manifest schema is stored in the spec package. We need a package that will contain the functionality to use the schema to both generate and validate the schema.

We want this package to be isomorphic so it needs to not have any server or web specific dependencies such file system libraries. Instead data should be passed to this package as a stream.

After careful evaluation, the following tech stack has been chosen:

1. Deno - Deno is the "next" iteration of Node.js. It provides native TypeScript and es6 support. It's compatible with Node.js and supports isomorphic (runs on back and front end) code. Finally it has the advantage of being able to target all major OS's and compile to a single executable / CLI tool.
2. Nx - for monorepo support. Nx is fairly platform agnostic however it is missing tooling for Deno which we'll need to provide as plugins.
3. TypeBox - TypeBox is a type validation system built on TypeScript and JSON Schema. It provides the ability to have static (or transpile time) TypeScript validation of types as well as runtime JSON Schema validation of types. It can also generate a JSON schema from the TypeScript types. It's available for both Node.js and Deno.

• Setup Repo with Nx #9

• Setup an Example Deno CLI project #10

  • Example CLI projects
    • https://dev.to/unorthodev/build-a-simple-cli-tool-with-deno-1fmk
    • https://startfunction.com/2020/07/03/how-to-create-deno-cli-tool/
    • https://www.twilio.com/blog/use-deno-build-cli
    • https://levelup.gitconnected.com/build-a-cli-tool-deno-by-example-79d39f25eb0a
    • https://www.loginradius.com/blog/engineering/build-a-cli-tool-using-deno/

• Create any necessary Nx plugins for Deno CLI project #9

  • Probably will need a dependency graph plugin using https://deno.land/x/[email protected]
  • Might need an inference plugin to detect Deno configs
  • Probably will want some Deno generator plugins for packages

• Setup GitHub Actions CI/CD #12

Task 2 describes at a high level developing a draft for fermentation data. However for the past 6 months we've been focused on creating a specification for fermentation protocols.

I think we should have a planning discussion to better understand the requirements of the work and ensure our current work is heading in the right direction.

1. Should protocols be part of the specification?
2. Should we be focusing on fermentation data specifications first instead of protocol specifications?

This is a summary listing of our tickets and status (open/closed)

• #2
• #3
• #4
• #5
• #6
• #7
• #8
• #9
• #10
• #11
• #12
• #14
• https://github.com/TeselaGen/FSML.org/issues/16
• #17
• #18
• #19
• #20

In addition to "regular" markdown we would like to support MDX v2 (markdown with JSX...aka React Components). I found an example of doing this in Node.js mdx-js/mdx#1985 (reply in thread) . We need to see if we can get the same thing working in Deno.

@tgadam can you flesh this ticket out a bit?

We will work with the BioMADE community to develop a draft
standard for fermentation data taken at the manufacturing operating systems level. Datasets will be
represented in a structure that reveals the characteristics of individual data items and the relationships among
them. In addition, we will develop parser generation software that will auto-generate a parser that can accept
relatively unstructured fermentation data and convert to the standard.

• #23
• #24

Deliverables

1. Operational, publicly accessible community website (provisionally FSML.org)(Month 12) https://github.com/TeselaGen/fsml.org/milestone/1
2. Complete specification of standard for data exchange of fermentation data,
   including syntax and semantics of FSML (Month 6 draft, Month 9 final) https://github.com/TeselaGen/fsml.org/milestone/2
3. libFSML: Open access tools library for reading, writing, and manipulating files
   and data streams in FSML format. Written in Python with APIs for Javascript
   and Python (Month 15) https://github.com/TeselaGen/fsml.org/milestone/3
4. Report on FSML development process, detailing both the process and
   community organization procedures that are followed by the SBML community
   (Month 18) https://github.com/TeselaGen/fsml.org/milestone/4
5. Test suite: Open access FSML test suite for conformance testing of FSML,
   including a collection of test models and a framework for running software
   through the suite (Month 21) https://github.com/TeselaGen/fsml.org/milestone/5
6. Demonstration of parser generation software that will auto-generate a parser
   that can accept unstructured fermentation data (Month 21) https://github.com/TeselaGen/fsml.org/milestone/6

protocols.io can provide us with a rich source of user vetted information on what constitutes the essential data and metadata needed to fully describe a scientific protocol. We will use this information as a shortcut to a JSON structure that will serve as a cornerstone for our open-source JSON representation of what traditionally constitutes a protocol, if not a complete workflow. One example of a data model based on protocols.io information, (but slightly extended to cover some other notions around fermentation) is shown:

@tgadam can we flesh this out a bit?

Currently the GitHub pages site is on a separate branch and hosted from the root of the repo. I think in order to facilitate auto generated documentation on the website we should merge it into the main branch and put it in a subfolder.

Additionally I plan to rename the repo to all lowercase to match the patter of our other repos.

• Move into website folder
• Merge into main
• Rename lowercase

Create a CLI that can take in a PDF static protocol and generate a standardized manifest file that conforms to a JSON Specification.

• #9
• #10
• #11
• #12
• #13
• #14

The current way in-app links are created in Gatsby makes it easy to create a broken link. We need a crawling test that can be run in a GitHub Action that will find broken links in the website and website previews (depends on #34 )

Currently Nx doesn't support Deno out of the box. However it has a flexible plugin system that will allow us to create plugins to support Deno packages. We have two ways to do this:

Inference Plugin

An inference plugin will crawl the directory structure and find directories that represent project directories and infer the project settings from files in that directory. Currently Deno packages can contain a config file such as deno.jsonc that is similar to a package.json that we could look for in order to infer projects for Nx. The downside of this is that we're limited to what we can infer from the Deno config.

Project Generator

Instead of inferring projects, you can also place a project.json file in a folder to designate it as an Nx project. To help in creating these and maintaining them Nx provides a set of libraries to create generators and tools that will create and update projects. Instead of inferring projects from a Deno config we could use a generator to create a Deno package or app as well as the config and project file.