DAG view should have params:

• currently selected node
• currently highlighted node (in ontology)
• cross-reference highlight
• sub-set node (aka root node)

Lots of blueprintjs errors from the use of popover2. Unfortunately, will not be fixed until mid-2022 with Blueprint 5.

Ref: https://github.com/palantir/blueprint/wiki/Blueprint-5.0
and palantir/blueprint#3979

Example:

Warning: Legacy context API has been detected within a strict-mode tree.

The old API will be supported in all 16.x releases, but applications using it should migrate to the new version.

Please update the following components: Blueprint4.Portal

Learn more about this warning here: https://reactjs.org/link/legacy-context

Warning: findDOMNode is deprecated in StrictMode. findDOMNode was passed an instance of Transition which is inside StrictMode. Instead, add a ref directly to the element you want to reference. Learn more about using refs safely here: https://reactjs.org/link/strict-mode-find-node

etc.

There are a number of graph filtering steps done in the front-end, eg, removing terms from CL that are likely not useful. These should all migrate to the create graph script.

Should also add whitelist - item to seed the graph with (beyond that which is in the dataset). This would generalize the currently hardcoded whitelist of CL & UBERON.

Pablo proposes Wilcoxon rank sum test

https://scanpy.readthedocs.io/en/latest/generated/scanpy.tl.rank_genes_groups.html#scanpy-tl-rank-genes-groups

Rather than subsetting, a substring match on endothelial or a range for "all cells with more than 1,000,000 count" would allow removal. This would be contrasted with subsetting operations which subset to a root node or compartment.

h/t Norbert Tavares

The placeholder text always says "cell type search", regardless of the ontology being displayed.

Perhaps it should say "search ontology" or something like that?

This likely appears in the left menu underneath ancestors and descendants. Ie.:

Ancestors, descendants, compartments, datasets

This code was removed after Tabula Sapiens hardcoded stuff was removed, but for posterity, the ideas are to add a label to the graph given an arbitrary set of cell types, as would exist in an arbitrary dataset being compared against the reference of 'every cell type'

      /**
       * check 'current' dataset distribution in ontology
       */

      const currentDatasetCelltypes = null;

      if (currentDatasetCelltypes && currentDatasetCelltypes.includes(node.id)) {
        context.fillStyle = datasetDistributionColor;
      }

and

    /**
     * Draw text on nodes, for alpha, this writes tabula sapiens nodes out
     */
    if (showTabulaSapiensDataset) {
      for (const node of nodes) {
        if (tabulaSapiensCelltypes.includes(node.id)) {
          const vertex: any = ontology.get(node.id);
          if (vertex && vertex.label && typeof vertex.label === "string") {
            context.fillStyle = tooltipColor;
            context.font = "18px serif";

            if (typeof node.x !== "number" || typeof node.y !== "number") {
              console.log(
                "in drawForceDag/ticked: while attempting to context.fillText, node.x or node.y was not a number"
              );
              return;
            }

            context.fillText(`${vertex.label.substring(0, 10)}`, node.x, node.y);
          }
        }
      }
    }

Need API surface that:

• given
  • an ontology (eg, CL)
  • search criteria
• returns a single node defining a sub-tree that is the minimum spanning tree of all nodes in the ontology matching the search

Search criteria -- likely needs are:

• search within the ontology - eg, match on label, synonyms, ID
• search within the cross-references (lattice xref)

Background info from Colin on a use case

Selecting an Uberon term in the search field subsets graph to selected CL terms

This subset also applies if user has subset by ancestor in CL, as in this case, subset to neuron:

ie., "less than 10% (n%) of the entire cellxgene corpus"

h/t @maniarathi

when the application receives an ontology on load up, various scales will need to know about the extents [min, max] of certain numeric parameters like n_cells

this suggests that whatever graph is being loaded up, like cl, might also have 'top level' attributes that hint components

User has to scroll to get to compartments:

react-helmet is unmaintained and no longer compatible with recent React versions. There is a fork react-helmet-async which should be used instead.

This change will resolve the startup error message seen in the console:

Warning: Using UNSAFE_componentWillMount in strict mode is not recommended and may indicate bugs in your code. See https://reactjs.org/link/unsafe-component-lifecycles for details.

* Move code with side effects to componentDidMount, and set initial state in the constructor.

Please update the following components: SideEffect(NullComponent)

For more details, see:

• nfl/react-helmet#465
• nfl/react-helmet#413

https://www.ebi.ac.uk/ols/docs/api#search seems to provide everything the lattice xref does, and would reduce our graph size dramatically.

If you open the console, then open the Component drawer, there are many errors generated. A partial list:

Warning: Legacy context API has been detected within a strict-mode tree.

The old API will be supported in all 16.x releases, but applications using it should migrate to the new version.

Please update the following components: Blueprint3.Portal

Learn more about this warning here: https://reactjs.org/link/legacy-context
    at Portal (http://localhost:3000/static/js/vendors~main.chunk.js:14410:43)
    at Overlay (http://localhost:3000/static/js/vendors~main.chunk.js:12328:43)
    at Drawer (http://localhost:3000/static/js/vendors~main.chunk.js:6647:38)
    at div
    at OntologyExplorer (http://localhost:3000/static/js/main.chunk.js:539:5)
    at div
    at OntologyExplorer (http://localhost:3000/static/js/main.chunk.js:2177:5)
    at Route (http://localhost:3000/static/js/vendors~main.chunk.js:124802:29)
    at Switch (http://localhost:3000/static/js/vendors~main.chunk.js:125004:29)
    at div
    at Router (http://localhost:3000/static/js/vendors~main.chunk.js:124433:30)
    at BrowserRouter (http://localhost:3000/static/js/vendors~main.chunk.js:124054:35)
    at App (http://localhost:3000/static/js/main.chunk.js:39:5)
    at HotkeysProvider (http://localhost:3000/static/js/vendors~main.chunk.js:18272:21)
console.<computed> @ index.js:1
printWarning @ react-dom.development.js:67
error @ react-dom.development.js:43
(anonymous) @ react-dom.development.js:11587
push../node_modules/react-dom/cjs/react-dom.development.js.ReactStrictModeWarnings.flushLegacyContextWarning @ react-dom.development.js:11571
flushRenderPhaseStrictModeWarningsInDEV @ react-dom.development.js:23819
commitRootImpl @ react-dom.development.js:23005
unstable_runWithPriority @ scheduler.development.js:468
runWithPriority$1 @ react-dom.development.js:11276
commitRoot @ react-dom.development.js:22990
performSyncWorkOnRoot @ react-dom.development.js:22329
(anonymous) @ react-dom.development.js:11327
unstable_runWithPriority @ scheduler.development.js:468
runWithPriority$1 @ react-dom.development.js:11276
flushSyncCallbackQueueImpl @ react-dom.development.js:11322
flushSyncCallbackQueue @ react-dom.development.js:11309
discreteUpdates$1 @ react-dom.development.js:22420
discreteUpdates @ react-dom.development.js:3756
dispatchDiscreteEvent @ react-dom.development.js:5889
index.js:1 Warning: Legacy context API has been detected within a strict-mode tree.

The old API will be supported in all 16.x releases, but applications using it should migrate to the new version.

Please update the following components: Transition, TransitionGroup

Learn more about this warning here: https://reactjs.org/link/legacy-context
    at TransitionGroup (http://localhost:3000/static/js/vendors~main.chunk.js:127118:30)
    at Portal (http://localhost:3000/static/js/vendors~main.chunk.js:14410:43)
    at Overlay (http://localhost:3000/static/js/vendors~main.chunk.js:12328:43)
    at Drawer (http://localhost:3000/static/js/vendors~main.chunk.js:6647:38)
    at div
    at OntologyExplorer (http://localhost:3000/static/js/main.chunk.js:539:5)
    at div
    at OntologyExplorer (http://localhost:3000/static/js/main.chunk.js:2177:5)
    at Route (http://localhost:3000/static/js/vendors~main.chunk.js:124802:29)
    at Switch (http://localhost:3000/static/js/vendors~main.chunk.js:125004:29)
    at div
    at Router (http://localhost:3000/static/js/vendors~main.chunk.js:124433:30)
    at BrowserRouter (http://localhost:3000/static/js/vendors~main.chunk.js:124054:35)
    at App (http://localhost:3000/static/js/main.chunk.js:39:5)
    at HotkeysProvider (http://localhost:3000/static/js/vendors~main.chunk.js:18272:21)

These controls presently only work if changed in code editor + hard reload. Needs thinking about re-render flow.

to “x-ref to uberon” or "cross reference to ONT" whatever ONT is.

add new term to ontology explorer — related terms ontology, it’s the ontology that you search in for any related terms, when you’re in CL, you would xref to uberon

• path names
• component params
• UI

Partially implemented in #60

• ✅ Ingest multiple ontologies on load-up
• ✅ Leverage EBI API for fetching data for cell cards view
• ✅ Link out to an ontology from cellxgene left sidebar quickly, but you can grab the term and put it in the url bar so that already works
• ✅ Display a page for a vertex
• ✅ Link multiple ontologies
• ✅ Allow the user to jump between ontologies and traverse the graph
• ✅ Visualize an entire ontology dag
  • ✅ forces
  • ✅ remove nodes
  • ✅ hulls
  • ✅ labels
• ✅ Click to pin node in graph (pins vertex on left)
• ✅ Click again to clear node
• ✅ Simple text match substring search, ie., visual or neuron
• ✅ Try with other ontologies, disease and uberon
• ✅ Disjoint graph capability!
• ✅ Light up ancestors on hover
• ✅ Light up descendants on hover
• ✅ Terminal nodes
  • ✅ Allow remove/filter terminal nodes
• ✅ Sugiyama layout
  • ✅ confirm that dag connect doesn't work 
  • ✅ ensure that the pairs dataset was created correctly
  • ✅ only if subset is small enough
  • ✅ Better scale to screen
  • ✅ HTML default hover tooltips
• ✅ Remove nodes with substring "​​Mus musculus"a
• ✅ Basic layout, horizontal scroll to include sugiyama
• ✅ Basic filtering given a root node
  • ✅ On click
• ✅ Sapiens as an example CL:000000 list of nodes for a dataset
• ✅ xyz See if we can remove hundreds of edges that skip over children nodes to terminal nodes
  • ✅ consider cells x y z, where x → y → z, remove links from x to z, so, if a descendant of x has a link to z, remove the link
  • ✅ Try only remove xyz links for nodes with out degree above a certain amount?
• ✅ Sugiyama from arbitrary root (cell) given a max tree size
• ✅ Affordances in drawer for many features above to make them accessible
• ✅ Canvas scaling
  • ✅ Account for DPR / retina 
  • ✅ Correctly clear the canvas per frame using context.save() and context.restore()
  • ✅ Mapping mouse position correctly
    • ✅ Does the canvas know the mouse position
    • ✅ Does d3 simulation know the mouse position
• ✅ Color the cell type graph given UBERON categories, demonstrate that cells are addressable in this way
  • ✅ Autocomplete for arbitrary uberon term

Graph load and graph layout are very slow. It can take minutes in some cases, 10's of seconds commonly.

Areas of work needed:

• createDag/createNodesLinksHulls
• layout time
• rendering time (ticked())

In addition to speedup, it may be beneficial to move some of it out of the main thread.

The force graph rendered in canvas is not responsive. This is a problem because it does not properly utilize the available space, which is determined by the available area and the size of the graph to be displayed. The implementation must:

• 'fit' the browser (target: 1280px non retina laptop --> 4k large monitor)
• 'fit' the currently displayed ontology (target: <10 celltypes --> >3000 cell types)
• ensure that mouseover continues to be pixel accurate at all scales
• ensure that click continues to be pixel accurate at all scales
• ensure that retina and non retina both display properly
• ensure that nodes are not 'fuzzy' because of canvas scaling issues

Update to a version of craco that is compatible with CRA 5.

Example: https://www.npmjs.com/package/@craco/craco/v/7.0.0-alpha.3

Details: dilanx/craco#378

I'm seeing scrollbars even when they are not needed.

In this snapshot, there are not active scrollbars and the lsb fits in the screen. But it still displays the scroll gutters.

Leads:

• https://github.com/hpcc-systems/hpcc-js-wasm#graphviz
• https://github.com/magjac/d3-graphviz

Repro:

1. search
2. hover
3. node clears

a fix for this will allow search even during initial layout, which is currently disabled

There are a bunch of packages which are outdated and need updating. This issue tracks the list, and any blockers that we need to do first.

• react-router-dom -- update to v6 blocked by converting the app to use functional components. V6 requires hooks, so rather than wrapping every class-based component in a HOC, it is easier to just convert, then update the router version. Once we update to functional components, it is easy to migrate from <Switch> to <Route> and the V6 useParams() hook.
• react-scripts (CRA )- update to v5 blocked by this CRA issue. TL;DR - v5 implies Webpack v5, which no longer polyfills node API (eg, fs). We have at least one package (d3-dag, which imports javascript-lp-solver) which somewhat carelessly imports fs and client_process. The aforementioned GH issue has several work-arounds, but all are gross [an example].
• react and react-dom v18
• Blueprint 4.X

fix scrollbar (cellxgene style full height responsive page with single pixel border)

Presently, the force layout resets quite a bit when user input which causes the graph to re-render, such as 'draw hulls', are activated. This likely means we need to tweak alpha to encourage more shifting around ahead of time.

The tradeoff here is that if this means the graph wiggles longer, it may be longer until we can enable controls which require the graph to converge

Or, we can be more clever about how we pass state to the canvas drawing function.

https://github.com/d3/d3-force#simulation_alphaDecay

From the docs:

If decay is specified, sets the alpha decay rate to the specified number in the range [0,1] and returns this simulation. If decay is not specified, returns the current alpha decay rate, which defaults to 0.0228… = 1 - pow(0.001, 1 / 300) where 0.001 is the default minimum alpha.

document and formalize the DAG route params and query params, and their side effects.

First cut (param names may change):

• / - display entire DAG
• /<termID> - display entire DAG, with pinned (aka selected).
• ?subtreeRootId=<termId> - one or more roots for the DAG. DAG is subset to all of these termIds.
• ?searchString=x - highlight all terms in the current ontology/graph that match the free-text search string 'x'. Only one query param allowed.
• ?part_of=x - highlight matching terms based on part_of relationship

In cases where there are very rapid updates (eg, the user moves the mouse rapidly through many graph vertices), the left side bar updates need to be debounced.

To facilitate sharing screenshots and using the ontology viewer for publications, better, sharper hulls and layout will help.

For example, via: https://www.sciencedirect.com/science/article/abs/pii/S0092867421005018

h/t Norbert Tavares