Environment

• Release number or git hash: 9.1.2-20221027-011632
• Browser: Chrome 106.0.5249.119
• Deployed / Local: deployed

Steps to reproduce the bug

1. Add an aggregated score of type boxplot

Observed Behavior

Domain values remain at 0 to 100, which causes the boxplots to be extremely skewed and illegible.

Expected Behavior

Boxplot visualizations are properly displayed

This might be related to #204 that fixed domain values for numeric scores

The new h38 release requires Postgres 10. We need to update the docker-compose.partial.yml.

We should use the latest versions both for requirements and dependencies.
Except for Typescript, tslint and d3 dependencies, which will be updated in a separate step.

• Release number or git hash: f045a07
• Web browser version and OS: Chrome 77
• Environment (local or deployed): local and deployed

Steps to reproduce

1. Open https://ordino.caleydoapp.org
2. Select Cancer Gene Census
3. Select one or multiple genes
4. Open Copy Number view
5. Filter for Organ: bladder (or any other filter)
6. Apply filter

Same behavior applies when starting with cell lines and switching to genes.

Observed behavior

• Shows an error message
• The selected gene is not added as score column

Expected behavior

• No error message and the score column for the filtered rows should be added

@zichner commented on Mon Oct 02 2017

Release: dev and stable

Description
Combined view only shows NaN for tissue samples

Steps to reproduce

1. Open list of all TCGA Tumor samples
2. Select multiple samples
3. Open Combined View
4. Select TPM and Relative Copy Number as data subtype

-> Only empty columns are shown (all values are NaN).

@zichner commented on Thu Oct 12 2017

If a filtered list contains less than 1000 entities, scores are only computed for this filtered subset. A checkbox was added to allow the user to still compute scores for all entities (#471).

By default, this checkbox should be ticked, i.e., scores should be computed for the entire list, since users seem to get confused if scores are missing after removing a filter criteria.

@zichner commented on Fri Oct 13 2017

It might also be good to invert the logic of the checkbox, so instead of ticking it to compute on all entities, one ticks it to only compute on current subset.

With this issue the PRISM drug screen data (https://depmap.org/portal/prism/) should be added as a cell line score in Ordino.

User path

1. Open list of cell lines
2. Add column menu
3. Select PRISM Drug Screen Score
4. Select one or multiple drugs via auto complete
5. Select one or multiple scores (Activity Area, IC50, or EC50)
6. Add one or multiple score columns

Integration details

• The data is already in our DB
  • We must add the related SQL schema as Alembic revision
  • Corresponding views: cellline.tdp_drugscore and public.tdp_drug
• For now, there can be a dedicated entry in the add column dialog (e.g., PRISM Drug Screen Score); later there will likely come more drug screen data sets and an additional selection will be needed
  • This score is similar to the Depletion Screen Score for genes
• When clicking on PRISM Drug Screen Score a dialog should open, where one selects
  • one of the several hundred drugs. Ideally, for each drug, the name, the mode-of-action, and the target gene(s) should be listed in the auto-complete
  • one of the three available scores (Activity Area, IC50, or EC50)

Implemenation pointers

• SQL queries:

  • tdp_publicdb/tdp_publicdb/data.py

    Lines 11 to 16 in a2d2db6

    	depletion_tables = ['depletionscore']
    	depletion_attributes = ['rsa', 'ataris', 'ceres']
    	cellline_data = DataEntity(cellline.schema, tables, attributes, operators)
    	tissue_data = DataEntity(tissue.schema, tables, attributes, operators)
    	cellline_depletion = DataEntity(cellline.schema, depletion_tables, depletion_attributes, operators)

  • tdp_publicdb/tdp_publicdb/sql.py

    Lines 35 to 37 in b9166e4

    	# depletion scores
    	create_gene_sample_score(views, gene, cellline, cellline_depletion, 'depletion_', callback=lambda x: x.filter('depletionscreen'))
    	create_gene_sample_score(views, cellline, gene, cellline_depletion, 'depletion_', inline_aggregate_sample_filter=True, callback=lambda x: x.filter('depletionscreen'))

• Score dialog

  • tdp_publicdb/src/config.ts

    Lines 367 to 401 in 920738f

    	export const depletion: IDataTypeConfig = {
    	id: 'depletion',
    	name: 'Depletion Screen ',
    	tableName: 'depletionscore',
    	query: 'depletion_score',
    	dataSubtypes: [
    	{
    	id: 'rsa',
    	name: 'DRIVE RSA (ER McDonald III et al., Cell, 2017)',
    	type: dataSubtypes.number,
    	domain: [-3, 3],
    	missingValue: NaN,
    	constantDomain: false,
    	useForAggregation: 'rsa'
    	},
    	{
    	id: 'ataris',
    	name: 'DRIVE ATARiS (ER McDonald III et al., Cell, 2017)',
    	type: dataSubtypes.number,
    	domain: [0, 10000],
    	missingValue: NaN,
    	constantDomain: false,
    	useForAggregation: 'ataris'
    	},
    	{
    	id: 'ceres',
    	name: 'Avana CERES (Robin M. Meyers et al., Nature Genetics, 2017)',
    	type: dataSubtypes.number,
    	domain: [0, 10000],
    	missingValue: NaN,
    	constantDomain: false,
    	useForAggregation: 'ceres'
    	}
    	]
    	};

  • tdp_publicdb/src/scores/SingleScore.ts

    Lines 153 to 160 in c853e45

    	// Factories for depletion scores for DRIVE data
    	export function createSingleDepletionScoreDialog(pluginDesc: IPluginDesc, extra: any, countHint?: number) {
    	return createScoreDialog(pluginDesc, extra, FORM_SINGLE_SCORE_DEPLETION.slice(), countHint);
    	}
    	export function createSingleDepletionScore(data: ISingleScoreParam, pluginDesc: IPluginDesc): IScore<number>|IScore<any>[] {
    	return initializeScore(data, pluginDesc, (parameter, dataSource, oppositeDataSource) => new SingleDepletionScore(parameter, dataSource, oppositeDataSource));
    	}

  • tdp_publicdb/src/scores/SingleScore.ts

    Lines 49 to 63 in c853e45

    	class SingleDepletionScore extends ASingleScore implements IScore<any> {
    	constructor(parameter: ISingleScoreParam, dataSource: IDataSourceConfig, oppositeDataSource: IDataSourceConfig) {
    	super(parameter, dataSource, oppositeDataSource);
    	}
    	protected getViewPrefix(): string {
    	return 'depletion_';
    	}
    	protected createFilter(): IParams {
    	return {
    	depletionscreen: this.dataSubType.id === 'ceres' ? 'Avana' : 'Drive'
    	};
    	}
    	}

  • tdp_publicdb/phovea.js

    Lines 544 to 559 in c341197

    	registry.push('tdpScore', prefix + '_depletion_single_score', function () {
    	return import ('./src/scores/SingleScore');
    	}, {
    	name: 'Depletion Screen Score (Single)',
    	idtype: idType,
    	primaryType: idType,
    	oppositeType: 'Ensembl',
    	factory: 'createSingleDepletionScoreDialog'
    	});
    	registry.push('tdpScoreImpl', prefix + '_depletion_single_score', function () {
    	return import ('./src/scores/SingleScore');
    	}, {
    	factory: 'createSingleDepletionScore',
    	primaryType: idType,
    	oppositeType: 'Ensembl'
    	});

Definition of done

• SQL schema for the new views is added as alembic revision
• Score must be listed in the add column dialog
• Score dialog allows multi-selection of drugs and attributes/scores
• Score adds multiple columns (drugs x scores)
• Scores show the correct results for the corresponding cell lines

See https://github.com/datavisyn/visyn2020/issues/40

Summary

• enable gif files where necessary

We should update the template files with the new generator-phovea v2 to update the dependencies and use Node.js 12.

@zichner commented on Wed Oct 11 2017

Release: dev

Steps to reproduce

1. Open list of all cell lines
2. Select A549
3. Open Copy Number detail view (-> Copy number values are shown)
4. Filter detail view by Predefined Named Set normal chromosome protein coding human genes
   -> No copy number information is shown

I guess this is due to a bug in the SQL statement:

SELECT d.ensg AS id, d.relativecopynumber AS score 
FROM cellline.tdp_copynumber d 
INNER JOIN cellline.tdp_cellline s ON d.celllinename = s.celllinename 
INNER JOIN public.tdp_gene g ON d.ensg = g.ensg 
WHERE g.species = 'human' 
AND d.celllinename = 'A549' 
AND d.celllinename = ANY(ARRAY(SELECT celllinename FROM cellline.tdp_panelassignment WHERE panel = 'normal chromosome protein coding human genes'))

The last part should filter genes but is filtering cell lines.

Please check again the code that builds the SQL queries very thoroughly to avoid these kind of bugs (the SQL queries are one of the most crucial parts of the application).

PS: Interestingly, filtering for predefined named sets is working in the Combined View. However, shouldn't both SQL queries be almost identical?

@zichner commented on Tue Sep 05 2017

Steps to reproduce

1. Open list of "Cancer gene census" genes
2. Add aggregated cell line score column
   • Tumor type: bladder carcinoma
   • Data type: AA Mutated
   • Aggregation: Frequency

-> Column header is "AA Mutated <0 Frequency", but should be "AA Mutated Frequency" (the "<0" should be removed)

Also the column header tool tip contains "Frequency: <0", but should be "Aggregation: Frequency"

• Release number or git hash: de96c5c
• Web browser version and OS: Chrome 91, Linux
• Environment (local or deployed): both

Steps to reproduce

1. Open https://ordino-daily.caleydoapp.org/#/
2. Go to Onboarding Tours
3. Select Add Column to GeneList Tour

Observed behavior

• In one step, the tour fails because one of the root selectors has been changed from col-sm-auto to col-sm-12

Expected behavior

• The tour works without any errors

We should migrate from Karma to Jest for testing due to compatibility issues.

After the update to Typescript 3.8, we can use the latest version.
For the usage of ts-jest it is important to use a pinned version without ^. The reason for that is that its Major version follows Jest, which means that minor changes can be breaking. See the docs.

Observed behavior

• This repository cannot be published to npm, due to this flag in package.json:

"private": true,

Expected behavior

• Option should be set to false in order to be able to publish it to npm.

Observed behaviour

• Build ID in webpack config causes errors if it is used without JSON.stringify()

Coming from https://github.com/Caleydo/tdp_bi_bioinfodb/issues/1085

As far as I can see, AA mutated can currently only be Non Mutated, Mutated and Unknown, i.e., there is no null. All values that are either null in the DB or are not existing in the DB are set to Unknown.

I think this should be changed in a way that everything which is currently Unknown (either null in DB or not in DB) becomes null in Ordino and gets shown as a dash.
This should be the case for AA mutated, DNA mutated, as well as Copy Number Class.

Checklist

• Create session that contain an Unknown filter before the refactoring
• Refactor code
• Check if existing and new provenance graphs still work after the refactoring

Select2 and Select3 have different placeholder titles, which becomes especially obvious when opening a dialog. The task is to unify the placeholders.

Coming from PR #63 we should add Alembic for SQLAchemy. The first alembic revision should create the initial Ordino database schema without data.

Tasks

• Connect to Ordino public DB
• Export schema pg_dump -U postgres -s postgres > exportFile.dmp (see https://www.postgresql.org/docs/10/app-pgdump.html and https://stackoverflow.com/a/31602990)
• Initialize alembic
• Write initial alembic revision that creates the schema in the upgrade method
• Always add IF NOT EXISTS to prevent errors due to duplicates
• Write downgrade method that drops tables and views
• Comment the content of downgrade so that we do not delete data by accident (but if somebody wants to use it, the method is ready to use)
• Test it

Definition of Done

• Creating a new Ordino instance and running alembic head should create the schema in the local docker publicdb container
• Ordino should start (with no data)

Observed behavior

Updating the typescript and typedoc versions according to the snippet below without using stricter linting makes it possible to use the current typescript version with a minimum amount of errors.

"typedoc": "~0.16.9",
"typescript": "~3.8.1-rc",

Expected behavior

Updating the dependencies in the package.json and adding the following line of code in tsconfig.json (as well as fixing appearing type errors) should be sufficient for the usage of Typescript 3.8 as a first step.

"downlevelIteration": true, // required as long as target is `es5`

@awernibi commented on Thu Sep 07 2017

Storing depletion screen data require a DB update, with the following SQL script.

set search_path = cellline,public;

DROP table PROCESSEDDEPLETIONSCORE CASCADE;
DROP table DEPLETIONSCREEN;

create table DEPLETIONSCREEN (
DEPLETIONSCREEN      TEXT                 not null,
DEPLETIONSCREENDESCRIPTION TEXT                 null,
constraint PK_DEPLETIONSCREEN primary key (DEPLETIONSCREEN)
);

create table PROCESSEDDEPLETIONSCORE (
ENSG                 TEXT                 not null,
CELLLINENAME         TEXT                 not null,
DEPLETIONSCREEN      TEXT                 not null,
RSA                  FLOAT4               null,
ATARIS               FLOAT4               null,
constraint PK_PROCESSEDDEPLETIONSCORE primary key (CELLLINENAME, ENSG, DEPLETIONSCREEN)
);

alter table PROCESSEDDEPLETIONSCORE
   add constraint FK_PROCESSE_REFERENCE_DEPLETIO foreign key (DEPLETIONSCREEN)
      references DEPLETIONSCREEN (DEPLETIONSCREEN)
      on delete restrict on update restrict;

alter table PROCESSEDDEPLETIONSCORE
   add constraint FK_PROCESSE_REFERENCE_GENE foreign key (ENSG)
      references GENE (ENSG)
      on delete restrict on update restrict;

alter table PROCESSEDDEPLETIONSCORE
   add constraint FK_PROCESSED_DEPLETION_CELLLINE foreign key (CELLLINENAME)
      references CELLLINE (CELLLINENAME)
      on delete restrict on update restrict;

CREATE INDEX idx_processeddepletionscore ON cellline.processeddepletionscore(ensg);

CREATE VIEW cellline.processeddepletionscoreview AS
SELECT d.ensg, symbol, celllinename, depletionscreen, rsa, ataris FROM cellline.processeddepletionscore d
JOIN gene g ON (d.ENSG = g.ENSG);

The Novartis Drive data set (http://dx.doi.org/10.1016/j.cell.2017.07.005) can be found in the AWS ordino database. In order to have a local version, copy the content of tables DEPLETIONSCREEN and PROCESSEDDEPLETIONSCORE from db schema cellline (~2.7 million rows).

@awernibi commented on Fri Sep 08 2017

Two views have been created:

CREATE VIEW cellline.tdp_depletionscreen AS
  SELECT depletionscreen, depletionscreendescription
  FROM cellline.depletionscreen;

CREATE VIEW cellline.tdp_depletionscore AS
  SELECT ensg, symbol, celllinename, depletionscreen, rsa, ataris
  FROM cellline.processeddepletionscoreview ;

@zichner commented on Fri Sep 15 2017

In order to make this data available in Ordino, please add the two menus:

• Aggregated RNAi Screen Score
• Single RNAi Screen Score

to the gene and cell line list.

The Score dialogs should (for now) be analog to Aggregated Cellline Score/Aggregated Gene Score as well as Single Cellline Score/Single Gene Score.

The Data Types should be DRIVE RSA as well as DRIVE ATARiS. The actual data is stored in the DB view cellline.tdp_depletionscore (as described above).

• Release number or git hash:
• Web browser version and OS:
• Environment (local or deployed): deployed master (and probably dev)

Steps to reproduce

0. Open dTiles (dtiles.app.datavisyn.io)
1. Select 1 gene
2. Open the Co-Expression view (the view should show the message "Select two or more genes")
3. Select another gene (the Co-Expression view renders the plots)
4. Add another view and split them

Observed behavior

• The Co-Expression view shows the initial message ("Selecct two ...") again
• When changing the view's parameters, the view rerenders again

Expected behavior

• It should render the same plots as before

We should prepare a pgAdmin container for development purposes in the docker-compose.partial.yml. The container should be commented out by default, so that developers can activate them on demand.

In the following a sample config that can be adapted.

  pgadmin:
    links:
      - db_publicdb:publicdb
    image: dpage/pgadmin4:latest
    ports:
      - "5050:80"
    environment:
      - PGADMIN_DEFAULT_EMAIL=admin
      - PGADMIN_DEFAULT_PASSWORD=admin
    networks:
      - db_publicdb

See https://github.com/datavisyn/visyn2020/issues/3

We should update all dependencies to the latest accessible versions, except for:

• all webpack dependencies
• all test dependencies
• d3 and bootstrap

It might be necessary to update jQuery and React in a separate step as well.

Environment

• Browser: Edge chromium
• deployed/local: https://ordino-daily.caleydoapp.org/

Steps to reproduce the bug

1. Open list of all genes
2. Select one gene
3. Open "PubMed" detail view

->Error: pubmed.ncbi.nlm.nih.gov refused connection

If showing PubMed results within Ordino is not possible anymore, please modify the detail view in a way that it just provides a link (analog to "Human Protein Atlas" detail view).

• Release number or git hash: 78f859a
• Web browser version and OS: Chrome 103, Ubuntu 22.04
• Environment (local or deployed): local

Steps to reproduce

1. Setup dTiles workspace using yo phovea:setup-workspace datavisyn/dTiles_product -b develop_aws
2. Run npm start and docker-compose up
3. Add dburl in config.json

Observed behavior

• localhost:8080 only shows a blank screen and the following error in the console:
  

Expected behavior

• dTiles runs normally

We can remove dTiles from the dependencies in the package.json since it is not used at the moment and has no priority for deployment.

• Release number or git hash: v7.0.0
• Web browser version and OS: Chrome 84
• Environment (local or deployed): both

Steps to reproduce

1. Upload a CSV or XSLX that contians boolean values

Observed behavior

Uncaught (in promise) TypeError: v.trim is not a function
    at Object.detectIDType (GeneSymbolDetector.ts:9)
    at valuetype_idtype.ts:98
    at Generator.next (<anonymous>)
    at fulfilled (tslib.es6.js:71)

Uncaught (in promise) TypeError: v.trim is not a function
    at Object.detectIDType (IDTypeDetector.ts:19)
    at valuetype_idtype.ts:98
    at Generator.next (<anonymous>)
    at fulfilled (tslib.es6.js:71)

Expected behavior

The detectors should check for strings before using trim().

http://cancer.sanger.ac.uk/cell_lines/sample/overview?id=<COSMICID>&genome=38 (change genome=37 to genome=38)

Environment

• Release number or git hash: 7de3f13
• Browser: Chrome 103
• Deployed / Local: both

Steps to reproduce the bug

1. Open https://ordino-daily.caleydoapp.org/#/
2. Click Start Analysis
3. Open normal chromosome protein coding human genes
4. Add a Cell Line Score (Single) column
5. Select any cell line
6. Select any data type

Observed Behavior

• The operation is aborted and the user comes back to the dataset cards

Expected Behavior

• score column is added to the ranking normally

The new h38 release requires Postgres 12. We need to update the docker-compose.partial.yml.

@zichner commented on Thu Sep 14 2017

Currently, we have 4 types of mutation information:

• DNA Mutated
• DNA Mutation
• AA Mutated
• AA Mutation

Please add a fifth one: Zygosity

The corresponding information is stored in the cellline / tissue view tdp_mutation

See https://github.com/datavisyn/visyn2020/issues/9

We should update all requirements to the latest accessible versions.

@zichner commented on Tue Oct 24 2017

Release: dev

When building an SQL query for a cell line list score that aggregates across a set of genes, the gene table is join by an INNER JOIN whereas the gene set table is joined by a LEFT JOIN.

Here is an example:

/api/tdp/db/publicdb/cellline_gene_frequency_copynumberclass_score/score?attribute=copynumberclass&species=human&table=copynumber&target=Cellline&value=-2&filter_panel_ensg=Cancer+Gene+Census

SELECT d.celllinename AS id, SUM((copynumberclass in (-2))::INT4) as count, COUNT(copynumberclass) as total 
FROM cellline.tdp_copynumber d 
INNER JOIN public.tdp_gene s ON d.ensg = s.ensg 
LEFT JOIN public.tdp_geneassignment ga ON (d.ensg = ga.ensg)  
WHERE s.species = 'human' AND ga.genesetname = 'Cancer Gene Census' GROUP BY d.celllinename;

Is there a reason for using different joins?
If not, please change the second join also into an INNER JOIN, which is much faster in some cases.
Additionally, you can harmonize the ON statement (either use always brackets or never).

@zichner commented on Mon Oct 30 2017

Just FYI: This issue is the most relevant at the moment. As soon as this change is made and deployed, I can start checking the DB queries for correctness and we can see if all steps of the described use cases are running fast enough or if we need to modify the DB queries or the use cases.

@mstreit commented on Mon Oct 30 2017

We discussed it this morning. @lehnerchristian will take care of it today.

Environment

• Browser: Firefox, Edge Chromium
• deployed/local: Ordino-daily

Steps to reproduce the bug

1. Start Ordino Welcome Tour and click through it

-> in step 5, only the gray overlay is visible and the dialog is not shown anymore. Consequently, Ordino is completely non-responsive.

I guess the problem is caused by changes in LineUp v4