jerdra / niviz-rater Goto Github PK

A summary statistics page will enable users to identify recurring QC trends in their dataset and, in addition, allow aggregating multi-user QC ratings to assess inter-rater reliability. The following requirements should be met:

Back-end

1. An HTTP endpoint to fetch the summary data statistics (# per category, total numbers of fails/passes/idks
2. Would be a good idea to keep in mind #8

Front-end

1. A big summarize button
2. Provide charts and numbers summarizing the current state of QC

The following values would be useful to display:

1. Total number of entities (a single QC item)
2. Number passed
3. Number failed
4. Number uncertain
5. Number un-rated
6. For each component (i.e anatomical, fmri), show the numbers/percentages of scans that fall under a user-defined category.

Ideally with multiple users in the database we should present:

1. An agreement matrix
2. A visual showing rater agreement for each QC item

The rater is still in the development sphere where it creates a .db file in the current directory - this is obviously not ideal.

Instead there should be a cmdline argument where the user may provide authentication and/or an endpoint for where the database should be created. Alternatively, Niviz may write to an existing database instead.

The feature requirements are as follows:

1. User may provide a local path, in which case a .db file will be outputted using SQLite, this is meant for local sessions only
2. A user may provide a database connection string, in which case we should require auth info to establish a connection. Niviz can look for an existing table in that database or it should create it's own if none exist
3. A user may choose not to save a persistent database, in which case an in-memory DB should be used (sqlite3)

During the QC process users may choose to add new rating categories that they discover. However, with the current setup of inputting a YAML config file into the program this is fixed and not allowed. The following requirements should be met when implementing configurable ratings:

Back-end

1. Configuration should be encoded in a DB table. This is preferred over using Bottle.config because if we ever want to work with multiple users then we'd like to allow all users to see the updated available ratings
2. HTTP endpoint should be added to update the configuration table
3. An export functionality should be provided so that users could save their updated YAML file to re-use for other projects or to share. This can be done via another HTTP endpoint (i.e fetch the current schema)

Front-end

1. The QC drop-down selector should have an option to add or remove a rating for a given QC Component (button placed beside selector)
2. This should call the appropriate API endpoint to propagate the requested changes to the database
3. Finally this should trigger a re-fetch of the primary data
4. Provide an export config option to save the current configuration into a YAML file.

Note that when removing a rating one should appropriately handle other QC images that have already been rated using category that is to be removed.

The following documentation components would be good to cover for new users:

Getting started

1. Tutorial dataset (host images on OSF) + expected dataset structure
2. YAML specification for constructing views
3. UI overview (grouping, shortcuts)

Provide python packaging for installing the niviz-rater scripts and dependencies.

See https://github.com/jerdra/niviz as an example for the files needed. Specifically:

1. pyproject.toml
2. setup.cfg
3. setup.py
4. requirements.txt

Are the key files that are needed. These should sit on the top level directory of the github repository. In addition, installation of the niviz_rater package should provide a command-line tool niviz-app that can be used to initialize the application (calling app.py)

Is your feature request related to a problem? Please describe.
Currently when the index tries to build without the --use-existing-index flag it assumes that no DB file exists in the current directory. With --use-existing-index the index build will skip indexing entirely. This means that if a researcher wants to update their existing database it is not possible to do so

Describe the solution you'd like
The --use-existing-index flag should be complimented by a --no-updates flag. This means that by default the index builder will attempt to find new files to update the database with. Using --no-updates should skip this step entirely.

A caveat to this process is that we expect that the user does not change their YAML config file from what is represented in the database. Appropriate handling will be required here (i.e represent the YAML file in the DB and check against it?).

At some point niviz-rater will need to handle multiple users performing ratings on a given study to allow for collaborative QC of imaging data. To this end the database should be extended to store a user-id for ratings. The following requirements need to be met:

1. A login page for the front-end application, the Bottle session will then keep this during the duration of the user-session
2. Any qc ratings applied during this session will have an associated userID
3. Users will only see their own ratings (for now)

Currently the transitions between modals when going from one QC item to another is a bit rough. Need a way to smooth this out using svelte's transitions.

I'm not aware of how to do this off the top of my head at the moment.

There's been user feedback for the implementation of exemplar QC items for each qc category. The purpose of this is to serve as an example of the typical characteristics a certain QC image should have in order to be rated for a given category.

The requirements we hope to meet in this issue are the following:

Back-end

1. Add a database boolean column for is_exemplar in models.py
2. Include this as part of initial data pull from the initial data fetch

Front-end

1. Include an "Mark as Exemplar" in the QC modal interface
2. When an item is an "Exemplar" it should have a visual cue to indicate that it is a defining feature (add a UI element over a tile marked as an exemplar).
3. When sorted by QC category, items marked as an Exemplar should always appear first

Describe the bug
Path provided to -i is used as-is without resolving to full path. As a result users need to use the path they originally gave to the database or otherwise risk images not being served correctly from the fileserver.

Expected behavior
The path fed to -i should be fully resolved so that it is independent of how the user specifies it

niviz-rater was a cow-boy project of trying to solve the problem of QC sucking. As a result, its lacking in structure and testing. This issue is my hopes and dreams of making this a properly set up project. Tests are needed for the following major items:

Backend

1. index.py does it create the correct database when initialized?
2. api.py does it return the correct values from GET requests
3. api.py does it implement the current SQL calls on POST requests

Note that part of this may require some refactoring to make sure that the backend is sufficiently testable

Frontend

Svelte is relatively new territory for me, but this is probably a good place to start:

https://github.com/testing-library/svelte-testing-library

Is your feature request related to a problem? Please describe.
Outputs of BIDS pipelines often-times include a dataset_description.json which instructs pyBIDS on how to define the entities it uses. This means that niviz-rater does not have a constant bids configuration it uses and specification YAMLs are highly dependent on how pyBIDS parses the input file tree.

The resultant behaviour is that niviz-rater sometimes fails to get images because it is unclear what bids configuration is being used - the only way to debug this is to investigate the dataset_description.json before making a .yaml file. This reduces specification YAML re-usability.

Describe the solution you'd like
Niviz-Rater should be wholly independent of the dataset_description.json potentially by forcing an override of PyBIDS' internal configuration and instead favouring the configuration file found in niviz_rater/data/bids.json.

Describe alternatives you've considered
Since we don't actually use many of pyBIDS' features and it is quite slow due to its construction of an index (which we do not actually use), it may be a good idea to remove the dependency entirely. This will also remove other unwanted dependencies as well.

Pagination is needed within each of the Row components so that larger datasets can be handled more elegantly than a large page

niviz-rater enforces a YAML schema for configuring the qc process. The following are captured:

1. Description for how to initialize rows

2. QC components

• Column name (generated from BIDS entities)
• Applicable ratings for a given component
• A set of BIDS descriptors to use when pulling in images for a given component

The following schema generation and validation framework seems like a promising option:

https://github.com/23andMe/Yamale