openimaging / miqa-phase1 Goto Github PK
View Code? Open in Web Editor NEWA web application for medical imaging quality assurance
License: MIT License
A web application for medical imaging quality assurance
License: MIT License
When multiple sessions are loading in an experiment (a feature added in #35), it always starts with the first session, even if you clicked on the last one. Instead, it should queue up the one you clicked first, then the others can load in the normal order.
If we press+hold the single arrow, it should scan as fast as possible through the timesteps in a scan. Eventually the rate at which it scans through the series could be scaled by some user configurable "frame rate".
I have noticed that after scanning through hundreds of timesteps, that the time to move to the next time step slowly grows the more timesteps we view. At first I see a framerate above 10 images/second, but eventually it can take more than a second to move to the next timestep. Reloading the application resets the performance back to expected level.
Use the performance tool in Chrome, it seems the time to switch is partly attributed to vtk.js
setting new inputData on the trivial producer which is the source for the visualization, and partly attributed to vue.js
re-rendering the DOM. As the timestep switching takes longer and longer, that time becomes dominated by vue.js
.
We need to figure out what is causing vue.js
to take longer and longer.
This performance degradation is distinct from that we have seen when loading many experiments, and the load time grows over time. However, the two may be related.
The base image for our Dockerfile is node:10.15.3-stretch
where the python3 in that image is python3.5, which reached end of life September 2020. Even the latest node
image still has python 3.5, so we need to install newer python in the image. We are starting to get errors and warnings when building the docker stack about this old python, such as the following:
/usr/local/lib/python3.5/dist-packages/OpenSSL/crypto.py:14: CryptographyDeprecationWarning: Python 3.5 support will be dropped in the next release of cryptography. Please upgrade your Python.
from cryptography import utils, x509
DEPRECATION: Python 3.5 reached the end of its life on September 13th, 2020. Please upgrade your Python as Python 3.5 is no longer maintained. pip 21.0 will drop support for Python 3.5 in January 2021. pip 21.0 will remove support for this functionality.
The window level sliders should not reset to default values on their own, but stick wherever you put them. Currently they seem to reset when you move to the next timestep in a scan.
As scans usually have different intensity ranges, resetting the window level when navigating between scans is probably fine.
Following the developer setup instructions, the step building the client application (npm run serve
) fails with a bunch of errors like the following:
error in ./src/App.vue?vue&type=style&index=0&lang=scss&
Module build failed (from ./node_modules/sass-loader/lib/loader.js):
Error: Cannot find module 'node-sass'
Require stack:
- /Users/scott/projects/miqa/miqa/client/node_modules/sass-loader/lib/loader.js
- /Users/scott/projects/miqa/miqa/client/node_modules/loader-runner/lib/loadLoader.js
- /Users/scott/projects/miqa/miqa/client/node_modules/loader-runner/lib/LoaderRunner.js
- /Users/scott/projects/miqa/miqa/client/node_modules/webpack/lib/NormalModule.js
- /Users/scott/projects/miqa/miqa/client/node_modules/webpack/lib/NormalModuleFactory.js
- /Users/scott/projects/miqa/miqa/client/node_modules/webpack/lib/Compiler.js
- /Users/scott/projects/miqa/miqa/client/node_modules/webpack/lib/webpack.js
- /Users/scott/projects/miqa/miqa/client/vue.config.js
- /Users/scott/projects/miqa/miqa/client/node_modules/@vue/cli-service/lib/Service.js
- /Users/scott/projects/miqa/miqa/client/node_modules/@vue/cli-service/bin/vue-cli-service.js
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:982:15)
at Function.Module._load (internal/modules/cjs/loader.js:864:27)
at Module.require (internal/modules/cjs/loader.js:1044:19)
at require (internal/modules/cjs/helpers.js:77:18)
at Object.sassLoader (/Users/scott/projects/miqa/miqa/client/node_modules/sass-loader/lib/loader.js:46:72)
@ ./node_modules/vue-style-loader??ref--8-oneOf-1-0!./node_modules/css-loader??ref--8-oneOf-1-1!./node_modules/vue-loader/lib/loaders/stylePostLoader.js!./node_modules/postcss-loader/src??ref--8-oneOf-1-2!./node_modules/sass-loader/lib/loader.js??ref--8-oneOf-1-3!./node_modules/cache-loader/dist/cjs.js??ref--0-0!./node_modules/vue-loader/lib??vue-loader-options!./src/App.vue?vue&type=style&index=0&lang=scss& 4:14-415 14:3-18:5 15:22-423
@ ./src/App.vue?vue&type=style&index=0&lang=scss&
@ ./src/App.vue
@ ./src/main.js
@ multi (webpack)-dev-server/client?http://localhost:8081/sockjs-node (webpack)/hot/dev-server.js ./src/main.js
Complete error log attached.
Is Admin one person or multiple people?
overall_QA: do multi-class instead of regression?
Correlate NN output with IQM metrics
Correlate human ratings with IQM metrics
Weighted random sampler for PyTorch link
Lower learning rate? Other hyper-parameter tuning
focal loss?
Learning rate decay? Or other LR schedules? Cosine
MIQA should continue to support importing the CSV format used previously.
Since #27 adds a conversion script to the repo already, it shouldn't be too hard simply to invoke that functionality internally if the user tries to load a csv file instead of the new json format.
The first QC related data needs to be shared for development purposes by September 15
MIQA uses v-mousetrap
from vue-utilities
, and actually moved that library code into the application since it's no longer being maintained at its previous home.
Debugging performance degradation over many experiment loads, I discovered over 100K listeners bound to the DOM, most of which were created by `v-mousetrap. In this branch commit, I experimented with just removing the use of that library, and the number of listeners stopped growing without bound, and experiment loading performance was greatly enhanced.
However, key bindings are an important feature in the application, so we need to either fix v-mousetrap
, or else replace it with something else.
Currently we have to import a whole new csv or json, which moves the currently imported session to a backup folder in girder, in some way losing the annotations made up to that point.
Instead we should support adding one or more scans or experiments to the existing session.
Whether this is done via some new UI element, or via a script run by an admin is still uncertain.
This should be fixed upstream, see girder/girder_web_components#306
Currently it just shows "N failed of M" but does not give a reason for the failure. Provide more information, perhaps the error stack trace directly.
Another option is to provide an error ID which can be queried in the server log by admins, but that is probably overkill for now.
We now have the ability to augment the JSON dataset spec with Quality Control metrics, currently attributes are added to a 'volumes' tag at the scan level. Do we want to rename / relocate this in the JSON and where should we load this data into MIQA. For example, in an annotations table with a foreign key to an image table?
Using the WorkerPool, we can save the repeated loads of the heavy javascript (emscripten wrapped C++ library methods) associated with the imageio worker. Instead of reloading those 3 heavy files for each image we need to read, we can just load them as many times as we have workers in the pool, which should provide much faster experiment loading.
This is under development in #39
Hans said there is a minor update to data, so I should probably wait for that.
Kilian Pohl reported that sometimes clicking the single arrow (either back or forward) skips one or more images in a time series. We need to make sure when we click the single arrows we never skip over a timestep. It's fine if the slider does that because the user is probably trying to skip directly to some timestep, but the single arrow needs to visit every step.
So far, I have not been able to reproduce this issue, either on the SRI instance or on the application running on my machine, so more work is required to identify the problem.
4/14 discussion that UI will show a "time remaining" when the timeout time is approaching. System will not auto-renew timeout for initial version, but this is enough to warn the user for the first version.
Currently the application fetches and caches only the timesteps for the current scan. We should increase to caching the entire current session. It would also be nice to show some indication in the UI of which scans are currently in memory.
We can evaluate how this works in practice (does it put too much memory pressure on the application), and perhaps eventually allow the user more control of which images are loaded.
Keep track of which scans are reviewed, which remain, and who has been informed of QA activity is currently done largely through email. This process might be streamlined if it could be managed better through MIQA directly. Explore what functionality we want and how to add or incorporate an existing system.
As a workaround, we have left collaborators with read only access and we just don't show them and "edit" type widgets in the UI, because none of those would work for them. Fixing this in the long run seems like it will involve at the least reorganizing how metadata is stored (currently notes and ratings are both part of generic metadata attached to the scan folder), but might be even more challenging given how Girder groups work in version 3.
The biggest complaint here is that the % loaded lives adjacent to the timestep slider, seeming to indicate that the % loaded has something to do with the which timestep is being viewed in the current scan.
The lower panels need a little reorganization in general, and this should include moving the percent loaded indicator closer to the metadata, and best would be to associate it with the experiment id in some way to help indicate that it's the amount of the experiment that is loaded.
Do #89 first and see if the link is now useful before removing.
When users create accounts, they are asked to check their email to confirm their email address, but the email never arrives, and then the administrator needs to explicitly activate their account for them.
User should need to log back in after a fixed time since login expires.
Design the next version of the MIQA application. The output is a design document that discusses the motivations for inclusion and exclusion of features in MIQA. This effort may also include restricted scope prototypes as needed.
This could be a very involved or quite simple issue, depending on what we intend to do in CI. At a minimum, we should decide on client/server code styles we want to enforce, and at least make sure contributors are adhering to those.
More involved would be deciding how we want the project to be tested, choosing and implementing a testing framework, and writing tests to cover the application code.
The phase2 team needs to be added with developer privileges so that we may more efficiently manage the issues.
The single arrows should not, when the end (or beginning) of time series is reached, automatically skip to the next scan. Instead these arrows should become greyed out (the left one when the beginning is reach, the right one when the end is reached) to indicate to the user there are no more timesteps in that direction.
The only ways to skip to the next (or previous) scan should be via 1) clicking the double arrow, or by 2) annotating the scan.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.