xaitk / xaitk-saliency-web-demo Goto Github PK

This may be a limitation of the data-passing required for trame, but this issue would be a basic exploration of potential bottlenecks in saliency map visualization (assuming they have already all been computed, and the user is selecting a specific class or detection). This currently occurs with a slight noticeable delay when using the app. Potential solutions should be noted and can be addressed based on an assessment of level of effort.

This issue addresses technical debt on the xaitk-saliency-web-demo, as it has not been updated recently. The latest version of trame and xaitk-saliency can be pulled in, and any dependency-related issues should either be addressed or noted for additional discussion.

The RandomGridStack algorithm (implemented under object detection saliency) requires tuple input for specifying the size of the masking grids. However, the demo only exposes a single integer input, which means the algorithm cannot be configured properly and saliency maps cannot be generated. A new tuple input field should be added whenever the RandomGridStack algorithm is chosen, and the demo app should be tested such that saliency maps can be generated.

Currently, running the notebook-version of the app (versus the native app form) is slow due to the fact that the app runs on CPU only. Similar to the app version, a call to update_ml_device should be used to move the data/models to GPU and enable faster computation.

    altair==4.1.0
    smqtk-classifier==0.19.0
    smqtk-core==0.18.1
    smqtk-dataprovider==0.16.0
    smqtk-descriptors==0.18.1
    smqtk-detection[torch,centernet]==0.19.0
    smqtk-image-io==0.16.2
    xaitk-saliency==0.6.1
    torchvision==0.11.2
    scikit-learn==0.24.2
    scikit-image==0.18.3
    ubelt==1.1.1

@Purg @brianhhu Any suggestion in term of version if we should update any of the listed above?

I followed the instructions of the ReadMe file, but after installing all prerequisites I get the following error, no matter if I use the PyPi package for installation or if I install it from the project directory.

zsh: command not found: xaitk_saliency_demo

What is missing to run the demo?

Currently, clearing the input image does not reset the other fields/visualizations in the app (e.g. model predictions, etc.) Another discovered edge case is clearing the input image for the image classification task, and then switching to the object detection task still shows the predicted bounding boxes (even without any input). It would be nice to clear the relevant data/fields so that the app is ready to be used each time a user clears the input image.

Similar to image classification saliency (which exposes an option for the number of top-K classes to explain), the object detection saliency GUI could have a similar dropdown for customizing the number of top-K detections to compute saliency for. This would make the context between the two tasks more similar for a user.