Describe the bug
The learning rate is not changed when using a config file to choose the learning rate scheduler.

Expected behavior
Change the learning rate according to the chosen learning rate scheduler.

Screenshots

Desktop (please complete the following information):

• OS: [e.g. Linux, Windows, macOS]
• DPAT version 4.1.0

Additional context
Using the varmil.yaml config file with

  model:
    scheduler:
      class_path: torch.optim.lr_scheduler.CosineAnnealingLR
      init_args:
        T_max: ${trainer.max_epochs}

Research question
What is the optimal number of prototypes for the SwAV model?
What is the optimal number of hidden dimension parameters?
What is the optimal number of output dimension parameters?

Hypothesis
prototypes
The number of prototypes that works best for classifying medullo/pilo = 6.
Reason: SwAV was previously trained on imagenet with 3000 prototypes for 1000 classes. Scaling it down linerarly to 2 classes means 6 prototypes.

hidden
?

output
?

Method
Perform a search with the multivariate Tree-structured Parzen Estimator algorithm to find the lowest loss in the loss landscape spanned by the SwAV parameters.

Why is this experiment worthwhile?
Presumably, the higher the contrast between tile representations, the better stage 2 will classify tumours.

Note
Probably, the choice of hidden en output dimension influence the optimal number of prototypes.

Result comments on this should include results and a discussion.

Describe the bug

OSError: cannot load library '/scistor/guest/sjg203/.conda/envs/pmchhg/lib/libvips.so.42': /cm/local/apps/gcc/10.2.0/lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /scistor/guest/sjg203/.conda/envs/pmchhg/lib/./libpoppler-glib.so.8)

Desktop (please complete the following information):

• OS: Linux
• DPAT version: 2.1.6

additional context

import pathlib

from torchvision.transforms import Compose, ToTensor

from dpat.data import PMCHHGImageDataset, H5Dataset 
from dpat.extract_features.models import SwAV

The torchmetrics package is not used efficiently. Special output objects are used while torchmetrics might help out here.

Replace lines such as

self.validation_output["target"].append(y)
self.validation_output["prediction"].append(self.softmax(y_hat)[:, 1])
self.validation_output["loss"].append(loss)

and the log_metrics() function.

v2.3.3

The images of the pmc-hhg project are RGB. However, the 2PEF (blue) and SGH (red) channels physically yield less information about the diseases in question. It is assumed that the most useful pathological information is embedded in the THG (green) channel, as that is the channel where individual cells are present.

To test for this, the whole experiment can be done 2 times: with RGB and grayscale images. Best image wins.

Why to RGB? Because ShuffleNetV2 and other SOTA models only accept RGB images.

This can be done in a transform:

class M2Gray2RGB:
    """Converting a many-channel image to grayscale image to RGB image"""
    def __init__(self, channel):
        """channel = channel to copy to other channels"""
        self.channel = channel
    def __call__(self):
        # get THG channel from self.channel arg
        # repeat THG channel to RB channels with torch.Tensor.repeat()
       return three_channel_gray

Is your feature request related to a problem? Please describe.
BERT in CCMIL is not trainable. This defeats the purpose of BERT. It can be instantiated from the ClinicalBERT pretrained on clinical and biological text and further finetuned for the Attention MIL Classification task.

Describe the solution you'd like
Allow BERT model to update weights during training and fix them during evaluation/testing.

Additional context
Finetuning makes for the model to better map the combination of the string "brain location" and the image of medulloblastoma to the classification "medulloblastoma". Otherwise, only the classifier is trained. The finetuned BERT model may also be able to aid in classifying.

This should be tested thoroughly. The model should not classify images as medulloblastoma just because some location was presented to BERT. The MIL aggregate should also be of great importance.

Is your feature request related to a problem? Please describe.
Currently, training has to be started with python script.py ... fit.

Describe the solution you'd like
Wouldn't it be nice to do dpat extract-features fit? click may allow this via @pass_context.

Describe the bug
The performance of the pre-trainer simclr spikes at regular 1900 steps, which is a pattern that cannot be ignored. Since the best model is saved (based on the lowest training loss), the best model becomes useless as the loss makes no sense.

To Reproduce
Steps to reproduce the behavior:

1. Train a SimCLR with PMCHHGImageDataModule for at least 6000 steps to see the pattern.
2. Batch size 1024

Expected behavior
Smooth loss.

Desktop (please complete the following information):

• OS: rocky linux
• DPAT version 3.2.1.

Additional context
Add any other context about the problem here. and linked issues.

Looks like it can be the loss reset at every epoch. Either don't do that and fix it to do something else, or use drop_last in the dataloader, such that the model has not varying batch size.

It's a stretch, but it could also be an issue of torch2.0 compile, where the loss function is traced with a large batch size and reused while the batch size is small for the last minibatch of an epoch.

Either scenarios do not explain why it does not happen every epoch, though. It only happend 6 times for 30 epochs.

DeepSMILE uses ShuffleNetV2 as SimCLR backbone, but does it perform best for our purpose?

Other backbones, like ResNets may perform better. ShuffleNetV2 was in 2018 the best for imagenet.

• ResNet18
• ResNet9
• ShuffleNetV2

For reproduction purposes, it might be more useful to store the paths of the splits inside the images/splits directory and have relative paths in images/splits/*.csv linking to images.

This way, device can use the precomputed splits without changing all the absolute paths in the csv files to relative paths.

Relative paths will then be read by dpat.data.datasets.PMCHHGImageDataset, before they are converted to absolute paths on the target system.

Describe the bug
The features may not have to be compiled using exactly the same settings for the datamodule as on which the feature extractor is trained.
This is a problem, as it might perform bad on images without a mask or tiles with a different size or mpp.

To Reproduce
Steps to reproduce the behavior:

1. Train a feature extractor
2. Compile features with a new PMCHHGImageDataset

Desktop (please complete the following information):

• DPAT version 4.3.2

Additional context
A solution might be to save the pytorch lightning hyperparameters with save_hyperparameters or ask for the specific config file the feature extractor trained with, and use those variables for instantiation of PMCHHGImageDataset

Research question
Does embedding clinical context (tumour location, sex, age, taken medication) improve performance?

Hypothesis
Yes. E.g. tissue morphology varies from location to location in the brain. There is a part in the brain, the x(?), that shows a lot of cells, while it is healthy. Characterizing them as malignant by the AI is therefore wrong.

Method
Proposal: introduce CC-VarMIL (Clinical Context VarMIL), where

$$\hat{Z}^m = [\overline{Z}^m,Z^m_\sigma,Z^m_c]^T$$

where $Z^m_c$ is an embedding that describes the clinical context and is concatenated to the VarMIL attention weighted vector.

Why is this experiment worthwhile?
Clinical context such as tumour location is relevant for the diagnosis made by humans. Why not allow the AI to work with the same clinical context as well?

Result comments on this should include results and a discussion.

Research question
Does BERT finetuning increase performance?

Hypothesis
Yes, the classifier in BERT will also pull apart the word embeddings belonging to specific locations, making it easier to distinguish images with different tumour classes.

Method
Train BERT with frozen and trainable parameters. Keep everything else the same. See if the performance is increased.

Finetuning BERT will take longer as a (backward) pass through BERT is costly.

Result comments on this should include results and a discussion.

Describe the bug
If the number of epochs is changed via the command line, the number of epochs used for the lr scheduler is not updated. The LR scheduler uses the maximum epochs that it is linked to in the default config file, but is not updated after command line override.

Expected behavior
Update the max_epochs for the learning rate scheduler, also after command line override.

Screenshots

Desktop (please complete the following information):

• DPAT version 4.0.1

Additional context
Maybe it is useful to use PL link arguments:

class MyLightningCLI(LightningCLI):
    def add_arguments_to_parser(self, parser):
        parser.link_arguments("trainer.max_epochs", "trainer.scheduler.max_pochs")


cli = MyLightningCLI(MyModel, MyDataModule)

or something like that.

Is your feature request related to a problem? Please describe.
Currently, the minority class is presented less often to the model. Therefore, the model is learning the majority classes better.

Describe the solution you'd like
It would be nice if the model sees as many examples from every available class.

Describe alternatives you've considered
https://pytorch.org/docs/stable/data.html#torch.utils.data.WeightedRandomSampler

Currently, the SwAV module uses the SwAV collate function. Lightly is deprecating all collate functions soon and will publish examples on their docs featuring the transform variants of the collate functions.

lightly-ai/lightly#1075

Is your feature request related to a problem? Please describe.
The PMCHHGImageDataset does not implement masking.

Describe the solution you'd like
Implement masking.

Alternatives I considered

1. Creating tissue segmentation algorithm, inspired by (improved) FESI.
2. Manually annotating tissue as such in QuPath and use the binary annotations as masks.

Additional context
Some tiles are black, because no tissue is present. They are still fed to the neural networks, without any benefit.

How do various feature extractors perform on the dataset?

• SwAV
• SimCLR
• Model pretrained on ImageNet

Is your feature request related to a problem? Please describe.
The vipsbin is hardcoded in the CLI main entry module and other modules. The package has to be installed as editable and the path to vipsbin needs to be changed.

Describe the solution you'd like
Add a config option to the cli to pass vipsbin path to install_windows. Or set vipsbin in a config file, which is passed to the cli.

Additional context
This is only a problem for Windows.

SwAV currently does not log numerical losses. Every step, the loss is NaN.

Is your feature request related to a problem? Please describe.
Pytorch Lightning provides a CLI to deal with training models. This is currently not used. Parameters need to be changed in the script.

Describe the solution you'd like
A configuration file that can be referenced from the cli would be nice.

Describe alternatives you've considered
PL CLI to accept a config file.

Normalization should be done using the mean and std of the (training) set.

For feature extraction, the Lightly collate function can take care of normalization. By default, it uses the mean and std of imagenet, which is not suitable for HHG purposes.

Research question
The last hidden layer of BERT is best suited for contextualized text embeddings.

Hypothesis
It is the layer where the structure is best defined, considering all previous relations in the other 11 layers.

Method

1. Instantiate pretrained ClinialBERT
2. Gather a dataset of medical terms with different classes. E.g. all brain locations, but locations are grouped by occurrence of tumours in those regions.
3. Generate embeddings from layers as proposed in https://jalammar.github.io/illustrated-bert/
4. Intrinsic evaluation per embedding strategy. Evaluation measure tbd

Why is this experiment worthwhile?
Papers report different accuracies when using different embedding strategies from pretrained models (ref!).

Is your feature request related to a problem? Please describe.
The current implementation of making splits allows for 25 folds (5 train-val folds for 1 train-test fold). We are not using all of them anyway.

Describe the solution you'd like
Do a fair creation of splits, such that testing can be performed across the whole set and validation is done fairly.

Describe alternatives you've considered

1. stratified 5-fold (train-test)
2. stratified split train -> train-val

The above procedure will give 5 folds:
fold-1-train|fold-1-val|fold-1-train, etc.

Additional context
For step 1, sklearn's StratifiedKFold can be used. For step 2, StratifiedShuffleSplit can be used.

Is your feature request related to a problem? Please describe.
The model has to be defined in a config file. It would be nice if --model CCMIL just works.

Describe the solution you'd like
Lightning CLI way of doing this

And include the defaults in the signature of CCMIL/VarMIL/ABMIL.

Describe the bug
In the final feature compiled file, the number of feature vectors is incorrect.

To Reproduce
Run feature compilation.

Expected behavior
Have as many feature vectors in the concatenated version as in the 'spread out' version

Desktop (please complete the following information):

• dpat version: 2.3.1

Autogeneration of the docs misses some parts. E.g. the create_splits module of the splits package is not documented well.

Is your feature request related to a problem? Please describe.
Writing to hdf5 file takes very long. About an hour for a fold.

Describe the solution you'd like
Concurrent file writing.

Describe alternatives you've considered
HDF5 for python allows for concurrency [1].

Additional context
[1] https://docs.h5py.org/en/stable/mpi.html

Describe the bug
The file is never closed well when interrupting with ctrl+c.

^CERROR:root:Received SIGINT or SIGTERM! Finishing this block, then exiting.                                                                                                
  5%|██████▋                                                                                                                                | 5/101 [00:10<03:25,  2.14s/it]
ERROR:dpat.data.pmchhg_h5_dataset:Something went wrong with closing the file. Please inspect the file and delete if broken.

Expected behavior
Close the file well.

Desktop (please complete the following information):

• OS: [e.g. Linux, Windows, macOS]
• DPAT version 4.3.1

What is the subject of the change request to the docs?
Add tutorial on how to make masks. Either provide a plug-and-play script or provide steps how to get the masks.

Describe the bug
Mypy doesn't run in CI, so is commented out. Fix typing until it works again and reactivate mypy in CI.

Research question
Do augmented compiled feature vectors matter when using imagenet?

Hypothesis
Yes. More data, better generalization.

Method

1. Choose augmentations
2. Apply imagenet pretrained model on augmentations of images
3. Save features
4. run MIL

Why is this experiment worthwhile?
To increase the accuracy yo