Framework for different fusion strategies in multi-view learning with PyTorch. Academic purposes
- Example on how to define and use different fusion strategies in the MVL model
- Example on how to train these MVL models
For definitions on the concepts used here please look at Common Practices and Taxonomy in Deep Multiview Fusion for Remote Sensing Applications.
- For Input fusion (with feature concatenation) you can just create it with
from mvlearning.fusion import InputFusion
InputFusion(single_pytorch_model, view_names=["a list of string", "with the names of the views"])
- For Decision fusion (with averaging output) you can just create it with
from mvlearning.fusion import DecisionFusion
DecisionFusion({"view 1": pytorch_model1, "view 2": pytorch_model2, ...})
- For Feature fusion you can just create it with
from mvlearning.fusion import FeatureFusion
FeatureFusion({"view 1": pytorch_encoder1, "view 2": pytorch_encoder2, ...}, pytorch_merge_module, pytorch_model_head)
- The encoders have to had a function called
get_output_size
where the dimension of the output is returned. - For detailed examples see the Examples folder.
- For installation you can run:
pip install --editable .