By Yikai Wang, Wenbing Huang, Fuchun Sun, Tingyang Xu, Yu Rong, Junzhou Huang.

[Paper] [Slides] [Poster] [BibTex]

This repository is an official PyTorch implementation of "Deep Multimodal Fusion by Channel Exchanging", accepted by NeurIPS 2020. The basic method and applications are introduced as belows.

If you find our work useful for your research, please consider citing the following paper.

@inproceedings{wang2020cen,
  title={Deep Multimodal Fusion by Channel Exchanging},
  author={Wang, Yikai and Huang, Wenbing and Sun, Fuchun and Xu, Tingyang and Rong, Yu and Huang, Junzhou},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year={2020}
}

python>=3.6
pytorch>=1.0.0
scikit-learn>=0.20.2
opencv-python>=4.0

For semantic segmentation task on NYUDv2 (official dataset), we provide a link to download the dataset here. The provided dataset is originally preprocessed in this repository, and we add depth data in it.

For image-to-image translation task, a link to download dataset is taskonomy-sample-model-1.

Please modify the data paths in the codes, where we add comments 'Modify data path'.

First,

cd semantic_segmentation

Training script for segmentation with RGB and Depth input, the default setting uses RefineNet (ResNet101),

python main.py --gpu 0 -c checkpoint_name  # or --gpu 0 1 2

Evaluation script,

python main.py --gpu 0 --resume path_to_pth --evaluate  # optionally use --save-img to visualize results

Pretrained models, training logs and the single-scale performance on NYUDv2 (with RefineNet) are provided as follows:

First,

cd image2image_translation

Training script, an example of translation from Shade (2) and Texture (7) to RGB (0) (could reach 62~63 FID score),

python main.py --gpu 0 --img-types 2 7 0 -c checkpoint_name

This script will auto-evaluate on the validation dataset every 5 training epochs. Predicted images will be automatically saved.

For training with other modalities, the index for each img-type is described as belows, and also in Line 70 of main.py.

0: 'rgb', 1: 'normal', 2: 'reshading', 3: 'depth_euclidean', 
4: 'depth_zbuffer', 5: 'principal_curvature', 6: 'edge_occlusion', 
7: 'edge_texture', 8: 'segment_unsup2d', 9: 'segment_unsup25d'