This is the repository for Generative Modeling for Multi-task Visual Learning, published at ICML 2022.

[Paper] [Slides] [Poster] [Talk]

root 
	- rgb 
		- scene_class_0
			- 001.png
			- 002.png
			- ...
		- scene_class_1
	- ss
		- scene_class_0
			- 001.npy
			- ...
	- de
		- scene_class_0
			- 001.npy
			- ...
	- sn
		- scene_class_0
			- 001.png
			- ...
	- msk
		- scene_class_0
			- 001.png
			- ...

To run the full model, due to the training mismatch of the GAN and discriminative model, we need to first pre-train the GAN and then train the full model. See train.py and train.sh for sampled training process.

The refinement network and the self-supervision networks are the key in the paper. These modeules can help as long as we have some weakly-labelled data. Thus, we provide a light-weight MGM model -- we did not include a generative model in the framework, but assume we have some weakly-labeled data (no matter it is real or synthesized) to train the model.

see MGM_light in mgm.py and train_light.py for a sample training.

This work is built on self-attention GAN and taskgrouping repo.

@inproceedings{bao2022generative,
    Author = {Bao, Zhipeng and Hebert, Martial and Wang, Yu-Xiong},
    Title = {Generative Modeling for Multi-task Visual Learning},
    Booktitle = {ICML},
    Year = {2022},
}