• Please follow the instructions in the COCO API README and here to download and setup the COCO2017 data. Link the dataset under data/coco. The folder data/coco should contain the folders val2017, train2017 and files annotations/instances_val2017.json, and annotations/instances_train2017.json.

PyTorch, torchvision, wandb, and torchtext. A full list can be found in requirements.txt.

to train an explainer on COCO for 50 epochs on layer4 representations of a ResNet50 trained on ImageNet, run

python train_explainer.py --layer layer4 --refer coco --wandb False --epochs 50 --name run_name

set wandb to True to log using wandb. Training on a single 24gb GPU takes 1-2 days.

To visualize filters on wandb using the trained explainer, run:

python infer_filter.py --layer layer4 --name run_name --wandb True --refer coco

Line 200 in infer_filter.py can be uncommented to visualize the filters without wandb Here is the readme from the OG LaViSE repository:

This is the official repository for paper "Explaining Deep Convolutional Neural Networks via Unsupervised Visual-Semantic Filter Attention" to appear in CVPR 2022.

Authors: Yu Yang, Seungbae Kim, Jungseock Joo

• Please follow the instructions in the COCO API README and here to download and setup the COCO data.

• Please follow the instructions in the README of the python wrapper for the Visual Genome API and here.

• We load the pretrained GloVe word embeddings directly from the torchtext library.

• The list of entities used to discover new concepts is provided in data/entities.txt.

Required packages can be found in requirements.txt.

Train an explainer with

python train_explainer.py

Explain a target filter of any model with

python infer_filter.py

More features will be added soon! 🍻

@inproceedings{yang2022explaining,
    author    = {Yang, Yu and Kim, Seungbae and Joo, Jungseock},
    title     = {Explaining Deep Convolutional Neural Networks via Unsupervised Visual-Semantic Filter Attention},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    year      = {2022},
}