This is the code for the paper
Inferring and Executing Programs for Visual Reasoning
Justin Johnson,
Bharath Hariharan,
Laurens van der Maaten,
Judy Hoffman,
Fei-Fei Li,
Larry Zitnick,
Ross Girshick
To appear at ICCV 2017
If you find this code useful in your research then please cite
@inproceedings{johnson2017inferring,
title={Inferring and Executing Programs for Visual Reasoning},
author={Johnson, Justin and Hariharan, Bharath and van der Maaten, Laurens and Hoffman, Judy
and Fei-Fei, Li and Zitnick, C Lawrence and Girshick, Ross},
booktitle={ICCV},
year={2017}
}
All code was developed and tested on Ubuntu 16.04 with Python 3.5.
You can set up a virtual environment to run the code like this:
virtualenv -p python3 .env # Create virtual environment
source .env/bin/activate # Activate virtual environment
pip install -r requirements.txt # Install dependencies
echo $PWD > .env/lib/python3.5/site-packages/iep.pth # Add this package to virtual environment
# Work for a while ...
deactivate # Exit virtual environmentYou can download and unzip the pretrained models by running bash scripts/download_pretrained_models.sh;
the models will take about 1.1 GB on disk.
We provide two sets of pretrained models:
- The models in
models/CLEVRwere trained on the CLEVR dataset; these were used to make Table 1 in the paper. - The models in
models/CLEVR-Humanswere first trained on CLEVR and then finetuned on the CLEVR-Humans dataset; these models were used to make Table 3 in the paper.
You can easily run any of the pretrained models on new images and questions. As an example, we will run several models on the following example image from the CLEVR validation set:
After downloading the pretrained models, you can use the pretrained model to answer questions about this image with the following command:
python scripts/run_model.py \
--program_generator models/CLEVR/program_generator_18k.pt \
--execution_engine models/CLEVR/execution_engine_18k.pt \
--image img/CLEVR_val_000013.png \
--question "Does the small sphere have the same color as the cube left of the gray cube?"This will print the predicted answer, as well as the program that the model used to produce the answer. For the example command we get the output:
Question: "Does the small sphere have the same color as the cube left of the gray cube?"
Predicted answer: yes
Predicted program:
equal_color
query_color
unique
filter_shape[sphere]
filter_size[small]
scene
query_color
unique
filter_shape[cube]
relate[left]
unique
filter_shape[cube]
filter_color[gray]
sceneThe procedure for training your own models is described here.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent