datatjej / knowing-when-to-look-adaptive-attention Goto Github PK

PyTorch Implementation of Knowing When to Look: Adaptive Attention via a Visual Sentinal for Image Captioning Paper
Original Torch Implementation by Lu. et al can be found here

1. Download the COCO 2014 dataset from here. In particualr, you'll need the 2014 Training, Validation and Testing images, as well as the 2014 Train/Val annotations.

2. Download Karpathy's Train/Val/Test Split. You may download it from here.
   

3. If you want to do evaluation on COCO, make sure to download the COCO API from here if your on Linux or from here if your on Windows. Then download the COCO caption toolkit from here and re-name the folder to cococaption. (This also requires java. Simply dowload it from here if you don't have it).

preprocess.py Creates the WORDMAP.json file and the .h5 files
dataset.py Creates the custom dataset
util.py Functions to be used throught the code
models.py Defines the architectures
train_eval For Training and Evaluation
run.ipynb For Testing and Visualization
The folder caption data includes data used along with images, mainly for evaluation purposes.

Place the test image in the folder 'test_imgs', and name it as test.jpg, and then run the run.ipynbjupyter notebook file to get the results.

The file here contains the obtained evalaution scores on the validation split. The pretrained model is provided here as well (trained for 12 epochs). Some results from Karpathy's Split is shown below. The visual grounding probability (1-beta) is shown in green.

Code adopted from sgrvinod implementation of "Show, Attend and Tell"