PyTorch implementation for TIP2023 paper of “Plug-and-Play Regulators for Image-Text Matching”.

It is built on top of the SGRAF, GPO and Awesome_Matching.

If any problems, please contact me at [email protected]. ([email protected] is deprecated)

The framework of RCAR:

The reported results (One can import GloVe Embedding or BERT for better results)

Utilize pip install -r requirements.txt for the following dependencies.

• Python 3.7.11
• PyTorch 1.7.1
• NumPy 1.21.5
• Punkt Sentence Tokenizer:

import nltk
nltk.download()
> d punkt

We follow SCAN to obtain image features and vocabularies, which can be downloaded by using:

https://www.kaggle.com/datasets/kuanghueilee/scan-features

Another download link is available below：

https://drive.google.com/drive/u/0/folders/1os1Kr7HeTbh8FajBNegW8rjJf6GIhFqC

data
├── coco
│   ├── precomp  # pre-computed BUTD region features for COCO, provided by SCAN
│   │      ├── train_ids.txt
│   │      ├── train_caps.txt
│   │      ├── ......
│   │
│   └── id_mapping.json  # mapping from coco-id to image's file name
│   
│
├── f30k
│   ├── precomp  # pre-computed BUTD region features for Flickr30K, provided by SCAN
│   │      ├── train_ids.txt
│   │      ├── train_caps.txt
│   │      ├── ......
│   │
│   └── id_mapping.json  # mapping from f30k index to image's file name
│   
│
└── vocab  # vocab files provided by SCAN (only used when the text backbone is BiGRU)

Modify the model_path, split, fold5 in the eval.py file. Note that fold5=True is only for evaluation on mscoco1K (5 folders average) while fold5=False for mscoco5K and flickr30K. Pretrained models and Log files can be downloaded from Flickr30K_RCAR and MSCOCO_RCAR.

Then run python eval.py in the terminal.

Uncomment the required parts of BASELINE, RAR, RCR, RCAR in the train_xxxx_xxx.sh file.

Then run ./train_xxx_xxx.sh in the terminal:

If RCAR is useful for your research, please cite the following paper:

  @article{Diao2023RCAR,
     author={Diao, Haiwen and Zhang, Ying and Liu, Wei and Ruan, Xiang and Lu, Huchuan},
     journal={IEEE Transactions on Image Processing}, 
     title={Plug-and-Play Regulators for Image-Text Matching}, 
     year={2023},
     volume={32},
     pages={2322-2334}
  }

Apache License 2.0.