Source code for our paper : MARVEL: Unlocking the Multi-Modal Capability of Dense Retrieval via Visual Module Plugin

Click the links below to view our papers and checkpoints

If you find this work useful, please cite our paper and give us a shining star 🌟

MARVEL unlocks the multi-modal capability of dense retrieval via visual module plugin. It encodes queries and multi-modal documents with a unified encoder model to bridge the modality gap between images and texts, conducts retrieval, modality routing, and result fusion within a unified embedding space.

1. Install the following packages using Pip or Conda under this environment

Python==3.7
Pytorch
transformers
clip
faiss-cpu==1.7.0
tqdm
numpy
base64
Install the pytrec_eval from https://github.com/cvangysel/pytrec_eval

We provide the version file requirements.txt of all our used packages, if you have any problems configuring the environment, please refer to this document.

2. Prepare the pretrained CLIP and T5-ANCE

MARVEL is built on CLIP and T5-ANCE model.

• First, use git clone to download this project:

git clone https://github.com/OpenMatch/MARVEL
cd MARVEL

• Download link for our WebQA: WebQA. If you want to use our ClueWeb22-MM and pretrain data, please obtain ClueWeb license first and contact us by email.
• Please make sure that the files under the data folder contain the following before running:

data/
├──WebQA/
│   ├── train.json
│   ├── dev.json
│   ├── test.json
│   ├── test_qrels.txt
│   ├── all_docs.json
│   ├── all_imgs.json
│   ├── imgs.tsv
│   └── imgs.lineidx.new
├──ClueWeb22-MM/
│   ├── train.parquet
│   ├── dev.parquet
│   ├── test.parquet
│   ├── test_qrels.txt
│   ├── text.parquet
│   └── image.parquet
└──pretrain/
    ├── train.parquet
    └── dev.parquet

Using the WebQA dataset as an example, I will show you how to reproduce the results in the MARVEL paper. The same is true for the ClueWeb22-MM dataset.

• First step: Go to the pretrain folder and pretrain MARVEL's visual module:

cd pretrain
bash train.sh

• Second step: Go to the DPR folder and train MARVEL-DPR using inbatch negatives:

cd DPR
bash train_webqa.sh

• Third step: Then using MERVEL-DPR to generate hard negatives for training MARVEL-ANCE:

bash get_hn_webqa.sh

• Final step: Go to the ANCE folder and train MARVEL-ANCE using hard negatives:

cd ANCE
bash train_ance_webqa.sh

• These experimental results are shown in Table 2 of our paper.
• Go to the DPR or ANCE folder and evaluate model performance as follow:

bash gen_embeds.sh
bash retrieval.sh

The results are shown as follows.

• WebQA

• ClueWeb22-MM

If you have questions, suggestions, and bug reports, please email:

[email protected]     [email protected]