Authors contributed equally to this repository, the commit history does not reflect contributions.

Code in this repo uses code from multimodal prompt learning, which in turn uses code from Co-CoOp and CoOp.

• Install dassl library and other requirements.

# Instructions borrowed from https://github.com/KaiyangZhou/Dassl.pytorch#installation

git clone https://github.com/KaiyangZhou/Dassl.pytorch.git
cd Dassl.pytorch/
pip install -r requirements.txt
python setup.py develop
cd ..

pip install open_clip_torch
pip install pytorch_metric_learning

• Create a directory somewhere called data/. Download all 15 zip files from this shared Google Drive and unzip them into data/. The resulting file tree should look like:

data/
|-- caltech-101
|-- dtd
|-- eurosat
|-- fgvc_aircraft
|-- food-101
|-- imagenet
|-- imagenet-adversarial
|-- imagenet-rendition
|-- imagenet-sketch
|-- imagenetv2
|-- oxford_flowers
|-- oxford_pets
|-- stanford_cars
|-- sun397
|-- ucf101

Alternatively, follow the download instructions here (some dataset links are stale; may also need to reorganize the directory structure): installing datasets

Modify the following two lines in argparse_parameters.py to reflect where you have your data/ dir and where you want the pretrained CLIP weights to be cached (which could be many gigabytes)

parser.add_argument('--cache_dir', default = "", type =str) # set to directory where you want large pretrained model weights to be cached
parser.add_argument('--data_dir', default = "", type =str)  # set to parent directory of data/

ICE works with any captioner or VLM. We tried CoCa, BLIP-2, and LLAVA. We provide three example captions for each image in each dataset in the captions/ folder. These can be used directly. However, if you want to generate them yourself, you can run:

Run: sh run_make_captions.sh

The above script generates three captions per image per dataset, for CoCa, BLIP-2 and LLAVA.

For LLAVA, the conda environment should be setup according to their github repo: here

Run: sh run_ice.sh

See ICE_results.xlsx.