Can you share the CLIP score calculation script? about datacomp HOT 10 CLOSED

Hi! Looking over your code the two major differences I noticed between your implementation and the one found in dataset2metadata are

1. we use the openai CLIP repo directly for the preprocessing, though the weights are the same and this should not be an issue
2. we conduct inference in half precision (fp16), not fp32 during metadata generation

Can you try 2. and let me know if you are still running into issues? Thanks!

from datacomp.

tagging @djghosh13, who will know cache_dir and flick_1k specifics

from datacomp.

@mingtan2 sorry for the inconvenience! Some of the datasets were added later as Huggingface datasets (as opposed to webdataset for the original datasets), and so were not included in the download_evalsets.py script. I'll push an update to fix this soon. Additionally, the flickr_1k error is most likely because it's trying to download a webdataset that doesn't exist.

from datacomp.

Submitted a PR to fix this at #21; once it goes through you should be able to re-run download_evalsets.py without error, and run evaluate.py as intended (specifying --data_dir)

from datacomp.

@sagadre , Thank you so much for the quick reply.

I found using the original un-resized images reduces the difference a lot (average absolute diff from 0.007 to 0.003). Changing to fp16 further helps but cannot eliminate the difference completely. Anyway, I think the final difference is acceptable.

BTW, are submitted results updated in realtime in https://www.datacomp.ai/leaderboard.html?

Thanks.

from datacomp.

@mingtan2 great that the differences are looking lower now. just to confirm, are you using our default img2dataset configuration from our download_upstream.py script? If not that could be yet another source of the discrepancy.

as for the leaderboard, submissions are not currently updated in realtime. we have been checking submissions frequently on our end 🙂

from datacomp.

@sagadre Yes, I used the default configs with python3 download_upstream.py --scale medium --data_dir /path/to/medium --processes_count 64.

Thanks for the info about the leaderboard update.

from datacomp.

@sagadre Can I know why a few datasets have disabled the cache_dir, e.g., here? In addition, it seems flick_1k cannot be downloaded using download_evalsets.py due to "Username/Password Authentication Failed." I need this because my some of my compute notes have no network access and need offline eval. Thanks a lot.

from datacomp.

@sagadre Can I know when the leaderboard will be updated and the updating frequency? Thanks!

from datacomp.

@mingtan2 We will definitely update the leaderboard with community submissions before the DataComp ICCV workshop.

from datacomp.