`zeroshot_templates` split error for FairFace / UTKFace about datacomp HOT 9 CLOSED

Hi @EIFY, thanks for digging deeper into the issue! Even if the code runs without errors, when commenting out the line, the zeroshot templates that you see are actually incorrect.

It seems like this was broken in a recent update to clip-benchmark (a dependency of this code). Make sure you have clip-benchmark==1.4.0 installed as per the environment.yml file; the version number should not be greater or less than that. Be sure to uncomment the code you commented out as well.

You may need to rerun all your evals after this fix, because it is likely all of the test sets were evaluated with the incorrect zeroshot templates.

from datacomp.

Looking over the issue again, I don't believe it would affect all of your evals, but it is probably still safer to rerun evals from scratch (delete the eval_results.jsonl file for any models you've tried to evaluate).

from datacomp.

@djghosh13 Thanks for the replies — rolling back to clip-benchmark==1.4.0 indeed solves the issue. Just to confirm, are these the intended zeroshot_templates?

zeroshot_templates=['age:a photo of a person {c} years old', 'gender:a photo of a {c}', 'race:a photo of a {c} person', 'toxic:a photo of a {c}']

from datacomp.

Yes, those are correct. Glad the issue was resolved!

from datacomp.

Personally I consider running the package with older or newer dependencies than specified some sort of off-label use, but I don't mind keeping this open till clip-benchmark is fixed as a warning sign 😅

from datacomp.

I think we should keep this open until clip benchmark is fixed

from datacomp.

@rom1504 do you mean as a reminder to other potential users of datacomp? I've already opened LAION-AI/CLIP_benchmark#109 in CLIP_benchmark as an issue to remind us to fix this problem

from datacomp.

from datacomp.