Is it possible to implement data filtering in training script? about datacomp HOT 7 CLOSED

Yes. Exactly it is what i am looking for!

By comparing the your uploaded parquets and my generated parquets using https://github.com/mlfoundations/dataset2metadata, I found there are some incompatible keys ('clip_b32_similarity_score' vs. 'oai-clip-vit-b32-score').

This problem I think can be solved by change postprocess_parquet_lookup (https://github.com/mlfoundations/dataset2metadata/blob/225408d90462323cf2afc5b40e789bc8cd966ce6/dataset2metadata/registry.py#L25)

I will post here once I finish my reproduction of baselines using this entire process.

I ran clip_score_l14_30_percent filtering on my generated parquets, and got 5.39% zero-shot ImageNet acc for small track, which is comparable with the official 5.1%.

from datacomp.

Hey @vtddggg. That's a great question, and it was a big point of discussion when we were designing our codebase. While it's certainly possible to implement the filtering solution, the problem is that this can make training quite slow (especially if are filtering a large fraction of the pool) because the dataloader still needs to iterate through all samples in the pool. Therefore, we decided to go with the first approach, where you pay an initial cost to reshard and need some extra storage, but in return the training is much faster.

We have some discussion about this decision in our paper in Appendix J (https://arxiv.org/abs/2304.14108)

from datacomp.

I got it. Thanks!

from datacomp.

@gabrielilharco Sorry, I just reopen the issue again.

When we want to compute clip score from raw image & text (instead of reading from metadata), we must learn to process tars files. However there are few reference examples about the use of webdatasets.

Regarding as the detailed implementation of baseline methods, can you provide a simple example about fetch uid & image & text data from tars with distributed data parallel, and generate sample_ids.npy? I think it will be very helpful

from datacomp.

No problem! We also open-sourced code that we used for generating our metadata in this repo: https://github.com/mlfoundations/dataset2metadata. Let me know if that covers what you're looking for

from datacomp.

Yes. Exactly it is what i am looking for!

By comparing the your uploaded parquets and my generated parquets using https://github.com/mlfoundations/dataset2metadata, I found there are some incompatible keys ('clip_b32_similarity_score' vs. 'oai-clip-vit-b32-score').

This problem I think can be solved by change postprocess_parquet_lookup (https://github.com/mlfoundations/dataset2metadata/blob/225408d90462323cf2afc5b40e789bc8cd966ce6/dataset2metadata/registry.py#L25)

I will post here once I finish my reproduction of baselines using this entire process.

from datacomp.

Nice—thanks for looking into this end to end! I will add a flag to dataset2metadata to use keys that are compatible with this repo

from datacomp.