Could you publish the used to deduplicate CommonPool against the datasets used

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

Deduplication against evaluation sets about datacomp HOT 1 CLOSED

mlfoundations commented on July 25, 2024

Deduplication against evaluation sets

from datacomp.

Comments (1)

sagadre commented on July 25, 2024 1

Hi @nopperl! Please see this repo for our dataset pre-processing implementation. If you are only interested in de-contamination against eval sets, this modified yml should be all u need:

models: # model directives, specifying the models to instantiate
  - dedup-isc-ft-v107
postprocess_columns: # postprocessing directives
  - dedup-isc-ft-v107-score
additional_fields: # fields in a webdataset json to carry over into the metadata
  - uid
nworkers: 2
batch_size: 512
device: 0
input_tars: "path/to/my/tars/000057{17..19}.tar" # braceexpand suported, can also be s3 paths
output_metadata_dir: "path/to/my/ouput/metadata" # can be arbitrary path
custom_pypath: null # if model, preprocessors, postprocessors not known, look in this python file for user provided custom implementation
reprocess: True # if true will process from scratch, else will just process tars not already processed

This should produce output parquet files with uid and dedup-isc-ft-v107-score fields. By thresholding the latter field at 0.604169 you can then recover the desired uids

from datacomp.

Recommend Projects