Some videos are unavailable about fivr-200k HOT 4 OPEN

Yes, you can still use the evaluation script. To do so, when you run the evaluation.py, you have to provide to the dataset_ids argument the path to file that contains the ids of the downloaded videos. For example:

python evaluation.py --result_file ./results.json --relevant_labels ND,DS --dataset_ids ./incomplete_dataset_ids.txt

from fivr-200k.

meet the same problem, can we still use the incomplete video set to evaluate the map of the retrieval results? About 40000 videos download failure.

from fivr-200k.

@gkordo Hi, what happens if some of the queries are missing, will the evaluation script handle missing queries? Thank you!

from fivr-200k.

The evaluation script handles this case and ignores the missing queries. Hence, the evaluation metrics (mAP and PR-curve) are calculated based solely on the queries in the result file.

from fivr-200k.