YouCook2-Leaderboard

This repo hosts leaderboards on YouCook2 and related resources. The main tasks concerned include text-to-video retrieval, video captioning, dense video description (captioning), and object grounding. Results are on the validation set unless otherwise specified. Note that until now, leaderboard on the test set is only available for the dense video description and object groudning (see FQA for details). See also our policy on arXiv papers.

To have your model added to the leaderboard, please reach out to us or submit a PR.

Text-to-Video Retrieval

"Video" here refers to video clips (a cooking video usually contains multiple steps/clips). Each clip is considered an individual video assuming the clip boundary is given. Text to whole video retrieval is not considered in this task.

All these results were evaluated using a subset of 3,350/3,492 video clips from the validation set that aren't present in HowTo100M.

Video Captioning

In this task, we assume GT event segments are available (i.e., the start and end timestamps of each recipe step).

Evaluation on this task diverges into two modes: micro-level and macro-level. In micro-level evaluation, the averaged score on all segments is reported. In macro-level evaluation, language metric scores are first averaged across segments within each video and then averaged across the dataset.

• Micro-level

• Macro-level

Note: B@3 - BLEU@3, B@4 - BLEU@4, M - METEOR, and C - CIDEr. The input modalities include video (V), transcript (T) or both (V+T). Evaluation code: Macro-level and Micro-level.

Dense Video Description

In this task, we need to predict both event segments and description. The latest evaluation tool from ActivityNet Captions challenge is adopted (vs. the 2017 buggy version). Starter code see densecap.

A dedicated evaluation server on the test set (to be updated to the latest eval code soon): https://competitions.codalab.org/competitions/20594

Object Grounding

The goal of this task is to spatial-temporally locate/ground object words (provided by a description) to the associated video clip. Starter code and evaluation protocol see DVSA.

A dedicated evaluation server on this task (including on the test set): https://competitions.codalab.org/competitions/20302

FAQ

• Why there is no test set?

We withheld the test split of YouCook2 for evaluation purposes on event proposal generation and dense video description. Releasing the event segments (for video captioning) or captions (for text-to-video retrieval) break the confidentiality of the leaderboard on the former two tasks. However, since there is a growing interest in the latter two tasks, we are considering releasing the full test annotation for research purposes. This will likely happen after the CVPR'21 deadline.

• Can I submit results from arXiv papers?

For now, we only accept published works to maintain a high result credibility. Therefore, we do not accept results from arXiv papers.

Reference (bibtex)

YouCook2 dataset paper:

@inproceedings{ZhXuCoAAAI18,
    author={Zhou, Luowei and Xu, Chenliang and Corso, Jason J},
    title = {Towards Automatic Learning of Procedures From Web Instructional Videos},
    booktitle = {AAAI Conference on Artificial Intelligence},
    pages={7590--7598},
    year = {2018},
    url = {https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17344}
}

Video description benchmark on YouCook2:

@inproceedings{ZhZhCoCVPR2018,
    title={End-to-End Dense Video Captioning with Masked Transformer},
    author={Zhou, Luowei and Zhou, Yingbo and Corso, Jason J and Socher, Richard and Xiong, Caiming},
    booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
    pages={8739--8748},
    year={2018}
}

YouCook2-BoundingBox:

@inproceedings{ZhLoCoBMVC18,
    author={Zhou, Luowei and Louis, Nathan and Corso, Jason J},
    title={Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction},
    booktitle = {British Machine Vision Conference},
    year = {2018},
    url = {http://bmvc2018.org/contents/papers/0070.pdf}
}

Other resources

A list of leaderboards from Paperwithcode (results might be incomplete):

https://paperswithcode.com/sota/video-retrieval-on-youcook2

https://paperswithcode.com/sota/video-captioning-on-youcook2

VideoPentathlon challenge: https://www.robots.ox.ac.uk/~vgg/challenges/video-pentathlon/challenge.html

VideoBERT and CBT:

https://openaccess.thecvf.com/content_ICCV_2019/papers/Sun_VideoBERT_A_Joint_Model_for_Video_and_Language_Representation_Learning_ICCV_2019_paper.pdf

https://arxiv.org/pdf/1906.05743.pdf

HowTo100M / MIL-NCE:

https://www.di.ens.fr/willow/research/howto100m/

https://openaccess.thecvf.com/content_CVPR_2020/papers/Miech_End-to-End_Learning_of_Visual_Representations_From_Uncurated_Instructional_Videos_CVPR_2020_paper.pdf

MSRVTT: https://www.microsoft.com/en-us/research/publication/msr-vtt-a-large-video-description-dataset-for-bridging-video-and-language/

Epic-Kitchen: https://epic-kitchens.github.io/2020-100

SOTA Dense Video Captioning metric: https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123510511.pdf

Acknowledgement

YouCook2 has been supported in part by Google, ARO W911NF-15-1-0354, NSF CNS 1463102, NSF CNS 1628987, NSF NRI 1522904 and NSF BIGDATA 1741472. This repo solely reflects the opinions and conclusions of its authors and neither Google, ARO nor NSF. Thanks to Santiago for suggesting and contributing to the leaderboard. Thanks for Nathan for contributing to the Object Grounding part.