This repository contains the data, scripts and baseline codes for DSTC11 Track 5.

This challenge track aims to support more informative and engaging task-oriented conversations by utilizing the subjective knowledge from review posts. Track participants will develop dialogue systems to understand relevant review posts, and generate system responses grounded on the selected review knowledge snippets.

Organizers: Seokhwan Kim, Spandana Gella, Chao Zhao, Di Jin, Alexandros Papangelis, Behnam Hedayatnia, Yang Liu, Dilek Hakkani-Tur

• Track Proposal
• Challenge Registration
• Data Formats
• Baseline Details

This challenge track distinguishes between turns that could be handled by the existing task-oriented conversational models with no extra knowledge and turns that require external subjective knowledge to be answered by the dialogue system. We focus on the turns that require knowledge access as the evaluation target in this track by the following three tasks:

Participants will develop systems to generate the outputs for each task. They can leverage the annotations and the ground-truth responses available in the training and validation datasets.

In the test phase, participants will be given a set of unlabeled test instances. And they will submit up to 5 system outputs for all three tasks.

NOTE: For teams who are interested in only one or two of the tasks, we recommend to use our baseline system for the remaining tasks to complete the system outputs.

Each submission will be evaluated in the following task-specific automated metrics first:

To consider the dependencies between the tasks, the scores for knowledge selection and knowledge-grounded response generation are weighted by knowledge-seeking turn detection performances. Please find more details from scores.py.

The final ranking will be based on human evaluation results only for selected systems according to automated evaluation scores. It will address the following aspects: appropriateness and relevance to given knowledge.

In this challenge track, participants will use an augmented version of MultiWoz 2.1 which includes newly introduced subjective knowledge-seeking turns. All the ground-truth annotations for Knowledge-seeking Turn Detection and Knowledge Selection tasks as well as the agent's responses for Knowledge-grounded Response Generation task are available to develop the components on the training and validation sets. In addition, relevant knowledge snippets for each domain and entity are also provided in knowledge.json.

In the test phase, participants will be evaluated on the results generated by their models for the unlabeled test set. To evaluate the generalizability and the portability of each model, the unseen test set may include different domains, entities and locales than MultiWoz.

Data and system output format details can be found from data/README.md.

• Training data released: Dec 19, 2022
• Test data released: Mar 24, 2023
• Entry submission deadline: Mar 31, 2023
• Objective evaluation completed: Apr 3, 2023
• Human evaluation completed: Apr 10, 2023

• Participation is welcome from any team (academic, corporate, non profit, government).
• The identity of participants will NOT be published or made public. In written results, teams will be identified as team IDs (e.g. team1, team2, etc). The organizers will verbally indicate the identities of all teams at the workshop chosen for communicating results.
• Participants may identify their own team label (e.g. team5), in publications or presentations, if they desire, but may not reveal the identities of other teams.
• Participants are allowed to use any external datasets, resources or pre-trained models.

• To join the mailing list: visit https://groups.google.com/a/dstc.community/forum/#!forum/list/join

• To post a message: send your message to [email protected]

• To leave the mailing list: visit https://groups.google.com/a/dstc.community/forum/#!forum/list/unsubscribe

Please feel free to contact: seokhwk (at) amazon (dot) com

The code is licensed under Apache 2.0 (see SOFTWARELICENSE) and the data files are licensed under CDLA-Sharing 1.0 (see DATALICENSE).