This repository provides an evaluation metric for generative question answering systems based on our NAACL 2021 paper KPQA: A Metric for Generative Question Answering Using Keyphrase Weights.
Here, we provide the code to compute KPQA-metric, and human annotated data.
Create a python 3.6 environment and then install the requirements.
Install packages using "requirements.txt"
conda create -name kpqa python=3.6
pip install -r requirements.txt
https://drive.google.com/file/d/1pHQuPhf-LBFTBRabjIeTpKy3KGlMtyzT/view?usp=sharing
Download the "ckpt.zip" and extract it. (default directory is "./ckpt") You can compute KPQA-metric using "compute_KPQA.py" as follows.
python compute_KPQA.py \
--data sample.csv \ # Target data to compute the score. Please see the "sample.csv" for file format
--model_path $CHECKPOINT_DIR \ # Path of checkpoint directory (extract path of "ckpt.zip")
--out_file results.csv \ # output file that has score for each question-answer pair. Please see the the sample result in "result.csv".
--num_ref 1 \ # For usage in computing the score with multiple references.
We provide human judgments of correctness for 4 datasets:MS-MARCO NLG, AVSD, Narrative QA and SemEval 2018 Task 11 (SemEval).
For MS-MARCO NLG and AVSD, we generate the answer using two models for each dataset.
For NarrativeQA and SemEval, we preprocessed the dataset from Evaluating Question Answering Evaluation.
If you find this repo useful, please consider citing:
@inproceedings{lee2021kpqa,
title={KPQA: A Metric for Generative Question Answering Using Keyphrase Weights},
author={Lee, Hwanhee and Yoon, Seunghyun and Dernoncourt, Franck and Kim, Doo Soon and Bui, Trung and Shin, Joongbo and Jung, Kyomin},
booktitle={Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
pages={2105--2115},
year={2021}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent