Our paper is accepted to ACL 2023! Please consider citing our paper if you find it useful.

Reported results on standard summarization factuality benchmarks can be misleading. Factuality metrics, even ChatGPT-based, don’t actually improve on more recent summarizers!

The propensity of abstractive summarization models to make factual errors has been studied extensively, including design of metrics to detect factual errors and annotation of errors in current systems’ outputs. However, the ever-evolving nature of summarization systems, metrics, and annotated benchmarks makes factuality evaluation a moving target, and drawing clear comparisons among metrics has become increasingly difficult. In this work, we aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model. We compare performance of state-of-the-art factuality metrics, including recent ChatGPT-based metrics, on this stratified benchmark and show that their performance varies significantly across different types of summarization models. Critically, our analysis shows that much of the recent improvement in the factuality detection space has been on summaries from older (pre-Transformer) models instead of more relevant recent summarization models. We further perform a finer-grained analysis per error-type and find similar performance variance across error types for different factuality metrics. Our results show that no one metric is superior in all settings or for all error types, and we provide recommendations for best practices given these insights.

Please check our latest paper version here.

The dataset can be found in the data folder. aggre_fact_sota.csv is a subset of aggre_fact_final.csv, which only contains summaries generated by SOTA models. The followings are descriptions of column names.

The new csv file containing the results from chatgpt based metrics will be released shortly.

We unified unique error type taxonomies from XSumFaith, FRANK, Goyal21 and CLIFF under data/error_type_mapping. More details can be found in Section 4 of the paper.