A collection of papers and resources across three principal components of long document summarization: Datasets, Models and Metrics.

Updates

Large language models (LLMs) can seemingly perform a limitless range of tasks, including long document summarization. Nonetheless, we believe significant performance gaps remain when dealing with long texts.

We suggest the following two papers for potential insights:

• Lost in the Middle: How Language Models Use Long Contexts [Paper]. Despite the buzzy headlines boasting of a 100K or 1B input length limit, one must question the models' actual performance. It would be of limited utility if a model could accept unlimited input (e.g., RNN) but could not effectively reason from it. The paper provides in-depth experiments, yielding numerous insights into the actual capabilities of these models.
• Unifying Large Language Models and Knowledge Graphs: A Roadmap [Paper]. Although it does not specifically discuss long document summarization, we believe it is highly relevant for addressing the limitations of LLMs across a broad range of tasks.

We maintain that the work conducted on long document summarization thus far remains highly relevant. Practitioners can draw lessons from this research to overcome the challenges arising from long texts.

Going forward, we intend to periodically update this repository for its ongoing relevance in the current era of LLMs. Stay tuned!

• Fundamentals
• Papers
  • Datasets
  • Models
    • Abstractive Summarization
    • Extractive Summarization
    • Hybrid Summarization
    • Unsupervised Abstractive Summarization
    • Unsupervised Extractive Summarization
  • Metrics
    • Relevance
    • Factual Consistency
  • Insightful Discussions
• Our Survey

1. An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics. Huan Yee Koh, Jiaxin Ju, Ming Liu, Shirui Pan. 2022. ACM Comput. Surv. paper
2. Automatic summarization of scientific articles: A survey Nouf Ibrahim Altmami, Mohamed El Bachir Menai. 2020. Journal of King Saud University - Computer and Information Sciences [paper]

1. From Standard Summarization to New Tasks and Beyond: Summarization with Manifold Information Shen Gao, Xiuying Chen, Zhaochun Ren, Dongyan Zhao, Rui Yan. 2020. IJCAI [paper]
2. Neural Abstractive Text Summarization with Sequence-to-Sequence Models Tian Shi, Yaser Keneshloo, Naren Ramakrishnan, Chandan K. Reddy. 2019. ACM/IMS Transactions on Data Science [paper]
3. Neural Text Summarization: A Critical Evaluation Wojciech Kryściński, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, Richard Socher. 2019. EMNLP [paper]
4. Text Summarization Techniques: A Brief Survey Mehdi Allahyari, Seyedamin Pouriyeh, Mehdi Assefi, Saeid Safaei, Elizabeth D. Trippe, Juan B. Gutierrez, Krys Kochut. 2017. IJACSA[paper]
5. Recent automatic text summarization techniques: a survey Mahak Gambhir, Vishal Gupta. 2017. Artif Intell Rev [paper]

Metrics listed below are not specific to long document summarization (most summarization metrics are studied under a short/normal summarization setting). Our EMNLP 2022 work investigated these metrics under long document setting: How Far are We from Robust Long Abstractive Summarization?

Papers that provide insightful discussions related to long document summarization.

Our survey work has been accepted by ACM Computing Surveys. For more information, please visit An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics.

Folders for metrics used to analyze intrinsic characteristics of dataset, metrics used to analyze model outputs and arXiv human annotated data are available. Will be uploaded soon. Please do not hesitate to contact me.

@article{10.1145/3545176,
author = {Koh, Huan Yee and Ju, Jiaxin and Liu, Ming and Pan, Shirui},
title = {An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics},
year = {2022},
month = {jun},
publisher = {Association for Computing Machinery},
journal = {ACM Comput. Surv.},
address = {New York, NY, USA},
issn = {0360-0300},
url = {https://doi.org/10.1145/3545176},
doi = {10.1145/3545176},
keywords = {datasets, neural networks, document summarization, language models, Transformer}
}