🇨🇳 Paper accepted at EMNLP 2025: LiteraryQA

Created on August 22, 2025

2025

Conference: EMNLP 2025 Paper: arXiv %F0%9F%A4%97 Dataset: LiteraryQA GitHub: Code

  1. Our paper, LiteraryQA: Towards Effective Evaluation of Long-document Narrative QA, has been accepted at EMNLP Main Conference 2025!

    👏 Huge thanks to my co-authors Tommaso Bonomo and Roberto Navigli.

    Abstract

    Question Answering (QA) on narrative text poses a unique challenge to current systems, requiring a deep understanding of long, complex documents. However, the reliability of NarrativeQA, the most widely used benchmark in this domain, is hindered by noisy documents and flawed QA pairs. In this work, we introduce LiteraryQA, a high-quality subset of NarrativeQA focused on literary works. Using a human- and LLM-validated pipeline, we identify and correct low-quality QA samples while removing extraneous text from source documents. We then carry out a meta-evaluation of automatic metrics to clarify how systems should be evaluated on LiteraryQA.This analysis reveals that all n-gram-based metrics have a low system-level correlation to human judgment, while LLM-as-a-Judge evaluations, even with small open-weight models, can strongly agree with the ranking identified by humans.Finally, we benchmark a set of long-context LLMs on LiteraryQA. We release our code and data at https://github.com/sapienzaNLP/LiteraryQA.

    See you in 修正 (Suzhou)! 🇨🇳

Are we correctly evaluating LLMs on Narrative Understanding?

Narrative Question-Answering is one of the most popular benchmarks to assess the performance of LLMs on lengthy documents. The de-facto standard is NarrativeQA, a dataset containing (book, summary) pairs with summary-based questions. While valuable, NarrativeQA may potentially suffer from two issues:

  1. Quality issues: document-summary pairs were created semi-automatically, texts were scraped from webpages and questions were generated by non-expert annotators
  2. Evaluation issues: all metrics used on the datasets have been taken from other, sometimes unrelated tasks (e.g., translation), and used as-is without any proper correlation analysis

To solve these problems, we introduce LiteraryQA, a high-quality, automatically refined and manually validated subset of NarrativeQA. After fixing the data, we carry out an extensive meta evaluation of 4 n-gram-based metrics (EM, F1, ROUGE-L, METEOR), 1 neural metric (BERTScore), and thre LLMs in the LLM-as-a-Judge paradigm.

Our Key Findings:

  1. Traditional evaluation strategies poorly align with human judgemenets
  2. LLM-as-a-Judge shows highest correlation but at a computational cost – METEOR is the only n-gram-based metric still usable for quick and cheap evaluation
  3. Data quality plays a crucial role and enables higher correlation with humans on all metrics, allowing local small models to surpass closed ones
  4. Evaluating an answer against two references may not be enough in the narrative domain

Our findings highlight the need for more reliable and standardized evaluation strategies to ensure fair model comparison.

Citations

If you found our work valuable, please cite it as:

  1. Plain text
    Tommaso Bonomo, Luca Gioffré, and Roberto Navigli. 2025. LiteraryQA: Towards Effective Evaluation of Long-document Narrative QA. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 34086–34107, Suzhou, China. Association for Computational Linguistics.
    Markdown
    [LiteraryQA: Towards Effective Evaluation of Long-document Narrative QA](https://aclanthology.org/2025.emnlp-main.1729/) (Bonomo et al., EMNLP 2025)
    BibTeX
    @inproceedings{bonomo-etal-2025-literaryqa,
  title = {{L}iterary{QA}: Towards Effective Evaluation of Long-document Narrative {QA}},
  author = {Bonomo, Tommaso and Gioffr{\'e}, Luca and Navigli, Roberto},
  editor = {Christodoulopoulos, Christos and Chakraborty, Tanmoy and Rose, Carolyn and Peng, Violet},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  month = nov,
  year = {2025},
  address = {Suzhou, China},
  publisher = {Association for Computational Linguistics},
  url = {https://aclanthology.org/2025.emnlp-main.1729/},
  doi = {10.18653/v1/2025.emnlp-main.1729},
  pages = {34086--34107},
  isbn = {979-8-89176-332-6},
}