publications

Under review

1. Under review

   On training LLMs from scratch

   Luca Moroni, Raffaele Pisano, Bruno Francesco Nocera, and 6 more authors

   2026

   Under review — not the submitted title
2. Under review

   On evaluating LLMs’ cultural knowledge recall

   Luca Gioffré, Luca Moroni, Lu Xu, and 10 more authors

   2026

   Under review — not the submitted title
3. Under review

   On teaching LLMs to understand oral vernaculars

   Luca Gioffré, Simone Laganà, Elena Marafatto, and 1 more author

   2026

   Under review — not the submitted title

2026

1. CLiC-it 2026

   Dromedario 3: Localizing the Tülu 3 Dataset to Italian

   Luca Gioffré, Marina Iuliana Aur, Francesco Ortame, and 4 more authors

   In Proceedings of the Twelfth Italian Conference on Computational Linguistics (CLiC-it 2026), Sep 2026

   Abs

   Despite growing interest in supervised fine-tuning (SFT) datasets, nearly all existing resources are English-only or English-dominated, leaving Italian without a comparable resource. To bridge this gap, we introduce Dromedario 3, a large-scale Italian instruction-following dataset of approximately 630K translated instruction-response pairs derived from the Tülu 3 SFT mixture. Rather than translating the entire mixture, we first apply a multi-stage filtering pipeline. Specifically, we combine source-dataset filtering with automatic task and domain classification based on the Super-NaturalInstructions taxonomy to exclude language-bound tasks, while existing non-English instances are retained untranslated to preserve multilingual diversity. The selected instances are translated using TranslateGemma-27B and post-processed to remove faulty outputs. Manual validation confirms the reliability of both the automatic classification (accuracy above 87% for task labels and 95% for domain labels) and the machine translations (correctness above 95%). We further conduct a mixing study evaluating five SFT recipes with varying Italian-to-English compositions on parallel Italian and English benchmarks using two models, Llama 3.1 8B and Minerva 7B. The model with stronger Italian pretraining benefits substantially from Italian SFT data in generative settings, while recipes combining both languages offer the most robust trade-off between gains in Italian and preservation of English capabilities.

2. EVALITA 2026

   INDAQA2 - A Large Italian Narrative QA Benchmark: A CALAMITA 2026 Challenge

   Luca Gioffré, Luca Moroni, Alberte Fernández-Castro, and 3 more authors

   In Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026), Feb 2026

   Abs

   Long-context comprehension and reasoning remain largely underexplored in the evaluation of Italian Large Language Models (LLMs). Existing Italian benchmarks primarily focus on short or medium-length inputs, offering limited insight into models’ ability to process extended narratives. To address this gap, we introduce INDAQA2, a substantially revised and expanded version of INDAQA, a benchmark for narrative question answering on original Italian literary texts. The new version comprises an expanded corpus of 461 total books, introduces a multiple-choice question answering format alongside the original open-ended tasks, and features manually curated texts drawn exclusively from works originally written in Italian, thus avoiding artifacts introduced by translation. The benchmark evaluates long-context understanding over complete books of up to 250K tokens, testing complementary comprehension skills through a dual-structure design: global narrative understanding, assessed via questions derived from book summaries, and local precision, assessed via questions grounded in specific passages and entity-level details. By supporting both open-ended and multiple-choice question answering formats, INDAQA2 enables evaluation of both generative capabilities and discriminative reasoning, facilitating comprehensive and scalable comparison across models. Our evaluation of several Italian-specialized and multilingual models reveals significant performance disparities across task formats and highlights limitations in how current Italian models utilize extended contexts.

2025

1. CLiC-it 2025

   What We Learned from Continually Training Minerva: A Case Study on Italian

   Luca Moroni, Tommaso Bonomo, Luca Gioffré, and 6 more authors

   In Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025), Sep 2025

   Abs

   Modern Large Language Models (LLMs) are commonly trained through a multi-stage pipeline encompassing pretraining and supervised finetuning. While recent studies have extensively investigated the benefits of continual pretraining on high-quality data, these efforts have focused primarily on English. In this work, we explore the effectiveness of various data mixtures in a continual pretraining setting to enhance performance on Italian-language tasks. Leveraging Minerva-7B, a fully open-source LLM pretrained on a corpus composed of 50% Italian, we define and evaluate three distinct data recipes–comprising mathematical, encyclopedic, and copyrighted content–spanning both Italian and English. We also investigate the effect of extending the model’s context window during continual pretraining on its ability to handle long-context tasks. To support our evaluation, we introduce INDAQA, a new benchmark for narrative question answering in Italian. Our results reveal that both data composition and increased context length substantially improve performance, offering valuable insights into continual pretraining strategies for less represented languages within an open scientific framework.

2. EMNLP 2025

   LiteraryQA: Towards Effective Evaluation of Long-document Narrative QA

   Tommaso Bonomo, Luca Gioffré, and Roberto Navigli

   In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Nov 2025

   Abs DOI

   Question Answering (QA) on narrative text poses a unique challenge to current systems, requiring a deep understanding of long, complex documents. However, the reliability of NarrativeQA, the most widely used benchmark in this domain, is hindered by noisy documents and flawed QA pairs. In this work, we introduce LiteraryQA, a high-quality subset of NarrativeQA focused on literary works. Using a human- and LLM-validated pipeline, we identify and correct low-quality QA samples while removing extraneous text from source documents. We then carry out a meta-evaluation of automatic metrics to clarify how systems should be evaluated on LiteraryQA.This analysis reveals that all n-gram-based metrics have a low system-level correlation to human judgment, while LLM-as-a-Judge evaluations, even with small open-weight models, can strongly agree with the ranking identified by humans.Finally, we benchmark a set of long-context LLMs on LiteraryQA. We release our code and data at https://github.com/sapienzaNLP/LiteraryQA.

3. ACL 2025

   Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering

   Francesco Maria Molfese, Luca Moroni, Luca Gioffré, and 3 more authors

   In Findings of the Association for Computational Linguistics: ACL 2025, Jul 2025

   Abs DOI

   One of the most widely used tasks for evaluating Large Language Models (LLMs) is Multiple-Choice Question Answering (MCQA). While open-ended question answering tasks are more challenging to evaluate, MCQA tasks are, in principle, easier to assess, as the model’s answer is thought to be simple to extract and is compared directly to a set of predefined choices. However, recent studies have started to question the reliability of MCQA evaluation, showing that multiple factors can significantly impact the reported performance of LLMs, especially when the model generates free-form text before selecting one of the answer choices. In this work, we shed light on the inconsistencies of MCQA evaluation strategies, which can lead to inaccurate and misleading model comparisons. We systematically analyze whether existing answer extraction methods are aligned with human judgment, and how they are influenced by answer constraints in the prompt across different domains. Our experiments demonstrate that traditional evaluation strategies often underestimate LLM capabilities, while LLM-based answer extractors are prone to systematic errors. Moreover, we reveal a fundamental trade-off between including format constraints in the prompt to simplify answer extraction and allowing models to generate free-form text to improve reasoning. Our findings call for standardized evaluation methodologies and highlight the need for more reliable and consistent MCQA evaluation practices.