Our paper, What We Learned from Continually Training Minerva: A Case Study on Italian, has been accepted at CLiC-it Conference 2025!

👏 Huge thanks to my co-authors Luca Moroni, Tommaso Bonomo, Lu Xu, Domenico Fedele, Leonardo Colosi, Andrei Stefan Bejgu, Alessandro Scirè and Roberto Navigli.

Abstract

Modern Large Language Models (LLMs) are commonly trained through a multi-stage pipeline encompassing pretraining and supervised finetuning. While recent studies have extensively investigated the benefits of continual pretraining on high-quality data, these efforts have focused primarily on English. In this work, we explore the effectiveness of various data mixtures in a continual pretraining setting to enhance performance on Italian-language tasks. Leveraging Minerva-7B, a fully open-source LLM pretrained on a corpus composed of 50% Italian, we define and evaluate three distinct data recipes–comprising mathematical, encyclopedic, and copyrighted content–spanning both Italian and English. We also investigate the effect of extending the model’s context window during continual pretraining on its ability to handle long-context tasks. To support our evaluation, we introduce INDAQA, a new benchmark for narrative question answering in Italian. Our results reveal that both data composition and increased context length substantially improve performance, offering valuable insights into continual pretraining strategies for less represented languages within an open scientific framework.

See you in Cagliari! 🇮🇹

What We Learned from Continually Training Minerva: Insights for Italian LLM Development

Training large language models for less-represented languages presents unique challenges. In this work, we investigated how different data recipes and context length extensions affect Italian LLM performance.

We used Minerva-7B, a fully open-source bilingual model, pretrained on 50% Italian and 50% English content, to test three data recipes during continual pretraining: mathematical, encyclopedic, and copyrighted literary content from both Italian and English. We also explored extending the model’s context window to handle longer documents.

To evaluate long-context understanding, we created INDAQA, the Italian Narrative Dataset for Question-Answering, the first narrative long-context benchmark for Italian.

Our Key Findings:

Context Extension Beats Brute Force: Extending Minerva’s context window to handle chapter- or book-length texts achieved state-of-the-art performance on long Italian documents. Our models outperformed both Italian-adapted models fine-tuned from English foundations and models trained on many more trillion tokens. The takeaway: strategic continual pretraining on well-designed Italian data can compete with—and surpass—the brute-force approach of adapting massive English-centric models.
Multiple-Choice Tests Mislead on Cultural Knowledge When testing cultural knowledge using multiple-choice questions, results were misleading—models could score well through pattern matching without genuine understanding. But with open-ended question answering, where models generate free-form responses, Minerva excelled and surpassed all competitors. For fair evaluation of language-specific capabilities, we need formats that truly test comprehension and generation.

We contribute INDAQA to the community and demonstrate the importance of evaluation format when assessing language-specific models.

[LLM]: Large Language Model _[OE]: Open-ended, also known as _free-form

🇮🇹 Paper accepted at CLiC-it 2025: What We Learned from Continually Training Minerva

What We Learned from Continually Training Minerva: Insights for Italian LLM Development

Citations

Plain text

Markdown

BibTeX