📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva-3B, a sovereign LLM trained from scratch with half Italian data, scored only 4.9% on the Italian school benchmark, highlighting the challenges of scaling native-language models. This contrasts with the European debate on LLM development strategies.
Italy’s Minerva-3B, a large language model trained entirely from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored just 4.9% on the INVALSI Italian school-exam benchmark, despite extensive investment and open data practices. This result raises questions about the relationship between training scale, model size, and language-specific performance.
Developed by Sapienza University of Rome’s NLP group led by Roberto Navigli, Minerva was built on a foundation of open weights, training data, and code, utilizing Italy’s national supercomputing resources. The project aimed to demonstrate the potential of sovereign-language models, with models ranging from 350 million to 7 billion parameters, and was part of Italy’s broader national AI strategy funded through the PNRR.
Despite the significant investment, including 128 GPUs on the Leonardo supercomputer and a dataset of 2.5 trillion tokens, Minerva-3B’s performance on complex Italian language tasks was underwhelming. The 4.9% score on the INVALSI benchmark is near chance levels, indicating a disconnect between training data size and real-world language understanding. Researchers noted that while dataset composition is important, overall size and parameter count are more crucial for complex tasks, suggesting current investments may still be insufficient for deep country-specific knowledge.
This empirical finding contrasts with the European approach exemplified by Portugal’s AMÁLIA, which layered specialization onto a multilingual foundation. Minerva’s results imply that larger native-language investments may be necessary to achieve meaningful language expertise, especially at the parameter scales currently used.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
The Cranky Man's Guide to LoRA & QLoRA: Personal Lessons from a Thousand LLM Fine-Tuning Fails
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
GPU Mastery: Advanced Architectures, Evolution, and Cutting-Edge Applications in Gaming & Machine Learning
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
Advanced Language Tool Kit: Teaching the Structure of the English Language
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
End-to-End AI Evaluation: Building Effective Metrics, Pipelines, and Monitoring for LLM Systems
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for Sovereign-Language Model Strategies
The results from Minerva challenge assumptions that training large models from scratch on native-language data automatically yields high performance on language-specific tasks. Despite Italy’s substantial investment and open approach, the findings suggest that achieving true country-knowledge depth requires even larger scale investments. This has broad implications for European sovereign-LLM strategies, indicating that current parameter scales may be insufficient for complex language understanding and that future efforts might need to prioritize scaling further.
For policymakers and AI strategists, the Minerva case underscores the importance of realistic expectations about what native-language models can achieve at given scales. It also highlights the need for ongoing research into the relationship between data size, model size, and language complexity, especially for non-English languages where resources are fewer.
European Sovereign-LLM Development and the Minerva Experiment
Italy’s Minerva project represents a deliberate departure from the European debate on LLM development strategies. While Portugal’s AMÁLIA layered European Portuguese data onto a multilingual foundation, Italy opted for a from-scratch approach, training on 2.5 trillion tokens with a focus on Italian content. Funded by Italy’s PNRR and supported by national supercomputing infrastructure, Minerva aimed to demonstrate the viability of sovereign-language models that can outperform multilingual counterparts on native tasks.
Despite the technical achievements—such as open weights and extensive data—the empirical results have been sobering. The 4.9% score on the INVALSI benchmark indicates that even significant investment and effort may not be enough to produce models with deep language understanding at current parameter scales. This contrasts with the broader European narrative that emphasizes the importance of data and model scaling for language-specific AI.
“Our model’s performance indicates that current native-language investments may need to be significantly scaled up to reach desired levels of language understanding.”
— Research team, Minerva project
Unresolved Questions About Scaling and Language Depth
It remains unclear whether further scaling—either in data volume or model size—will significantly improve Minerva’s performance on complex Italian language tasks. The ongoing research aims to determine if current empirical results are a ceiling or if future iterations can overcome these limitations. Additionally, the broader applicability of these findings to other languages and contexts is still under exploration.
Next Steps for the Minerva Research Program
The Minerva team plans to continue iterative training, including experiments with larger models and more diverse data, to assess whether increased scale can bridge the performance gap. They also intend to publish updated benchmarks and analysis, contributing to the broader understanding of native-language LLM development. Policymakers and researchers will likely monitor these developments to inform future European AI strategies.
Key Questions
Why did Minerva perform poorly on the Italian benchmark despite large-scale training?
The empirical results suggest that model size and data volume are critical for complex language understanding, and current scales may still be insufficient, especially for nuanced language tasks.
Does this mean native-language models are not feasible?
Not necessarily, but it indicates that achieving high proficiency requires significant scaling beyond current efforts, and more research is needed to determine optimal strategies.
How does Minerva compare to multilingual models?
Minerva outperforms comparable multilingual models on Italian benchmarks at similar sizes, but the low INVALSI score reveals limitations in handling complex language tasks despite the focused training.
What are the broader implications for European AI policy?
The findings suggest that European sovereign-LLM strategies may need to account for the substantial scale required to develop truly effective native-language models, potentially demanding more resources and infrastructure.
Source: ThorstenMeyerAI.com
