📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva-3B, a European sovereign language model trained from scratch, performs poorly on Italian academic benchmarks despite large-scale data. This challenges assumptions about investment and scale in national AI projects.
Italy’s Minerva-3B, a large language model trained from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored only 4.9% on the INVALSI Italian school-exam benchmark, despite significant national investment. This performance raises questions about the effectiveness of large-scale native-language training for complex language understanding, challenging assumptions about the relationship between data scale and model proficiency.
Minerva was developed by Sapienza University of Rome’s NLP group, led by Roberto Navigli, with support from Italy’s FAIR consortium and CINECA’s supercomputing infrastructure. The project aimed to create a high-quality, open-weight LLM specifically for Italian, training on 2.5 trillion tokens, with nearly half being Italian data. The model family, ranging from 350 million to 7 billion parameters, outperformed comparable multilingual models on Italian benchmarks, demonstrating technical success in scale and architecture.
However, empirical evaluations revealed a stark contrast: Minerva-3B scored just 4.9% on the INVALSI Italian school exams, a result near chance level. Researchers noted that while dataset composition and size are important, the overall scale of data and parameters are more crucial for handling complex language tasks. The result suggests that even large-scale native-language training may not be sufficient to develop deep country-specific knowledge or academic language proficiency.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Large Language Models (LLMs)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
AI Engineering: Building Applications with Foundation Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
Mastering Small Language Models: A Practical Guide to Building Lightweight NLP Systems with Python, Transformers, and Quantization Techniques
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
Supercomputing for Artificial Intelligence: Foundations, Architectures, and Scaling Deep Learning Workloads
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign-Language Models
This finding challenges the assumption that larger datasets and parameters inherently lead to better language understanding in sovereign models. It suggests that the European approach to developing national LLMs must consider the actual scale of native-language investment necessary to achieve meaningful expertise. Italy’s experience indicates that even substantial investments may not produce the desired academic and complex language capabilities, prompting a reevaluation of strategic priorities and resource allocations in national AI initiatives.
Background on Italy’s Sovereign LLM Effort
Italy embarked on a pioneering effort to develop a European sovereign LLM, building Minerva from scratch with a focus on Italian language and data. The project involved extensive collaboration among academic institutions, government agencies, and supercomputing infrastructure, supported by Italy’s National Strategy for Artificial Intelligence and PNRR funding. Previous efforts like Portugal’s AMÁLIA model took a different approach, layering specialization onto multilingual foundations, but Italy chose to prioritize native-language training at scale, resulting in a model that outperformed multilingual counterparts on benchmarks.
Despite these successes, the evaluation results reveal a significant gap between technical capability and real-world academic language understanding, emphasizing that scale alone may not suffice for complex language tasks.
“Our results suggest that the investment in data and parameters must reach a certain threshold to develop country-specific expertise, which we are still exploring.”
— Research team member
Unresolved Questions About Scale and Effectiveness
It remains unclear what specific scale of data and parameters is truly necessary to achieve high-level language understanding in national models. The results from Minerva-3B suggest current scale may be insufficient, but the precise thresholds and the role of data quality versus quantity are still under investigation. Additionally, the potential for iterative training or alternative architectures to improve performance is not yet determined.
Future Steps for Italy’s Sovereign AI Strategy
The Minerva team plans ongoing iterations, including continued training and fine-tuning, to address the performance gaps. Researchers will also analyze the impact of dataset composition and explore architectural modifications. Policymakers and stakeholders may reassess investment levels and strategic priorities based on these findings, potentially shifting focus toward quality, diversity, or larger-scale data collection efforts for future models.
Key Questions
Why did Minerva-3B perform poorly on Italian academic tests?
The evaluation indicates that despite large-scale native-language training, the model lacks the depth of understanding needed for complex academic tasks, suggesting that scale alone is not sufficient for high proficiency in such areas.
Does this mean large-scale training is ineffective for national models?
Not necessarily. It highlights that scale must reach certain thresholds and be complemented by quality and task-specific training to produce meaningful country-specific expertise.
What are the implications for other European countries developing sovereign LLMs?
They may need to reevaluate their investment strategies, considering not just data volume but also the scale of native-language data and the architecture’s ability to handle complex language understanding.
Will Italy increase its investment to improve Minerva’s performance?
The team is likely to continue refining and expanding training efforts, but specific plans depend on ongoing research outcomes and strategic priorities.
Source: ThorstenMeyerAI.com
