The Common Corpus: An Open Training Dataset for AI Goes Global, and So Should Support for It

The burgeoning field of artificial intelligence (AI), particularly the development of large language models (LLMs), is grappling with a fundamental challenge: the sourcing of its vast training data. As AI capabilities accelerate, the demand for comprehensive datasets to fuel these powerful models has intensified, leading many companies to adopt a “scrape-it-all” approach from the internet. This widespread practice, however, is increasingly mired in legal uncertainties surrounding copyright infringement, as numerous lawsuits are currently underway, with outcomes yet to definitively shape the legal landscape. Amidst this contentious environment, a significant development has emerged with the expansion and enhanced multilingual capabilities of the Common Corpus, an open and permissively licensed dataset designed to provide a transparent and legally sound alternative for AI training.

The Challenge of AI Training Data Acquisition

The creation of sophisticated LLMs, the engines behind many current generative AI applications, necessitates colossal amounts of data. Estimates suggest that models like OpenAI’s GPT series have been trained on datasets comprising trillions of tokens, which can represent words or parts of words. Historically, this data has been aggregated through broad web scraping, often encompassing copyrighted material without explicit permission. This method, while efficient in sheer volume acquisition, has ignited a firestorm of legal disputes. Creators and copyright holders argue that their intellectual property is being exploited without consent or compensation, leading to a wave of litigation that could fundamentally alter the economics and ethics of AI development. The ambiguity surrounding fair use and the legal permissibility of using copyrighted content for AI training remains a critical hurdle for the industry.

An Open Alternative: The Genesis and Evolution of the Common Corpus

In response to this growing challenge, the French startup Pleias, in collaboration with the AI Alliance, launched the Common Corpus just over a year ago. This initiative aimed to create a comprehensive, high-quality dataset composed exclusively of publicly available or permissively licensed materials. The core philosophy behind the Common Corpus is to provide a transparent, auditable, and legally compliant foundation for AI development, thereby sidestepping the ethical and legal quagmires associated with opaque data sourcing.

The initial release of the Common Corpus was a significant undertaking, boasting over 2 trillion tokens. Its key characteristics, as outlined by the AI Alliance, underscore its commitment to responsible AI development:

• Truly Open: The dataset contains only data that is permissively licensed, with clear provenance documentation. This ensures that developers can use the data with confidence, knowing its legal standing.
• Multilingual: While predominantly featuring English and French data, the corpus extends its reach to over 30 languages, with at least one billion tokens for each. This inclusivity is crucial for developing AI systems that can serve a global audience.
• Diverse: The dataset encompasses a wide array of content, including scientific articles, governmental and legal documents, source code, and cultural heritage materials like books and newspapers. This breadth ensures the development of versatile and knowledgeable AI models.
• Extensively Curated: Beyond mere aggregation, the Common Corpus undergoes rigorous curation. This includes correcting spelling and formatting errors in digitized texts, removing harmful and toxic content, and filtering out material with low educational value. This meticulous approach aims to enhance the quality and safety of AI models trained on its data.

Expanding Horizons: The Latest Global Update

A significant recent update to the Common Corpus has further bolstered its value and reach. The dataset has now grown to over 2.267 trillion tokens, a testament to its ongoing development and the increasing availability of open data. Crucially, this expansion includes major additions of material from China, Japan, Korea, Brazil, India, Africa, and Southeast Asia. This geographic diversification is a pivotal step towards creating truly global AI systems.

Specifically, the latest release details a more robust multilingual offering:

• Eight languages with over 10 billion tokens: English, French, German, Spanish, Italian, Polish, Greek, and Latin.
• Thirty-three languages with over 1 billion tokens: This broadens the corpus’s applicability across numerous linguistic communities.

This enhanced multilingual capacity is vital for fostering AI development that is inclusive and representative of the world’s diverse populations. By providing accessible training data in a wider array of languages, the Common Corpus directly addresses the historical bias towards English-centric AI development.

Structuring the Common Corpus: A Categorical Approach

The Common Corpus is organized into five primary categories, each designed to cater to specific aspects of AI development:

• OpenGovernment: This category includes Finance Commons, a multimodal dataset (text and PDF) of financial documents from governmental and regulatory bodies, and Legal Commons, a collection of legal and administrative texts. These datasets are invaluable for training AI models that can understand and process complex regulatory and financial information.
• OpenCulture: This segment houses cultural heritage data, such as digitized books and newspapers, with a significant portion dating back to the 18th and 19th centuries, and even earlier. This provides a rich source for training AI models on historical context, literary analysis, and cultural understanding.
• OpenScience: Primarily sourced from publicly available academic and scientific publications, often in PDF format, this category is essential for developing AI capable of scientific research, data analysis, and understanding complex scientific literature.
• OpenWeb: This category includes datasets derived from public domain YouTube videos (YouTube Commons transcripts) and websites like Stack Exchange. It offers a wealth of conversational data and technical information, crucial for training chatbots and question-answering systems.
• OpenSource: This component comprises code collected from permissively licensed GitHub repositories. It is fundamental for training AI models that can understand, generate, and assist with software development.

Legal and Ethical Compliance: Exceeding Regulatory Standards

One of the most compelling advantages of the Common Corpus is its adherence to stringent legal and ethical standards. Pleias has proactively addressed regulatory requirements, including the European Union’s AI Act, by ensuring clear provenance and the use of permissively licensed data. Furthermore, the project has implemented custom procedures to ensure GDPR compliance, specifically through the development of multilingual Personally Identifiable Information (PII) removal processes. This makes the Common Corpus an attractive option for enterprises seeking to build secure, compliant, and auditable AI models.

The proactive removal of content with high "toxicity scores" is another significant feature. This curation process helps to mitigate the risk of training AI models that exhibit harmful biases or generate offensive content, a persistent concern in the generative AI landscape. By providing a cleaner, more ethically sound dataset, the Common Corpus contributes to the development of safer and more responsible AI.

The Power of Openness and Permissive Licensing

The Common Corpus stands as a powerful demonstration of the benefits offered by open access and permissive copyright licensing. It enables the training of AI models that are compatible with the Open Source Initiative’s definition of open-source AI, which includes the freedom to use the AI for "any purpose and without having to ask for permission." This openness is crucial for fostering innovation, collaboration, and the democratization of AI technology.

The multilingual nature of the Common Corpus also positions it as an ideal resource for initiatives aimed at creating "public AI" systems, such as those being explored by the European Union. Such public AI infrastructure, built on open-source principles and transparent data, could serve as a vital counterpoint to proprietary AI systems that often operate as "black boxes," with their data sources and methodologies obscured.

Broad Support and Future Implications

The development of the Common Corpus has garnered significant support from various governmental and organizational entities, highlighting its strategic importance:

• The AI Alliance and the French Ministry of Culture: Support from these entities, particularly within the framework of the Alliance for Language Technologies EDIC (ALT-EDIC), underscores the commitment to building robust language technologies based on open principles.
• Wikimedia Enterprise and Wikidata/Wikimedia Germany: The partnership with these organizations, known for their vast open knowledge repositories, provides valuable data and expertise.
• Libraries Without Borders: This partner’s assistance in extending support for low-resource languages is critical for ensuring global inclusivity.
• Jean Zay (Eviden, Idris), Tracto AI, and Mozilla: These organizations have provided essential infrastructure and processing support, demonstrating a collaborative effort to advance open AI.

The unique advantages offered by the Common Corpus present a compelling case for increased government and publisher investment. For governments, it represents an opportunity to foster national AI capabilities without reliance on proprietary, often foreign-developed, AI systems. Publishers, too, stand to benefit by supporting a resource explicitly designed to navigate the complex copyright issues that currently plague the generative AI field. By investing in the Common Corpus, they can contribute to a sustainable AI ecosystem that respects intellectual property while enabling innovation.

As the AI industry continues to mature, the debate over data sourcing will undoubtedly intensify. The Common Corpus offers a tangible, scalable, and legally sound alternative to the prevalent data-scraping practices. Its continued growth, expansion into new languages and regions, and unwavering commitment to openness position it as a cornerstone for the future of responsible and inclusive AI development. The broader adoption and support for such initiatives are not merely beneficial; they are increasingly essential for shaping an AI future that is ethical, equitable, and beneficial for all.