BLOOM
By BigScience
BLOOM (BigScience Large Open-science Open-access Multilingual Language Model) is a 176-billion-parameter autoregressive large language model trained collaboratively by over 1,000 researchers as part of the BigScience workshop, released in…
Definition
BLOOM (BigScience Large Open-science Open-access Multilingual Language Model) is a 176-billion-parameter autoregressive large language model trained collaboratively by over 1,000 researchers as part of the BigScience workshop, released in July 2022 as a fully open alternative to closed models like GPT-3.
Overview
BLOOM was created to answer a specific gap in AI research: at the time, the most capable large language models — GPT-3 among them — were proprietary, with weights, training data, and methodology kept private. BigScience, a year-long open research collective coordinated by Hugging Face and hosted on the Jean Zay supercomputer in France, set out to build a model of comparable scale that anyone could inspect, run, and fine-tune. Architecturally, BLOOM is a decoder-only Transformer-based model trained with a standard autoregressive next-token objective, similar in structure to GPT-3. What distinguishes it is its training data: the ROOTS corpus spans 46 natural languages and 13 programming languages, deliberately weighted to include underrepresented languages such as Swahili, Yoruba, and Bengali alongside English, French, and Chinese — a marked contrast to English-dominant training sets used by most contemporary LLMs. BLOOM's release included full model weights, training logs, and documentation of the data pipeline, licensed under the RAIL (Responsible AI License), which permits broad research and commercial use with some behavioral restrictions. This made it a foundational resource for researchers studying multilingual generalization, model interpretability, and the effects of training data composition on model behavior. While later open-weight models such as Llama and Mistral have generally surpassed BLOOM on standard benchmarks, it remains historically significant as one of the first large-scale demonstrations that state-of-the-art-scale language models could be built entirely in the open, shaping norms around transparency that influence how open models are documented today.
Key Features
- 176 billion parameters, comparable in scale to GPT-3
- Trained on the ROOTS corpus spanning 46 natural languages and 13 programming languages
- Fully open weights, training code, and documentation released publicly
- Decoder-only Transformer architecture with autoregressive next-token prediction
- Released under the RAIL license permitting broad research and commercial use
- Built collaboratively by over 1,000 researchers across 70+ countries
- Trained on the Jean Zay public supercomputer in France