OPT (model)
OPT (Open Pre-trained Transformer) is a family of open-source decoder-only language models released by Meta AI in 2022, ranging from 125 million to 175 billion parameters, designed to replicate GPT-3-class performance with full…
Definition
OPT (Open Pre-trained Transformer) is a family of open-source decoder-only language models released by Meta AI in 2022, ranging from 125 million to 175 billion parameters, designed to replicate GPT-3-class performance with full transparency.
Overview
OPT was released by Meta AI in May 2022 as a suite of models spanning 125M to 175B parameters, with the largest, OPT-175B, explicitly designed to match the scale and architecture of OpenAI's GPT-3. What distinguished OPT from GPT-3 was Meta's commitment to openness: the company released not just the model weights (to researchers under a noncommercial license) but also detailed logbooks documenting the training process, including hardware failures, loss spikes, and the engineering decisions made to keep training stable — a level of transparency rarely seen from a lab operating at that scale. OPT models use a standard GPT-style decoder-only transformer architecture and were trained on a mix of publicly available text datasets. Meta's stated goal was to enable reproducible research on large language models, including work on bias, toxicity, and robustness, by giving the research community access to a GPT-3-scale model without requiring API access to a closed commercial system. The accompanying "OPT-175B Logbook" became a widely cited resource for understanding the practical, often messy realities of training models at that scale. OPT's release predates and helped set the stage for Meta's later, more consequential open-model efforts, most notably the Llama series, which built on lessons from OPT and adopted a more permissive licensing approach as Meta's open-source AI strategy matured. While OPT models are now considered outdated relative to modern open-weight models, they remain historically important as one of the first serious attempts by a major AI lab to openly replicate and document a GPT-3-scale system.
Key Concepts
- Family of models from 125M to 175B parameters
- OPT-175B designed to match GPT-3's scale and architecture
- Released with detailed training logbooks documenting the process
- Decoder-only transformer trained on public text datasets
- Released by Meta AI under a noncommercial research license
- Precursor to Meta's later Llama model series