Falcon 180B
TII's large open-weight foundation model
5 trillion tokens and, at release, one of the largest openly available causal decoder-only transformer models.
Definition
Falcon 180B is a 180-billion-parameter open-weight large language model developed by the Technology Innovation Institute (TII) in Abu Dhabi, trained on 3.5 trillion tokens and, at release, one of the largest openly available causal decoder-only transformer models.
Overview
Falcon 180B extended TII's earlier Falcon model series (Falcon-7B and Falcon-40B) to a substantially larger scale, trained on the RefinedWeb dataset, a large curated and deduplicated web-text corpus, supplemented with curated sources. At the time of its release, it stood out as one of the largest models made available with open weights, offering researchers and developers access to frontier-scale model capacity without relying on a closed API. Architecturally, Falcon 180B is a standard dense decoder-only transformer, using multi-query attention to improve inference efficiency compared to full multi-head attention at that scale. Despite being dense (activating all parameters for every token, unlike mixture-of-experts models), TII engineered the training and inference stack to make a model of this size runnable on multi-GPU setups, though it remains resource-intensive to deploy compared to smaller open models. Falcon 180B was released under a custom license permitting commercial use with some conditions, distinguishing it from stricter research-only licenses used by some other large open models at the time. It demonstrated competitive performance with leading proprietary models of its era on several benchmarks, helping validate that openly released models could approach frontier-level capability. The Falcon series, including 180B, has since been followed by smaller and more efficient successor releases from TII as the broader open-model ecosystem shifted toward mixture-of-experts and more compute-efficient architectures.
Key Features
- 180 billion parameters, dense decoder-only transformer architecture
- Trained on 3.5 trillion tokens from the RefinedWeb dataset and curated sources
- Uses multi-query attention for more efficient inference at scale
- Released with open weights under a license permitting commercial use
- One of the largest open-weight models available at its release
- Developed by the Technology Innovation Institute (TII), Abu Dhabi
- Competitive benchmark performance with leading proprietary models of its era
- Predecessor context for TII's Falcon model series, including Falcon 2