RedPajama
By Together AI and collaborators
2-trillion-token dataset and a family of trained models (RedPajama-INCITE).
Definition
RedPajama is an open effort, led by Together AI with academic and industry collaborators, to reproduce the training dataset and models described in Meta's LLaMA paper, releasing both an open 1.2-trillion-token dataset and a family of trained models (RedPajama-INCITE).
Overview
When Meta released LLaMA in February 2023, it described its training dataset in detail but did not release the data itself, and initially restricted model weights to research access. RedPajama, launched in April 2023 by Together AI in collaboration with organizations including Ontocord.ai, ETH DS3Lab, Stanford CRFM, Hazy Research, and MILA Québec AI Institute, set out to fill that gap by reconstructing a dataset following the same composition described in the LLaMA paper: CommonCrawl, C4, GitHub, Wikipedia, books, ArXiv, and StackExchange, totaling roughly 1.2 trillion tokens. The project released the dataset first, followed by a family of trained models called RedPajama-INCITE, developed in partnership with several of the same collaborators plus additional compute partners, at 3B and 7B parameter scales, including base, instruction-tuned, and chat variants. These models were explicitly positioned as fully open (data, training code, and weights) alternatives to LLaMA, at a time when LLaMA's own weights required a research license and were only accessible after a leak made them widely available outside their intended terms. RedPajama was an important early entry in the wave of open LLaMA-reproduction efforts that also included OpenLLaMA and others, collectively demonstrating that open, community-driven data curation could approximate the training recipes of closed or restricted frontier labs. A follow-up dataset, RedPajama-V2, released in late 2023, expanded to over 30 trillion raw tokens across five languages with extensive quality-signal annotations, shifting focus from being a LLaMA reproduction specifically to being a general-purpose, richly annotated web-scale pretraining corpus used well beyond the original RedPajama-INCITE models.
Key Concepts
- Reconstructs a dataset matching the composition described in Meta's LLaMA paper
- RedPajama-1T dataset totals roughly 1.2 trillion tokens across CommonCrawl, GitHub, Wikipedia, books, ArXiv, and more
- RedPajama-INCITE models released at 3B and 7B scales with base, instruct, and chat variants
- Fully open data, training code, and model weights
- RedPajama-V2 expanded to over 30 trillion tokens with quality-signal annotations
- Multi-organization collaboration including Together AI, Stanford CRFM, and ETH DS3Lab
- Widely used as a base pretraining corpus beyond the original INCITE models