Grok-1
By xAI
0 license in March 2024, making its architecture and weights freely available.
Definition
Grok-1 is the first large language model developed by Elon Musk's xAI, announced in November 2023 as a 314-billion-parameter mixture-of-experts model, and later open-sourced under the Apache 2.0 license in March 2024, making its architecture and weights freely available.
Overview
Grok-1 was xAI's debut model, introduced as the engine behind the initial version of the Grok chatbot, which xAI made available to X (formerly Twitter) Premium+ subscribers. Positioned from the outset with a distinctive brand identity — a 'rebellious' personality willing to answer questions other chatbots might decline, along with real-time access to information from X — Grok-1 was trained by xAI's research team using a custom training and inference stack built largely from scratch over a compressed development timeline of a few months. Architecturally, Grok-1 is a mixture-of-experts (MoE) transformer with 314 billion total parameters, of which roughly a quarter are active for any given token, a design choice aimed at balancing model capacity against inference cost. At its initial announcement, xAI positioned Grok-1 as competitive with contemporaneous models like GPT-3.5 on a range of standard reasoning and knowledge benchmarks, though trailing the most capable models of the time, such as GPT-4. In March 2024, xAI open-sourced Grok-1's base model weights and architecture under the Apache 2.0 license, releasing the raw pretrained (not instruction-tuned or RLHF-aligned) model via GitHub and Hugging Face. This made Grok-1 one of the larger openly released mixture-of-experts models available to the research community at the time, notable for its scale even though, as a raw base model without chat fine-tuning, it required additional work to use effectively as a conversational assistant. Grok-1 was subsequently succeeded within xAI's own lineup by Grok-2, Grok-3, and Grok-4, each bringing substantial capability improvements, with xAI generally not open-sourcing the weights of these more capable successor models at their initial release.
Key Features
- xAI's first large language model, powering the original Grok chatbot
- 314-billion-parameter mixture-of-experts (MoE) transformer architecture
- Roughly a quarter of parameters active per token due to MoE design
- Integrated with X (Twitter) for real-time information access
- Open-sourced under the Apache 2.0 license in March 2024
- Released as a raw pretrained base model without instruction tuning
- Positioned at release as competitive with GPT-3.5-class models
- Trained using a custom infrastructure and training stack built by xAI