PaLM
By Google Research
PaLM (Pathways Language Model) is a 540-billion-parameter dense large language model developed by Google Research and announced in April 2022, trained using Google's Pathways system for efficiently distributing model training across…
Definition
PaLM (Pathways Language Model) is a 540-billion-parameter dense large language model developed by Google Research and announced in April 2022, trained using Google's Pathways system for efficiently distributing model training across thousands of accelerator chips.
Overview
PaLM was Google's flagship large language model prior to the Gemini family, notable both for its scale and for the infrastructure used to train it. It takes its name from Pathways, a Google system designed to train a single model efficiently across multiple TPU pods without the communication bottlenecks that typically limit large-scale distributed training. Like GPT-3, PaLM is a decoder-only Transformer trained with a standard next-token prediction objective, but at 540 billion parameters it was significantly larger than most publicly documented models at the time of release. Google reported that PaLM achieved state-of-the-art results on a wide range of language, reasoning, and code-generation benchmarks, and demonstrated notable capability in multi-step reasoning tasks when combined with chain-of-thought prompting. PaLM was followed by PaLM 2 in 2023, a more compute-efficient successor trained on a more multilingual and code-heavy dataset, which powered products including Google Bard (the predecessor to Gemini) and various Google Cloud generative AI offerings. Both PaLM and PaLM 2 were eventually superseded by the Gemini model family, which unified Google's language, vision, and multimodal research efforts. PaLM's research contributions — particularly around Pathways-based distributed training and evidence that reasoning ability improves with scale — influenced both Google's subsequent models and the broader field's understanding of how scaling laws apply to emergent capabilities in LLMs.
Key Features
- 540 billion parameters in a dense (non-sparse) Transformer architecture
- Trained using Google's Pathways system across thousands of TPU v4 chips
- Strong performance on multi-step reasoning tasks via chain-of-thought prompting
- State-of-the-art results reported on numerous language and code benchmarks at release
- Succeeded by PaLM 2 (2023), a more efficient, multilingual successor
- Research foundation for Google Bard before the transition to Gemini