Top-p Sampling
Top-p sampling, also called nucleus sampling, is a decoding strategy that selects the next token from the smallest set of candidates whose cumulative probability meets a specified threshold p, discarding the long tail of unlikely tokens.
Definition
Top-p sampling, also called nucleus sampling, is a decoding strategy that selects the next token from the smallest set of candidates whose cumulative probability meets a specified threshold p, discarding the long tail of unlikely tokens.
Overview
Top-p sampling was introduced as an alternative to simpler decoding strategies like top-k sampling, which always considers a fixed number of top candidates regardless of how the model's confidence is distributed. Instead, top-p dynamically adjusts the candidate pool size at each generation step: the model's next-token probabilities are sorted from highest to lowest, and tokens are added to the sampling pool until their cumulative probability reaches the threshold p (for example, 0.9). If the model is very confident, this nucleus might contain only one or two tokens; if it is uncertain, the nucleus expands to include many more. This adaptiveness is the key advantage over top-k sampling: in situations where the model has a sharply peaked distribution, top-p avoids needlessly including low-probability, potentially incoherent tokens that a fixed top-k would still allow. In situations where many tokens are plausible, top-p allows for more diversity than a rigid top-k cutoff would. The technique was proposed in the 2019 paper "The Curious Case of Neural Text Degeneration," which showed it produced more natural, less repetitive text than pure greedy or beam search decoding for open-ended generation tasks. Top-p is typically used together with temperature — temperature reshapes the overall probability distribution, and top-p then trims the tail before a token is randomly sampled from the remaining nucleus. Common defaults across LLM APIs set top-p around 0.9 to 1.0. Lowering top-p tightens the pool of candidate tokens and produces more focused, conservative output, similar in effect to lowering temperature, though the two parameters operate through different mechanisms and are sometimes tuned together for finer control over generation quality.
Key Concepts
- Dynamically sizes the candidate token pool based on cumulative probability mass
- Also known as nucleus sampling
- Adapts pool size to model confidence, unlike fixed top-k sampling
- Introduced in "The Curious Case of Neural Text Degeneration" (Holtzman et al., 2019)
- Commonly combined with temperature for fine-grained generation control
- Reduces repetitive or degenerate text compared to greedy decoding
- Typical default values range from 0.9 to 1.0
- Lower p values produce more focused, conservative output