Greedy Decoding
Greedy decoding is a text generation strategy in which a language model selects the single highest-probability token at each generation step, without considering how that choice affects the probability of future tokens in the sequence.
Definition
Greedy decoding is a text generation strategy in which a language model selects the single highest-probability token at each generation step, without considering how that choice affects the probability of future tokens in the sequence.
Overview
When a language model generates text, it produces a probability distribution over its entire vocabulary at each step, representing how likely each possible next token is given everything generated so far. Greedy decoding is the simplest possible strategy for turning this distribution into an actual output: at every step, it deterministically picks the single token with the highest probability, appends it to the sequence, and repeats the process for the next token, continuing until an end-of-sequence token is produced or a maximum length is reached. Because it always makes the locally optimal choice at each individual step, greedy decoding is fast, fully deterministic (the same input always produces the same output), and simple to implement, requiring no additional search or sampling machinery beyond taking the argmax of the model's output distribution. However, this local optimality does not guarantee a globally optimal or even high-quality overall sequence — a token that looks best in isolation at one step can lead the model into a much lower-probability, less coherent continuation later, a problem that more sophisticated search strategies like beam search are designed to mitigate by keeping multiple candidate sequences in consideration simultaneously. Greedy decoding also tends to produce repetitive, generic, or overly predictable text, especially in open-ended generation tasks like storytelling or conversation, because it always defaults to the statistically safest, most common continuation rather than allowing for the variation and creativity that sampling-based methods like top-k sampling, nucleus (top-p) sampling, or temperature-based sampling can introduce. Despite these quality tradeoffs, greedy decoding remains useful and often preferred for tasks where determinism and speed matter more than diversity — such as tasks with a single clearly correct answer, translation of unambiguous text, or situations requiring fully reproducible outputs for testing and debugging — and it serves as a simple, fast baseline against which more sophisticated decoding strategies are compared.
Key Concepts
- Selects the single highest-probability token at every generation step
- Fully deterministic — identical inputs always produce identical outputs
- Fast and simple, requiring no search or sampling beyond an argmax
- Does not guarantee a globally optimal or most probable full sequence
- Prone to producing repetitive or generic text in open-ended generation
- Contrasted with beam search, which explores multiple candidate sequences
- Contrasted with sampling methods like top-k and nucleus sampling
- Preferred when determinism, speed, or reproducibility matter most