ROUGE Score
ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a set of recall-focused metrics for evaluating automatic text summarization by measuring the overlap of n-grams, word sequences, or word pairs between a generated summary and one…
Definition
ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a set of recall-focused metrics for evaluating automatic text summarization by measuring the overlap of n-grams, word sequences, or word pairs between a generated summary and one or more reference summaries.
Overview
ROUGE was introduced in 2004 specifically to address the evaluation needs of automatic summarization, where the goal is often to capture as much of the key information from the reference as possible, making recall a more natural focus than the precision-oriented approach of BLEU. The ROUGE family includes several variants: ROUGE-N measures n-gram overlap (ROUGE-1 for unigrams, ROUGE-2 for bigrams, and so on), ROUGE-L measures the longest common subsequence between generated and reference text, capturing sentence-level structural similarity without requiring consecutive matches, and ROUGE-S measures skip-bigram overlap, allowing gaps between matched word pairs. Most ROUGE implementations report precision, recall, and an F1 score for each variant, giving a fuller picture than a single number. Because it is recall-oriented, ROUGE rewards summaries that capture a large portion of the reference's content, which suits summarization's goal of information coverage, though it can still be gamed by longer outputs that happen to include more overlapping words without necessarily being better summaries. ROUGE remains a standard benchmark metric reported in most academic summarization papers and is also used for evaluating machine translation and other text generation tasks where multiple reasonable outputs exist. Like BLEU, its core weakness is that it measures surface lexical overlap rather than true semantic equivalence — a summary that paraphrases the reference accurately using different words can score poorly despite being an excellent summary. This has driven adoption of complementary metrics such as BERTScore and, increasingly, LLM-as-a-judge evaluation, which better capture semantic fidelity, factual consistency, and coherence beyond n-gram matching.
Key Concepts
- Recall-oriented metric family designed for automatic summarization evaluation
- ROUGE-N measures n-gram overlap (ROUGE-1, ROUGE-2, etc.)
- ROUGE-L measures longest common subsequence for structural similarity
- ROUGE-S measures skip-bigram overlap allowing gaps between matches
- Typically reports precision, recall, and F1 for each variant
- Standard benchmark metric in academic summarization research
- Supports comparison against multiple reference summaries
- Measures lexical overlap rather than true semantic equivalence