How is GQA different from Multi-Query Attention?

MQA takes the extreme of a single shared key/value head for all query heads, maximizing memory savings but sometimes hurting quality; GQA groups query heads into several clusters each with their own key/value head, trading a bit less memory savings for better retained quality.

Why does reducing KV heads matter so much for serving cost?

The KV cache size scales directly with the number of key/value heads, and KV cache memory is often the binding constraint on how many requests a GPU can serve concurrently — fewer KV heads means more headroom for batch size and context length.

Can a model trained with standard multi-head attention be converted to GQA?

Yes — the original GQA paper showed that mean-pooling the key/value heads within each group and then fine-tuning for a small fraction of the original training compute recovers most of the quality of a model trained with GQA from scratch.

Does GQA reduce model quality compared to full multi-head attention?

Generally only slightly — benchmarks in the GQA paper and subsequent model releases show quality close to full MHA while achieving inference speed and memory characteristics much closer to MQA.

Which well-known models use GQA?

LLaMA 2 70B, LLaMA 3 (all sizes), Mistral 7B, and many subsequent open-weight LLMs adopted GQA as a standard efficiency optimization.

AI Fundamentals

Grouped Query Attention

AdvancedTechnique8.8K learners

Grouped Query Attention (GQA) is a transformer attention variant in which multiple query heads share a single set of key and value heads, reducing the size of the KV cache and inference memory bandwidth compared to standard multi-head…

#GroupedQueryAttention #AIFundamentals#Technique#Advanced#KVCache #FlashAttention #SparseAttention #RotaryPositionalEmbedding #ArtificialIntelligence#Glossary#SkillVeris

Definition

Overview

Standard multi-head attention (MHA) gives every attention head its own independent query, key, and value projections. This maximizes representational flexibility but means the KV cache — the stored keys and values needed for autoregressive generation — grows proportionally to the number of heads, which becomes a major memory and bandwidth bottleneck at inference time, especially for large models serving long contexts and large batch sizes. Multi-Query Attention (MQA), proposed by Noam Shazeer in 2019, took the opposite extreme: all query heads share a single shared key/value head, drastically shrinking the KV cache but sometimes at a noticeable cost to model quality and training stability. Grouped Query Attention, introduced by Google Research in 2023 (the GQA paper by Ainslie et al.), is a middle ground: query heads are divided into groups, and each group shares one set of key/value heads. With, say, 32 query heads divided into 8 groups, GQA uses only 8 key/value heads instead of 32, cutting the KV cache by 4x, while retaining more per-group specialization than full MQA. The paper also showed that existing MHA-trained models can be converted to GQA cheaply, by mean-pooling existing key/value heads within each group and then fine-tuning briefly, avoiding a full retrain from scratch. GQA has become a standard architectural choice in modern open-weight LLMs because it directly attacks the KV-cache memory bottleneck described in the KV cache entry: smaller KV cache means more concurrent requests can be served per GPU, longer contexts fit in memory, and inference throughput improves, all with only a small, often negligible, quality trade-off compared to full multi-head attention. Models including LLaMA 2 70B, LLaMA 3, Mistral, and many others use GQA, typically tuned to a specific query-to-KV-head ratio (e.g. 4:1 or 8:1) chosen to balance quality against serving efficiency for that model's size and target hardware.

Key Concepts

Multiple query heads share a smaller number of key/value heads
Sits between full Multi-Head Attention and Multi-Query Attention on the quality/efficiency spectrum
Directly reduces KV cache memory footprint proportional to the group size
Existing MHA models can be 'uptrained' into GQA via head pooling plus light fine-tuning
Improves inference throughput and maximum batch size on fixed GPU memory
Standard in LLaMA 2/3, Mistral, and many other modern open-weight LLMs
Query-to-KV-head ratio is a tunable architectural hyperparameter (e.g. 4:1, 8:1)

Use Cases

Reducing inference memory and cost for large language model serving

Enabling longer context windows within fixed GPU memory budgets

Increasing maximum concurrent request batch size in production LLM APIs

Converting existing multi-head-attention checkpoints to a more efficient architecture without full retraining

Frequently Asked Questions

From the Blog

AI & Technology

How Large Language Models Actually Work

LLMs seem magical until you understand what they are: next-token predictors trained on massive text corpora. This guide explains tokenisation, embeddings, the transformer architecture, attention mechanism, and how training works — without requiring a maths degree.

Learn SQL Through Football Data

Football generates rich match data — goals, assists, passes, xG, red cards. This project uses a Premier League dataset to teach SQL SELECT, WHERE, GROUP BY, JOIN, and HAVING in a context that makes every query meaningful rather than abstract.