GQA, MQA and MLA: how attention decides the KV cache

Attention is where the KV cache comes from, and the way a model arranges its attention heads can change the cache by an order of magnitude at the same parameter count. Four arrangements cover today's open models.

Multi-head attention (MHA)

Every query head has key and value heads of its own, and the cache stores all of them: 2 × layers × heads × head dimension × bytes per token. It is the original Transformer design and the most expensive to cache. Phi-3 mini 4k still uses it — 32 heads over 32 layers, 384.0 KiB per token.

Multi-query attention (MQA)

All query heads share a single key head and a single value head (Shazeer, 2019, “Fast Transformer Decoding: One Write-Head is All You Need”). The cache shrinks by the full number of heads — the largest saving there is — but sharing one set of keys and values across every head can cost quality.

Grouped-query attention (GQA)

Query heads are split into groups, and each group shares one key-value head: more than one, fewer than the number of query heads. Ainslie et al. (2023) proposed it as the middle ground — quality close to multi-head attention, speed close to multi-query — and 28 of the 31 models LLMScale bundles use it. Llama 3.3 70B Instruct groups 64 query heads onto 8 key-value heads, so it caches 320.0 KiB per token where multi-head attention would need 2560.0 KiB: 8× less.

Multi-head latent attention (MLA)

DeepSeek-V2 took another route. Instead of keys and values per head, each layer caches one compressed latent vector per token, plus a small decoupled key that carries the rotary position encoding. The cache per token becomes layers × (kv_lora_rank + qk_rope_head_dim) × bytes, and the head count drops out. For its own dimensions, the DeepSeek-V2 paper (§2.1.3) puts that at the size of GQA with only 2.25 groups.

DeepSeek V3 caches 512 + 64 = 576 values per layer across 61 layers: 68.6 KiB per token, 24.9× less than the per-head formula gives. At 8k context that is 549 MiB instead of 13.34 GiB. A calculator that applies the per-head formula to DeepSeek models overstates the cache by that factor; LLMScale switches formulas whenever a config carries both fields.

Side by side

FP16. “vs MHA”: how many times more a multi-head version of the same model would cache.

What it means when you choose a model

• Check the key-value heads before the parameter count. Qwen2.5 7B Instruct and Llama 3.1 8B Instruct differ by 5% in parameters and by 2.3× in cache per token.
• The attention layout comes with the model. To shrink a served model's cache you have an FP8 KV cache, a shorter context or fewer sequences — see KV cache size.
• Tensor parallelism splits key-value heads across GPUs, so the calculator suggests GPU counts that divide them evenly.

• See a model's attention layout
• KV cache size
• Quantization and VRAM