KV cache size: the formula, with real models

When a language model generates text, it keeps the keys and values it computed for every earlier token, in every layer, so it does not have to compute them again for the next one. That store is the KV cache. It is the part of GPU memory rules of thumb leave out, and at long context or with many users it outgrows the model's weights.

The formula

For each token, every layer stores one key vector and one value vector per key-value head:

KV bytes per token = 2 × layers × key-value heads × head dimension × bytes per value
KV cache           = KV bytes per token × context length × concurrent sequences

The 2 counts the key and the value. A value takes 2 bytes at FP16 or BF16 and 1 byte at FP8. Layers, key-value heads and head dimension are fixed by the architecture — num_hidden_layers, num_key_value_heads and head_dim in the model's config.json — which is why two models of the same size can need very different caches. Models with multi-head latent attention replace the middle of the formula; see GQA, MQA and MLA.

A worked example: Llama 3.1 8B Instruct

Llama 3.1 8B Instruct has 32 layers, 8 key-value heads and a head dimension of 128. At FP16 that is 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128.0 KiB, per token. One user at 8k tokens needs 1.00 GiB of cache; at its full 131,072-token context, 16.00 GiB — more than its 14.96 GiB of BF16 weights.

Parameter count is not what decides it. Qwen2.5 7B Instruct and Llama 3.1 8B Instruct are close in size — 7.62B and 8.03B parameters — but Llama 3.1 8B Instruct stores 128.0 KiB per token and Qwen2.5 7B Instruct 56.0 KiB, a 2.3× difference that comes from layers and key-value heads.

Per-token cache in popular models

FP16, one sequence. —: beyond the model's maximum context. DeepSeek V3 caches a compressed latent rather than per-head keys and values. Gemma 2 and gpt-oss are left out of this table: their sliding-window layers stop growing at the window, so their cost per token falls as the context grows. The calculator counts that layer by layer.

Concurrency multiplies it

Every concurrent sequence holds a cache of its own. On one H100 SXM 80GB, with 72.82 GiB usable, Llama 3.1 8B Instruct at BF16 leaves room for 55 users at 8k tokens each, 13 at 32k and 3 at 128k. The weights set the floor; the cache decides how many people fit above it.

What shrinks it, and what does not

• Fewer key-value heads. Grouped-query and multi-query attention share key-value heads across query heads, and multi-head latent attention caches a compressed latent. This comes with the model you choose.
• An FP8 KV cache. One byte per value instead of two halves the cache. vLLM offers it as --kv-cache-dtype fp8; it can cost accuracy on long-context work, so validate it first.
• A lower context limit. The cache grows with every token a sequence holds, so capping sequence length — --max-model-len in vLLM — caps the worst case.
• Not weight quantization. INT4 or FP8 weights shrink the weights only; the cache keeps its own precision, so the saving narrows as context grows. See quantization and VRAM.
• Paged attention and prefix caching use the cache better rather than making it smaller: memory is handed out in blocks as sequences grow instead of being reserved up front, and requests with the same prefix can share its blocks. LLMScale does not model that efficiency, so its figures are the full need.

• Size the KV cache for your model
• GQA, MQA and MLA
• Quantization and VRAM
• VRAM by model