LLMScalelive Hugging Face data
ENTR

gpt-oss 20B: how much VRAM does it need?

gpt-oss 20B has 20.91B parameters in 24 layers. It uses grouped-query attention: 64 query heads share 8 key-value heads, so its KV cache is 8× smaller than full multi-head attention would need. It is a Mixture-of-Experts model with about 3.60B of its 20.91B parameters active per token: memory has to hold all of them, while each token passes through only the active share. It takes up to 131,072 tokens of context; one user at that length needs 3.10 GiB of KV cache at FP16 on top of the weights.

BF16 · 8k context · one user
41.22 GiB

38.96 GiB of weights, 291 MiB of KV cache and 1.98 GiB of vLLM's overhead. At INT4 the same workload needs 15.08 GiB.

  • 24 layers
  • hidden 2880
  • 64/8 heads
  • head_dim 64
  • 128k max context
  • MoE

gpt-oss 20B memory by precision and context

Precision8k context32k context128k context
BF1641.22 GiBKV cache 291 MiB41.78 GiBKV cache 867 MiB44.03 GiBKV cache 3.10 GiB
FP822.82 GiBKV cache 291 MiB23.38 GiBKV cache 867 MiB25.63 GiBKV cache 3.10 GiB
INT415.08 GiBKV cache 291 MiB15.64 GiBKV cache 867 MiB17.89 GiBKV cache 3.10 GiB

One user, FP16 KV cache and vLLM's overhead on one GPU at its defaults for a card under 70 GB. Quantizing shrinks the weights, not the KV cache, so the saving narrows as the context grows.

Which single GPUs hold it

At 8k context for one user, the smallest cards first. A discrete card is budgeted at vLLM's default 92% of its memory, a unified-memory device at the share its GPU can address, and each carries vLLM's overhead at its own defaults.

How many users fit

GPUBF16FP8INT4
RTX 4090does not fitdoes not fit21
RTX 5090does not fit2246
RTX 6000 Ada97498
H100 SXM 80GB82131151
RTX PRO 6000 Blackwell120169188
AMD Ryzen AI Max+ 395 (128GB)123172189
H200 141GB230279300
B200 180GB321370388

Users at 8k context each, on one card. ✗: the weights leave no room for even one.

Fine-tuning gpt-oss 20B

MethodMemorySmallest GPU
Full fine-tune312.9 GiBApple M3 Ultra (512GB)
LoRA (rank 16, attention)40.31 GiBNVIDIA L40S
QLoRA (rank 16, attention)11.41 GiBNVIDIA T4

Sequence length 2,048, batch size 1, gradient checkpointing on, Adam. Full fine-tuning keeps 16 bytes per parameter before activations; QLoRA stores the frozen base in 4-bit NF4.

Where the numbers come from

Weights are parameters × bytes per parameter. The KV cache is 2 × layers × key-value heads × head dimension × bytes × context × users. For gpt-oss 20B that is 48.0 KiB per token at FP16. 12 of its 24 layers attend over a sliding window of the last 128 tokens and hold no more than that, so past it each further token adds 24.0 KiB, the other 12 layers' share. These are capacity-planning figures: the serving engine — vLLM, TGI, llama.cpp — and its paged-attention efficiency still move the real number.

More on KV cache size and on GQA, MQA and MLA.

Work with gpt-oss 20B

Other OpenAI models