LLMScalelive Hugging Face data
ENTR

Qwen3 8B: how much VRAM does it need?

Qwen3 8B has 8.19B parameters in 36 layers. It uses grouped-query attention: 32 query heads share 8 key-value heads, so its KV cache is 4× smaller than full multi-head attention would need. It takes up to 40,960 tokens of context; one user at that length needs 5.63 GiB of KV cache at FP16 on top of the weights.

BF16 · 8k context · one user
18.37 GiB

15.26 GiB of weights, 1.13 GiB of KV cache and 1.99 GiB of vLLM's overhead. At INT4 the same workload needs 8.87 GiB.

  • 36 layers
  • hidden 4096
  • 32/8 heads
  • head_dim 128
  • 40k max context

Qwen3 8B memory by precision and context

Precision8k context32k context40k context
BF1618.37 GiBKV cache 1.13 GiB21.74 GiBKV cache 4.50 GiB22.87 GiBKV cache 5.63 GiB
FP811.90 GiBKV cache 1.13 GiB15.27 GiBKV cache 4.50 GiB16.40 GiBKV cache 5.63 GiB
INT48.87 GiBKV cache 1.13 GiB12.24 GiBKV cache 4.50 GiB13.37 GiBKV cache 5.63 GiB

One user, FP16 KV cache and vLLM's overhead on one GPU at its defaults for a card under 70 GB. Quantizing shrinks the weights, not the KV cache, so the saving narrows as the context grows.

Which single GPUs hold it

At 8k context for one user, the smallest cards first. A discrete card is budgeted at vLLM's default 92% of its memory, a unified-memory device at the share its GPU can address, and each carries vLLM's overhead at its own defaults.

How many users fit

GPUBF16FP8INT4
RTX 40904912
RTX 5090101618
RTX 6000 Ada232931
H100 SXM 80GB495457
RTX PRO 6000 Blackwell616770
AMD Ryzen AI Max+ 395 (128GB)626871
H200 141GB98104107
B200 180GB129135137

Users at 8k context each, on one card. ✗: the weights leave no room for even one.

Fine-tuning Qwen3 8B

MethodMemorySmallest GPU
Full fine-tune123.6 GiBH200 141GB
LoRA (rank 16, attention)17.02 GiBNVIDIA L4
QLoRA (rank 16, attention)5.70 GiBRTX 3060

Sequence length 2,048, batch size 1, gradient checkpointing on, Adam. Full fine-tuning keeps 16 bytes per parameter before activations; QLoRA stores the frozen base in 4-bit NF4.

Where the numbers come from

Weights are parameters × bytes per parameter. The KV cache is 2 × layers × key-value heads × head dimension × bytes × context × users. For Qwen3 8B that is 144.0 KiB per token at FP16. These are capacity-planning figures: the serving engine — vLLM, TGI, llama.cpp — and its paged-attention efficiency still move the real number.

More on KV cache size and on GQA, MQA and MLA.

Work with Qwen3 8B

Other Qwen models