Mistral Nemo 12B Instruct: how much VRAM does it need?

Mistral Nemo 12B Instruct has 12.25B parameters in 40 layers. It uses grouped-query attention: 32 query heads share 8 key-value heads, so its KV cache is 4× smaller than full multi-head attention would need. It takes up to 131,072 tokens of context; one user at that length needs 20.00 GiB of KV cache at FP16 on top of the weights.

BF16 · 8k context · one user

26.23 GiB

22.82 GiB of weights, 1.25 GiB of KV cache and 2.17 GiB of vLLM's overhead. At INT4 the same workload needs 11.31 GiB.

40 layers
hidden 5120
32/8 heads
head_dim 128
128k max context

Mistral Nemo 12B Instruct memory by precision and context

Precision	8k context	32k context	128k context
BF16	26.23 GiBKV cache 1.25 GiB	29.98 GiBKV cache 5.00 GiB	44.98 GiBKV cache 20.00 GiB
FP8	16.07 GiBKV cache 1.25 GiB	19.82 GiBKV cache 5.00 GiB	34.82 GiBKV cache 20.00 GiB
INT4	11.31 GiBKV cache 1.25 GiB	15.06 GiBKV cache 5.00 GiB	30.06 GiBKV cache 20.00 GiB

One user, FP16 KV cache and vLLM's overhead on one GPU at its defaults for a card under 70 GB. Quantizing shrinks the weights, not the KV cache, so the saving narrows as the context grows.

Which single GPUs hold it

BF16 (26.23 GiB): RTX 5090, Apple M4 Pro, A100 40GB, NVIDIA L40S
FP8 (16.07 GiB): NVIDIA L4, Apple M4, NVIDIA A30, RTX 3090 / Ti
INT4 (11.31 GiB): NVIDIA T4, RTX 4060 Ti 16GB, RTX 4070 Ti Super, RTX 4080 / Super

At 8k context for one user, the smallest cards first. A discrete card is budgeted at vLLM's default 92% of its memory, a unified-memory device at the share its GPU can address, and each carries vLLM's overhead at its own defaults.

How many users fit

GPU	BF16	FP8	INT4
RTX 4090	does not fit	5	9
RTX 5090	3	11	15
RTX 6000 Ada	14	23	26
H100 SXM 80GB	37	46	49
RTX PRO 6000 Blackwell	49	57	61
AMD Ryzen AI Max+ 395 (128GB)	50	58	62
H200 141GB	82	90	94
B200 180GB	110	118	122

Users at 8k context each, on one card. ✗: the weights leave no room for even one.

Fine-tuning Mistral Nemo 12B Instruct

Method	Memory	Smallest GPU
Full fine-tune	184.3 GiB	Apple M3 Ultra (256GB)
LoRA (rank 16, attention)	24.87 GiB	RTX 5090
QLoRA (rank 16, attention)	7.94 GiB	RTX 3060

Sequence length 2,048, batch size 1, gradient checkpointing on, Adam. Full fine-tuning keeps 16 bytes per parameter before activations; QLoRA stores the frozen base in 4-bit NF4.

Where the numbers come from

Weights are parameters × bytes per parameter. The KV cache is 2 × layers × key-value heads × head dimension × bytes × context × users. For Mistral Nemo 12B Instruct that is 160.0 KiB per token at FP16. These are capacity-planning figures: the serving engine — vLLM, TGI, llama.cpp — and its paged-attention efficiency still move the real number.

More on KV cache size and on GQA, MQA and MLA.