Mixtral 8x7B Instruct: how much VRAM does it need?

Mixtral 8x7B Instruct has 46.70B parameters in 32 layers. It uses grouped-query attention: 32 query heads share 8 key-value heads, so its KV cache is 4× smaller than full multi-head attention would need. It is a Mixture-of-Experts model with about 12.90B of its 46.70B parameters active per token: memory has to hold all of them, while each token passes through only the active share. It takes up to 32,768 tokens of context; one user at that length needs 4.00 GiB of KV cache at FP16 on top of the weights.

BF16 · 8k context · one user

89.46 GiB

86.99 GiB of weights, 1.00 GiB of KV cache and 1.48 GiB of vLLM's overhead. At INT4 the same workload needs 25.94 GiB.

32 layers
hidden 4096
32/8 heads
head_dim 128
32k max context
MoE

Mixtral 8x7B Instruct memory by precision and context

Precision	8k context	32k context
BF16	89.46 GiBKV cache 1.00 GiB	92.46 GiBKV cache 4.00 GiB
FP8	46.21 GiBKV cache 1.00 GiB	49.21 GiBKV cache 4.00 GiB
INT4	25.94 GiBKV cache 1.00 GiB	28.94 GiBKV cache 4.00 GiB

One user, FP16 KV cache and vLLM's overhead on one GPU at its defaults for a card under 70 GB. Quantizing shrinks the weights, not the KV cache, so the saving narrows as the context grows.

Which single GPUs hold it

BF16 (89.46 GiB): Apple M4 Max (128GB), NVIDIA DGX Spark (GB10), H200 141GB, Apple M2 Ultra
FP8 (46.21 GiB): Apple M4 Max (64GB), NVIDIA Jetson AGX Orin (64GB), AMD MI210, RTX PRO 5000 Blackwell (72GB)
INT4 (25.94 GiB): RTX 5090, Apple M4 Pro, A100 40GB, NVIDIA L40S

At 8k context for one user, the smallest cards first. A discrete card is budgeted at vLLM's default 92% of its memory, a unified-memory device at the share its GPU can address, and each carries vLLM's overhead at its own defaults.

How many users fit

GPU	BF16	FP8	INT4
RTX 4090	does not fit	does not fit	does not fit
RTX 5090	does not fit	does not fit	3
RTX 6000 Ada	does not fit	does not fit	18
H100 SXM 80GB	does not fit	25	45
RTX PRO 6000 Blackwell	does not fit	39	59
AMD Ryzen AI Max+ 395 (128GB)	does not fit	40	60
H200 141GB	37	80	101
B200 180GB	69	112	133

Users at 8k context each, on one card. ✗: the weights leave no room for even one.

Fine-tuning Mixtral 8x7B Instruct

Method	Memory	Smallest GPU
Full fine-tune	697.4 GiB	several GPUs, sharded
LoRA (rank 16, attention)	88.66 GiB	GH200 96GB
QLoRA (rank 16, attention)	24.12 GiB	RTX 5090

Sequence length 2,048, batch size 1, gradient checkpointing on, Adam. Full fine-tuning keeps 16 bytes per parameter before activations; QLoRA stores the frozen base in 4-bit NF4.

Where the numbers come from

Weights are parameters × bytes per parameter. The KV cache is 2 × layers × key-value heads × head dimension × bytes × context × users. For Mixtral 8x7B Instruct that is 128.0 KiB per token at FP16. These are capacity-planning figures: the serving engine — vLLM, TGI, llama.cpp — and its paged-attention efficiency still move the real number.

More on KV cache size and on GQA, MQA and MLA.