LLMScalelive Hugging Face data
ENTR

Mixtral 8x22B Instruct: how much VRAM does it need?

Mixtral 8x22B Instruct has 140.60B parameters in 56 layers. It uses grouped-query attention: 48 query heads share 8 key-value heads, so its KV cache is 6× smaller than full multi-head attention would need. It is a Mixture-of-Experts model with about 39.00B of its 140.60B parameters active per token: memory has to hold all of them, while each token passes through only the active share. It takes up to 65,536 tokens of context; one user at that length needs 14.00 GiB of KV cache at FP16 on top of the weights.

BF16 · 8k context · one user
265.6 GiB

261.9 GiB of weights, 1.75 GiB of KV cache and 2.01 GiB of vLLM's overhead. At INT4 the same workload needs 73.87 GiB.

  • 56 layers
  • hidden 6144
  • 48/8 heads
  • head_dim 128
  • 64k max context
  • MoE

Mixtral 8x22B Instruct memory by precision and context

Precision8k context32k context64k context
BF16265.6 GiBKV cache 1.75 GiB270.9 GiBKV cache 7.00 GiB277.9 GiBKV cache 14.00 GiB
FP8135.1 GiBKV cache 1.75 GiB140.3 GiBKV cache 7.00 GiB147.3 GiBKV cache 14.00 GiB
INT473.87 GiBKV cache 1.75 GiB79.12 GiBKV cache 7.00 GiB86.12 GiBKV cache 14.00 GiB

One user, FP16 KV cache and vLLM's overhead on one GPU at its defaults for a card under 70 GB. Quantizing shrinks the weights, not the KV cache, so the saving narrows as the context grows.

Which single GPUs hold it

At 8k context for one user, the smallest cards first. A discrete card is budgeted at vLLM's default 92% of its memory, a unified-memory device at the share its GPU can address, and each carries vLLM's overhead at its own defaults.

How many users fit

GPUBF16FP8INT4
RTX 4090does not fitdoes not fitdoes not fit
RTX 5090does not fitdoes not fitdoes not fit
RTX 6000 Adadoes not fitdoes not fitdoes not fit
H100 SXM 80GBdoes not fitdoes not fitdoes not fit
RTX PRO 6000 Blackwelldoes not fitdoes not fit7
AMD Ryzen AI Max+ 395 (128GB)does not fitdoes not fit7
H200 141GBdoes not fitdoes not fit30
B200 180GBdoes not fit1348

Users at 8k context each, on one card. ✗: the weights leave no room for even one.

Fine-tuning Mixtral 8x22B Instruct

MethodMemorySmallest GPU
Full fine-tune2097.4 GiBseveral GPUs, sharded
LoRA (rank 16, attention)264.7 GiBAMD MI355X
QLoRA (rank 16, attention)70.33 GiBA100 80GB

Sequence length 2,048, batch size 1, gradient checkpointing on, Adam. Full fine-tuning keeps 16 bytes per parameter before activations; QLoRA stores the frozen base in 4-bit NF4.

Where the numbers come from

Weights are parameters × bytes per parameter. The KV cache is 2 × layers × key-value heads × head dimension × bytes × context × users. For Mixtral 8x22B Instruct that is 224.0 KiB per token at FP16. These are capacity-planning figures: the serving engine — vLLM, TGI, llama.cpp — and its paged-attention efficiency still move the real number.

More on KV cache size and on GQA, MQA and MLA.

Work with Mixtral 8x22B Instruct

Other Mistral models