Llama 3.1 405B Instruct: how much VRAM does it need?

Llama 3.1 405B Instruct has 405.85B parameters in 126 layers. It uses grouped-query attention: 128 query heads share 8 key-value heads, so its KV cache is 16× smaller than full multi-head attention would need. It takes up to 131,072 tokens of context; one user at that length needs 63.00 GiB of KV cache at FP16 on top of the weights. Its repository is gated on Hugging Face, so LLMScale ships its architecture in a bundled database.

Llama 3.1 405B Instruct memory by precision and context

One user, FP16 KV cache and vLLM's overhead on one GPU at its defaults for a card under 70 GB. Quantizing shrinks the weights, not the KV cache, so the saving narrows as the context grows.

Which single GPUs hold it

• BF16 (768.3 GiB): no single GPU holds it — the fewest that serve it, as the calculator lays them out: 12× H100 SXM 80GB, 7× H200 141GB, 6× B200 180GB.
• FP8 (394.2 GiB): no single GPU holds it — the fewest that serve it, as the calculator lays them out: 6× H100 SXM 80GB, 4× H200 141GB, 3× B200 180GB.
• INT4 (218.9 GiB): AMD MI325X, B300 (Blackwell Ultra) 270GB, AMD MI355X, Apple M3 Ultra (512GB)

At 8k context for one user, the smallest cards first. A discrete card is budgeted at vLLM's default 92% of its memory, a unified-memory device at the share its GPU can address, and each carries vLLM's overhead at its own defaults.

How many users fit

No single card here holds it at any precision; serving it means splitting it across several GPUs with tensor parallelism.

Fine-tuning Llama 3.1 405B Instruct

Sequence length 2,048, batch size 1, gradient checkpointing on, Adam. Full fine-tuning keeps 16 bytes per parameter before activations; QLoRA stores the frozen base in 4-bit NF4.

Where the numbers come from

Weights are parameters × bytes per parameter. The KV cache is 2 × layers × key-value heads × head dimension × bytes × context × users. For Llama 3.1 405B Instruct that is 504.0 KiB per token at FP16. These are capacity-planning figures: the serving engine — vLLM, TGI, llama.cpp — and its paged-attention efficiency still move the real number.

More on KV cache size and on GQA, MQA and MLA.

Work with Llama 3.1 405B Instruct

• Size it for your workload
• Fine-tuning memory
• vLLM parameters
• Model anatomy
• Its config.json, explained
• What its files say about serving it

Other Llama models

• Llama 3.2 1B Instruct
• Llama 3.2 3B Instruct
• Llama 3.1 8B Instruct
• Llama 3.3 70B Instruct
• Llama 3.1 70B Instruct
• Llama 4 Scout 17B-16E Instruct