LLM VRAM calculator — model weights, KV cache and GPU fit
How much GPU memory does a model really need? Weights and KV cache scale with your context window and concurrent users. Pick a model, tune the workload, and see which GPUs fit.
Choose a model
Precision & workload
When off, every user holds the full context window: the worst case vLLM reports as “Maximum concurrency”. When on, a request holds its prompt and half its reply on average, and a prefix shared by every prompt is cached once.
Pick a model to see the VRAM breakdown.
Weights, KV cache and overhead are computed in your browser from the model's own config — search Hugging Face above, or start from a preset.
How the numbers are calculated ▸▾
Weights = parameters × bytes/param (FP16/BF16 = 2, FP8/INT8 = 1, INT4 = 0.5). For Mixture-of-Experts models, all experts are loaded, so total params are used — not the active subset.
KV cache = 2 × layers × kv_heads × head_dim × bytes × context × concurrency. Grouped-Query Attention (kv_heads < attn_heads) shrinks this dramatically — it's read from each model's config. Multi-head latent attention (DeepSeek) caches layers × (kv_lora_rank + qk_rope_head_dim) × bytes per token instead. It is sized as vLLM reserves it: a sliding-window layer keeps its window plus every token in flight (two batches of --max-num-batched-tokens), and a Mamba or linear-attention layer keeps two state blocks.
Overhead is what vLLM 0.31 keeps on each GPU besides the weights and the cache: runtime buffers (4 bytes per vocabulary entry for every sequence it may batch), the peak of its profiling pass (the larger of the feed-forward block over the batched tokens and the output layer) and its CUDA graphs. With tensor parallelism every GPU keeps its own. Checked against eleven vLLM startups, it lands within 0.13 GiB of vLLM's own figures. Each card is budgeted at vLLM's --gpu-memory-utilization, 0.92 by default.
Estimates are for planning, not a guarantee. Real usage varies by serving engine (vLLM, TGI, llama.cpp) and its paged-attention efficiency.
What this calculator assumes
LLMScale reads each model's architecture from its config.json on the Hugging Face Hub — layers, attention and key-value heads, head size — and sizes GPU memory from that, not from the parameter count alone. Gated models such as Llama, Gemma and Mistral, whose config a browser cannot read, come from a bundled database; a repository without a config of its own, such as a GGUF fork, is resolved through its base_model.
Every card is budgeted at vLLM's default share, 92% of its memory; the rest holds the CUDA context. On top of the weights and the KV cache, each GPU carries vLLM's own overhead: runtime buffers, the peak of its profiling pass and its CUDA graphs, measured against real vLLM startups. Unified-memory devices use a realistic share instead of the whole pool: 75% of an Apple Mac's memory (two-thirds on one under 36 GB); on Strix Halo and DGX Spark, vLLM's 92% of what the GPU reports — 88 of the 96 GB Strix Halo gives its GPU, 112 of DGX Spark's 121.6 GB (where vLLM is commonly run at 80%). Each card's budget starts from the memory its runtime reports, which is a little under its nominal size, and a sixteenth less on GDDR datacenter cards with ECC on (L40S, L4, T4). Decode speed is determined by the slower of the limits imposed by memory bandwidth, taken at 80% of the vendor's peak, and dense tensor throughput. Time to first token is calculated from the prompt's FLOPs for one request without queueing. Tensor throughput is taken at 90% of the rated peak on GeForce cards, based on measurements with vLLM on an RTX 4070 Ti (the test card ran above the clock used for its rated figure), and at 60% on other cards, as an estimate. The figures are for capacity planning — the serving engine and its paged-attention efficiency still move the real number.
Questions
- How much VRAM does a 70B model need?
- For Llama 3.3 70B the weights alone take about 131 GiB at BF16, 68 GiB at FP8 and 38 GiB at INT4: 70.6 billion parameters at 2 bytes, then 1 byte or about 4.25 bits for the transformer layers, with the embedding and the output layer kept at 16 bits as quantized checkpoints keep them. The KV cache and overhead come on top, and both depend on the context window and the number of users — which is what the calculator works out.
- Why does the context window change the answer so much?
- The KV cache grows linearly with context length and with concurrent users. Llama 3.1 8B stores 131,072 bytes per token at FP16: 1 GiB for one user at 8k tokens, 16 GiB at 128k — and 16 GiB again at 8k with 16 users.
- Does a Mixture-of-Experts model need less memory?
- Not for memory. Every expert has to be loaded, so memory is sized from the total parameter count. Only some of the experts run for each token, which is why MoE models generate faster than their size suggests.
- How is DeepSeek's KV cache handled?
- DeepSeek-V2 and V3 use multi-head latent attention (MLA), which caches one compressed latent per layer instead of keys and values for every head. The calculator reads kv_lora_rank and qk_rope_head_dim from the config and uses layers × (kv_lora_rank + qk_rope_head_dim) × bytes per token. For DeepSeek-V3 at 8k context that is 0.54 GiB, where the usual formula would give 13.34 GiB.