LLMScaleWhich LLMs fit on A100 80GB?
A100 80GB has 80 GB of memory. LLMScale budgets 72.82 GiB of it for a model: vLLM's default --gpu-memory-utilization of 0.92 of what the card reports, the rest kept for the CUDA context. Its memory bandwidth, 2,039 GB/s, is what limits generation speed once a model fits. Of the 31 popular open models below, 22 fit at BF16, 23 at FP8 and 28 at INT4 with an 8k context for one user.
28 of 31 models fit at INT4, 22 at BF16.
Which LLMs fit on A100 80GB
| Model | BF16 | FP8 | INT4 | Users | Max context |
|---|---|---|---|---|---|
| Qwen2.5 0.5B Instruct | 1.85 GiB | 1.52 GiB | 1.36 GiB | 757BF16 | 32kBF16 |
| Gemma 3 1B IT | 3.27 GiB | 2.62 GiB | 2.31 GiB | 508BF16 | 32kBF16 |
| Llama 3.2 1B Instruct | 3.63 GiB | 2.72 GiB | 2.29 GiB | 277BF16 | 128kBF16 |
| Llama 3.2 3B Instruct | 8.28 GiB | 5.65 GiB | 4.42 GiB | 74BF16 | 128kBF16 |
| Phi-3 mini 4k | 9.42 GiB | 6.04 GiB | 4.46 GiB | 43BF16 | 4kBF16 |
| Gemma 3 4B IT | 10.57 GiB | 7.19 GiB | 5.61 GiB | 84BF16 | 128kBF16 |
| Mistral 7B Instruct v0.3 | 15.43 GiB | 8.93 GiB | 5.88 GiB | 58BF16 | 32kBF16 |
| Qwen2.5 Coder 7B | 16.37 GiB | 10.30 GiB | 7.45 GiB | 130BF16 | 32kBF16 |
| Qwen2.5 7B Instruct | 16.37 GiB | 10.30 GiB | 7.45 GiB | 130BF16 | 32kBF16 |
| Llama 3.1 8B Instruct | 17.71 GiB | 11.21 GiB | 8.16 GiB | 56BF16 | 128kBF16 |
| Qwen3 8B | 18.37 GiB | 11.90 GiB | 8.87 GiB | 49BF16 | 40kBF16 |
| Gemma 2 9B IT | 22.48 GiB | 14.73 GiB | 11.09 GiB | 20BF16 | 8kBF16 |
| Gemma 3 12B IT | 26.85 GiB | 16.44 GiB | 11.55 GiB | 23BF16 | 128kBF16 |
| Mistral Nemo 12B Instruct | 26.23 GiB | 16.07 GiB | 11.31 GiB | 38BF16 | 128kBF16 |
| Qwen2.5 14B Instruct | 31.48 GiB | 19.17 GiB | 13.40 GiB | 28BF16 | 32kBF16 |
| gpt-oss 20B | 41.22 GiB | 22.82 GiB | 15.96 GiB | 112BF16 | 128kBF16 |
| Gemma 2 27B IT | 56.79 GiB | 32.55 GiB | 21.19 GiB | 6BF16 | 8kBF16 |
| Gemma 3 27B IT | 56.53 GiB | 32.29 GiB | 20.93 GiB | 6BF16 | 128kBF16 |
| Qwen3 30B A3B | 59.07 GiB | 31.21 GiB | 18.15 GiB | 19BF16 | 40kBF16 |
| Qwen3 Coder 30B A3B | 59.07 GiB | 31.21 GiB | 18.15 GiB | 19BF16 | 155kBF16 |
| Qwen2.5 32B Instruct | 65.64 GiB | 36.58 GiB | 22.96 GiB | 4BF16 | 32kBF16 |
| Qwen3 32B | 65.64 GiB | 36.58 GiB | 22.96 GiB | 4BF16 | 37kBF16 |
| Mixtral 8x7B Instruct | 89.46 GiB (does not fit) | 46.21 GiB | 25.94 GiB | 27FP8 | 32kFP8 |
| Llama 3.3 70B Instruct | 137.6 GiB (does not fit) | 73.89 GiB (does not fit) | 44.01 GiB | 12INT4 | 100kINT4 |
| Llama 3.1 70B Instruct | 137.6 GiB (does not fit) | 73.89 GiB (does not fit) | 44.01 GiB | 12INT4 | 100kINT4 |
| Qwen2.5 72B Instruct | 142.0 GiB (does not fit) | 76.64 GiB (does not fit) | 45.99 GiB | 11INT4 | 32kINT4 |
| Llama 4 Scout 17B-16E Instruct | 205.4 GiB (does not fit) | 106.2 GiB (does not fit) | 59.64 GiB | 9INT4 | 78kINT4 |
| gpt-oss 120B | 220.1 GiB (does not fit) | 112.4 GiB (does not fit) | 68.40 GiB | 11INT4 | 128kINT4 |
| Mixtral 8x22B Instruct | 265.6 GiB (does not fit) | 135.1 GiB (does not fit) | 73.87 GiB (does not fit) | — | — |
| Llama 3.1 405B Instruct | 768.3 GiB (does not fit) | 394.2 GiB (does not fit) | 218.9 GiB (does not fit) | — | — |
| DeepSeek V3 | 1279.1 GiB (does not fit) | 643.3 GiB (does not fit) | 345.2 GiB (does not fit) | — | — |
Memory at 8k context for one user (the model's maximum, where that is shorter), FP16 KV cache and vLLM's overhead at this card's defaults. Users (each at 8k) and maximum context are for the highest precision that fits.
What BF16, FP8 and INT4 mean for memory: quantization and VRAM.
A model that does not fit one card can still run across several: tensor parallelism splits its weights and KV cache between them. The calculator works out how many.