LLMScaleWhich LLMs fit on RTX 4090?
RTX 4090 has 24 GB of memory. LLMScale budgets 21.76 GiB of it for a model: vLLM's default --gpu-memory-utilization of 0.92 of what the card reports, the rest kept for the CUDA context. Its memory bandwidth, 1,008 GB/s, is what limits generation speed once a model fits. Of the 31 popular open models below, 11 fit at BF16, 15 at FP8 and 20 at INT4 with an 8k context for one user.
20 of 31 models fit at INT4, 11 at BF16.
Which LLMs fit on RTX 4090
| Model | BF16 | FP8 | INT4 | Users | Max context |
|---|---|---|---|---|---|
| Qwen2.5 0.5B Instruct | 1.85 GiB | 1.52 GiB | 1.36 GiB | 213BF16 | 32kBF16 |
| Gemma 3 1B IT | 3.27 GiB | 2.62 GiB | 2.31 GiB | 135BF16 | 32kBF16 |
| Llama 3.2 1B Instruct | 3.63 GiB | 2.72 GiB | 2.29 GiB | 73BF16 | 128kBF16 |
| Llama 3.2 3B Instruct | 8.28 GiB | 5.65 GiB | 4.42 GiB | 16BF16 | 128kBF16 |
| Phi-3 mini 4k | 9.42 GiB | 6.04 GiB | 4.46 GiB | 9BF16 | 4kBF16 |
| Gemma 3 4B IT | 10.57 GiB | 7.19 GiB | 5.61 GiB | 16BF16 | 128kBF16 |
| Mistral 7B Instruct v0.3 | 15.43 GiB | 8.93 GiB | 5.88 GiB | 7BF16 | 32kBF16 |
| Qwen2.5 Coder 7B | 16.37 GiB | 10.30 GiB | 7.45 GiB | 13BF16 | 32kBF16 |
| Qwen2.5 7B Instruct | 16.37 GiB | 10.30 GiB | 7.45 GiB | 13BF16 | 32kBF16 |
| Llama 3.1 8B Instruct | 17.71 GiB | 11.21 GiB | 8.16 GiB | 5BF16 | 40kBF16 |
| Qwen3 8B | 18.37 GiB | 11.90 GiB | 8.87 GiB | 4BF16 | 32kBF16 |
| Gemma 2 9B IT | 22.48 GiB (does not fit) | 14.73 GiB | 11.09 GiB | 3FP8 | 8kFP8 |
| Gemma 3 12B IT | 26.85 GiB (does not fit) | 16.44 GiB | 11.55 GiB | 3FP8 | 93kFP8 |
| Mistral Nemo 12B Instruct | 26.23 GiB (does not fit) | 16.07 GiB | 11.31 GiB | 5FP8 | 44kFP8 |
| Qwen2.5 14B Instruct | 31.48 GiB (does not fit) | 19.17 GiB | 13.40 GiB | 2FP8 | 22kFP8 |
| gpt-oss 20B | 41.22 GiB (does not fit) | 22.82 GiB (does not fit) | 15.96 GiB | 21INT4 | 128kINT4 |
| Gemma 2 27B IT | 56.79 GiB (does not fit) | 32.55 GiB (does not fit) | 21.19 GiB | 1INT4 | 8kINT4 |
| Gemma 3 27B IT | 56.53 GiB (does not fit) | 32.29 GiB (does not fit) | 20.93 GiB | 1INT4 | 19kINT4 |
| Qwen3 30B A3B | 59.07 GiB (does not fit) | 31.21 GiB (does not fit) | 18.15 GiB | 5INT4 | 40kINT4 |
| Qwen3 Coder 30B A3B | 59.07 GiB (does not fit) | 31.21 GiB (does not fit) | 18.15 GiB | 5INT4 | 46kINT4 |
| Qwen2.5 32B Instruct | 65.64 GiB (does not fit) | 36.58 GiB (does not fit) | 22.96 GiB (does not fit) | — | — |
| Qwen3 32B | 65.64 GiB (does not fit) | 36.58 GiB (does not fit) | 22.96 GiB (does not fit) | — | — |
| Mixtral 8x7B Instruct | 89.46 GiB (does not fit) | 46.21 GiB (does not fit) | 25.94 GiB (does not fit) | — | — |
| Llama 3.3 70B Instruct | 137.6 GiB (does not fit) | 73.89 GiB (does not fit) | 44.01 GiB (does not fit) | — | — |
| Llama 3.1 70B Instruct | 137.6 GiB (does not fit) | 73.89 GiB (does not fit) | 44.01 GiB (does not fit) | — | — |
| Qwen2.5 72B Instruct | 142.0 GiB (does not fit) | 76.64 GiB (does not fit) | 45.99 GiB (does not fit) | — | — |
| Llama 4 Scout 17B-16E Instruct | 205.4 GiB (does not fit) | 106.2 GiB (does not fit) | 59.64 GiB (does not fit) | — | — |
| gpt-oss 120B | 220.1 GiB (does not fit) | 112.4 GiB (does not fit) | 68.40 GiB (does not fit) | — | — |
| Mixtral 8x22B Instruct | 265.6 GiB (does not fit) | 135.1 GiB (does not fit) | 73.87 GiB (does not fit) | — | — |
| Llama 3.1 405B Instruct | 768.3 GiB (does not fit) | 394.2 GiB (does not fit) | 218.9 GiB (does not fit) | — | — |
| DeepSeek V3 | 1279.1 GiB (does not fit) | 643.3 GiB (does not fit) | 345.2 GiB (does not fit) | — | — |
Memory at 8k context for one user (the model's maximum, where that is shorter), FP16 KV cache and vLLM's overhead at this card's defaults. Users (each at 8k) and maximum context are for the highest precision that fits.
What BF16, FP8 and INT4 mean for memory: quantization and VRAM.
A model that does not fit one card can still run across several: tensor parallelism splits its weights and KV cache between them. The calculator works out how many.