LLMScaleWhich LLMs fit on NVIDIA L40S?
NVIDIA L40S has 48 GB of memory. LLMScale budgets 40.81 GiB of it for a model: vLLM's default --gpu-memory-utilization of 0.92 of what the card reports, the rest kept for the CUDA context. Its memory bandwidth, 864 GB/s, is what limits generation speed once a model fits. Of the 31 popular open models below, 15 fit at BF16, 22 at FP8 and 23 at INT4 with an 8k context for one user.
23 of 31 models fit at INT4, 15 at BF16.
Which LLMs fit on NVIDIA L40S
| Model | BF16 | FP8 | INT4 | Users | Max context |
|---|---|---|---|---|---|
| Qwen2.5 0.5B Instruct | 1.85 GiB | 1.52 GiB | 1.36 GiB | 416BF16 | 32kBF16 |
| Gemma 3 1B IT | 3.27 GiB | 2.62 GiB | 2.31 GiB | 274BF16 | 32kBF16 |
| Llama 3.2 1B Instruct | 3.63 GiB | 2.72 GiB | 2.29 GiB | 149BF16 | 128kBF16 |
| Llama 3.2 3B Instruct | 8.28 GiB | 5.65 GiB | 4.42 GiB | 38BF16 | 128kBF16 |
| Phi-3 mini 4k | 9.42 GiB | 6.04 GiB | 4.46 GiB | 21BF16 | 4kBF16 |
| Gemma 3 4B IT | 10.57 GiB | 7.19 GiB | 5.61 GiB | 41BF16 | 128kBF16 |
| Mistral 7B Instruct v0.3 | 15.43 GiB | 8.93 GiB | 5.88 GiB | 26BF16 | 32kBF16 |
| Qwen2.5 Coder 7B | 16.37 GiB | 10.30 GiB | 7.45 GiB | 56BF16 | 32kBF16 |
| Qwen2.5 7B Instruct | 16.37 GiB | 10.30 GiB | 7.45 GiB | 56BF16 | 32kBF16 |
| Llama 3.1 8B Instruct | 17.71 GiB | 11.21 GiB | 8.16 GiB | 24BF16 | 128kBF16 |
| Qwen3 8B | 18.37 GiB | 11.90 GiB | 8.87 GiB | 20BF16 | 40kBF16 |
| Gemma 2 9B IT | 22.48 GiB | 14.73 GiB | 11.09 GiB | 7BF16 | 8kBF16 |
| Gemma 3 12B IT | 26.85 GiB | 16.44 GiB | 11.55 GiB | 7BF16 | 128kBF16 |
| Mistral Nemo 12B Instruct | 26.23 GiB | 16.07 GiB | 11.31 GiB | 12BF16 | 101kBF16 |
| Qwen2.5 14B Instruct | 31.48 GiB | 19.17 GiB | 13.40 GiB | 7BF16 | 32kBF16 |
| gpt-oss 20B | 41.22 GiB (does not fit) | 22.82 GiB | 15.96 GiB | 64FP8 | 128kFP8 |
| Gemma 2 27B IT | 56.79 GiB (does not fit) | 32.55 GiB | 21.19 GiB | 3FP8 | 8kFP8 |
| Gemma 3 27B IT | 56.53 GiB (does not fit) | 32.29 GiB | 20.93 GiB | 3FP8 | 117kFP8 |
| Qwen3 30B A3B | 59.07 GiB (does not fit) | 31.21 GiB | 18.15 GiB | 13FP8 | 40kFP8 |
| Qwen3 Coder 30B A3B | 59.07 GiB (does not fit) | 31.21 GiB | 18.15 GiB | 13FP8 | 110kFP8 |
| Qwen2.5 32B Instruct | 65.64 GiB (does not fit) | 36.58 GiB | 22.96 GiB | 3FP8 | 25kFP8 |
| Qwen3 32B | 65.64 GiB (does not fit) | 36.58 GiB | 22.96 GiB | 3FP8 | 25kFP8 |
| Mixtral 8x7B Instruct | 89.46 GiB (does not fit) | 46.21 GiB (does not fit) | 25.94 GiB | 15INT4 | 32kINT4 |
| Llama 3.3 70B Instruct | 137.6 GiB (does not fit) | 73.89 GiB (does not fit) | 44.01 GiB (does not fit) | — | — |
| Llama 3.1 70B Instruct | 137.6 GiB (does not fit) | 73.89 GiB (does not fit) | 44.01 GiB (does not fit) | — | — |
| Qwen2.5 72B Instruct | 142.0 GiB (does not fit) | 76.64 GiB (does not fit) | 45.99 GiB (does not fit) | — | — |
| Llama 4 Scout 17B-16E Instruct | 205.4 GiB (does not fit) | 106.2 GiB (does not fit) | 59.64 GiB (does not fit) | — | — |
| gpt-oss 120B | 220.1 GiB (does not fit) | 112.4 GiB (does not fit) | 68.40 GiB (does not fit) | — | — |
| Mixtral 8x22B Instruct | 265.6 GiB (does not fit) | 135.1 GiB (does not fit) | 73.87 GiB (does not fit) | — | — |
| Llama 3.1 405B Instruct | 768.3 GiB (does not fit) | 394.2 GiB (does not fit) | 218.9 GiB (does not fit) | — | — |
| DeepSeek V3 | 1279.1 GiB (does not fit) | 643.3 GiB (does not fit) | 345.2 GiB (does not fit) | — | — |
Memory at 8k context for one user (the model's maximum, where that is shorter), FP16 KV cache and vLLM's overhead at this card's defaults. Users (each at 8k) and maximum context are for the highest precision that fits.
What BF16, FP8 and INT4 mean for memory: quantization and VRAM.
A model that does not fit one card can still run across several: tensor parallelism splits its weights and KV cache between them. The calculator works out how many.