LLMScaleWhich LLMs fit on RTX 3060?
RTX 3060 has 12 GB of memory. LLMScale budgets 11.04 GiB of it for a model: vLLM's default --gpu-memory-utilization of 0.92 of what the card reports, the rest kept for the CUDA context. Its memory bandwidth, 360 GB/s, is what limits generation speed once a model fits. Of the 31 popular open models below, 6 fit at BF16, 9 at FP8 and 11 at INT4 with an 8k context for one user.
11 of 31 models fit at INT4, 6 at BF16.
Which LLMs fit on RTX 3060
| Model | BF16 | FP8 | INT4 | Users | Max context |
|---|---|---|---|---|---|
| Qwen2.5 0.5B Instruct | 1.85 GiB | 1.52 GiB | 1.36 GiB | 98BF16 | 32kBF16 |
| Gemma 3 1B IT | 3.27 GiB | 2.62 GiB | 2.31 GiB | 57BF16 | 32kBF16 |
| Llama 3.2 1B Instruct | 3.63 GiB | 2.72 GiB | 2.29 GiB | 30BF16 | 128kBF16 |
| Llama 3.2 3B Instruct | 8.28 GiB | 5.65 GiB | 4.42 GiB | 4BF16 | 33kBF16 |
| Phi-3 mini 4k | 9.42 GiB | 6.04 GiB | 4.46 GiB | 2BF16 | 4kBF16 |
| Gemma 3 4B IT | 10.57 GiB | 7.19 GiB | 5.61 GiB | 1BF16 | 32kBF16 |
| Mistral 7B Instruct v0.3 | 15.43 GiB (does not fit) | 8.93 GiB | 5.88 GiB | 3FP8 | 25kFP8 |
| Qwen2.5 Coder 7B | 16.37 GiB (does not fit) | 10.30 GiB | 7.45 GiB | 2FP8 | 22kFP8 |
| Qwen2.5 7B Instruct | 16.37 GiB (does not fit) | 10.30 GiB | 7.45 GiB | 2FP8 | 22kFP8 |
| Llama 3.1 8B Instruct | 17.71 GiB (does not fit) | 11.21 GiB (does not fit) | 8.16 GiB | 3INT4 | 31kINT4 |
| Qwen3 8B | 18.37 GiB (does not fit) | 11.90 GiB (does not fit) | 8.87 GiB | 2INT4 | 23kINT4 |
| Gemma 2 9B IT | 22.48 GiB (does not fit) | 14.73 GiB (does not fit) | 11.09 GiB (does not fit) | — | — |
| Gemma 3 12B IT | 26.85 GiB (does not fit) | 16.44 GiB (does not fit) | 11.55 GiB (does not fit) | — | — |
| Mistral Nemo 12B Instruct | 26.23 GiB (does not fit) | 16.07 GiB (does not fit) | 11.31 GiB (does not fit) | — | — |
| Qwen2.5 14B Instruct | 31.48 GiB (does not fit) | 19.17 GiB (does not fit) | 13.40 GiB (does not fit) | — | — |
| gpt-oss 20B | 41.22 GiB (does not fit) | 22.82 GiB (does not fit) | 15.96 GiB (does not fit) | — | — |
| Gemma 2 27B IT | 56.79 GiB (does not fit) | 32.55 GiB (does not fit) | 21.19 GiB (does not fit) | — | — |
| Gemma 3 27B IT | 56.53 GiB (does not fit) | 32.29 GiB (does not fit) | 20.93 GiB (does not fit) | — | — |
| Qwen3 30B A3B | 59.07 GiB (does not fit) | 31.21 GiB (does not fit) | 18.15 GiB (does not fit) | — | — |
| Qwen3 Coder 30B A3B | 59.07 GiB (does not fit) | 31.21 GiB (does not fit) | 18.15 GiB (does not fit) | — | — |
| Qwen2.5 32B Instruct | 65.64 GiB (does not fit) | 36.58 GiB (does not fit) | 22.96 GiB (does not fit) | — | — |
| Qwen3 32B | 65.64 GiB (does not fit) | 36.58 GiB (does not fit) | 22.96 GiB (does not fit) | — | — |
| Mixtral 8x7B Instruct | 89.46 GiB (does not fit) | 46.21 GiB (does not fit) | 25.94 GiB (does not fit) | — | — |
| Llama 3.3 70B Instruct | 137.6 GiB (does not fit) | 73.89 GiB (does not fit) | 44.01 GiB (does not fit) | — | — |
| Llama 3.1 70B Instruct | 137.6 GiB (does not fit) | 73.89 GiB (does not fit) | 44.01 GiB (does not fit) | — | — |
| Qwen2.5 72B Instruct | 142.0 GiB (does not fit) | 76.64 GiB (does not fit) | 45.99 GiB (does not fit) | — | — |
| Llama 4 Scout 17B-16E Instruct | 205.4 GiB (does not fit) | 106.2 GiB (does not fit) | 59.64 GiB (does not fit) | — | — |
| gpt-oss 120B | 220.1 GiB (does not fit) | 112.4 GiB (does not fit) | 68.40 GiB (does not fit) | — | — |
| Mixtral 8x22B Instruct | 265.6 GiB (does not fit) | 135.1 GiB (does not fit) | 73.87 GiB (does not fit) | — | — |
| Llama 3.1 405B Instruct | 768.3 GiB (does not fit) | 394.2 GiB (does not fit) | 218.9 GiB (does not fit) | — | — |
| DeepSeek V3 | 1279.1 GiB (does not fit) | 643.3 GiB (does not fit) | 345.2 GiB (does not fit) | — | — |
Memory at 8k context for one user (the model's maximum, where that is shorter), FP16 KV cache and vLLM's overhead at this card's defaults. Users (each at 8k) and maximum context are for the highest precision that fits.
What BF16, FP8 and INT4 mean for memory: quantization and VRAM.
A model that does not fit one card can still run across several: tensor parallelism splits its weights and KV cache between them. The calculator works out how many.