Phi-3 mini 4k: how much VRAM does it need?

Phi-3 mini 4k has 3.82B parameters in 32 layers. It uses full multi-head attention: each of its 32 heads keeps its own keys and values, so the KV cache is as large as its head count makes it. It takes up to 4,096 tokens of context; one user at that length needs 1.50 GiB of KV cache at FP16 on top of the weights.

Phi-3 mini 4k memory by precision and context

One user, FP16 KV cache and vLLM's overhead on one GPU at its defaults for a card under 70 GB. Quantizing shrinks the weights, not the KV cache, so the saving narrows as the context grows.

Which single GPUs hold it

• BF16 (9.42 GiB): RTX 3060, NVIDIA T4, RTX 4060 Ti 16GB, RTX 4070 Ti Super
• FP8 (6.04 GiB): RTX 3060, NVIDIA T4, RTX 4060 Ti 16GB, RTX 4070 Ti Super
• INT4 (4.46 GiB): RTX 3060, NVIDIA T4, RTX 4060 Ti 16GB, RTX 4070 Ti Super

At 4k context for one user, the smallest cards first. A discrete card is budgeted at vLLM's default 92% of its memory, a unified-memory device at the share its GPU can address, and each carries vLLM's overhead at its own defaults.

How many users fit

Users at 4k context each, on one card. ✗: the weights leave no room for even one.

Fine-tuning Phi-3 mini 4k

Sequence length 2,048, batch size 1, gradient checkpointing on, Adam. Full fine-tuning keeps 16 bytes per parameter before activations; QLoRA stores the frozen base in 4-bit NF4.

Where the numbers come from

Weights are parameters × bytes per parameter. The KV cache is 2 × layers × key-value heads × head dimension × bytes × context × users. For Phi-3 mini 4k that is 384.0 KiB per token at FP16. These are capacity-planning figures: the serving engine — vLLM, TGI, llama.cpp — and its paged-attention efficiency still move the real number.

More on KV cache size and on GQA, MQA and MLA.

Work with Phi-3 mini 4k

• Size it for your workload
• Fine-tuning memory
• vLLM parameters
• Model anatomy
• Its config.json, explained
• What its files say about serving it