Capacity plan — how many GPUs a load needs to meet its latency targets
Describe the load: how many requests arrive, how long their prompts and replies are, and how fast they must be answered. LLMScale works out how many vLLM replicas are needed and how many GPUs each one runs on.
Choose a model
Load
Latency targets
Hardware
What it takes
| Layout | Replicas | GPU | req/s each | Limit |
|---|---|---|---|---|
| 1 GPU | 1 | 1 | 11.1 | limited by time per token |
| TP 2 | 1 | 2 | 23.1 | limited by time per token |
| PP 2 | 1 | 2 | 22.3 | limited by time per token |
| PP 3 | 1 | 3 | 32.2 | limited by time per token |
| TP 4 | 1 | 4 | 47.5 | limited by time per token |
| TP 2 · PP 2 | 1 | 4 | 46.2 | limited by time per token |
| PP 4 | 1 | 4 | 44.5 | limited by time per token |
| PP 5 | 1 | 5 | 50.1 | limited by time per token |
A planning model for steady traffic at average lengths, without spikes. Reading prompts takes a share of each replica's time and stretches decode by the same factor; the number of sequences in decode follows Little's law. Speeds are constrained by memory bandwidth at 92% of the vendor's peak and dense tensor throughput at 90% on GeForce cards (measured) or 60% on other cards (estimated), plus a fixed cost per layer for every step, two all-reduces per layer across a tensor-parallel group, and about 6 ms per request before its first token. Pipeline stages each run a batch, at the speed as many replicas would have; their handovers and expert parallelism's dispatch are costed like all-reduces, an estimate not yet measured; on one NVIDIA GPU this lands within about 10–15% of vLLM 0.31 measurements. Memory usage comes from vLLM's own accounting. Leave headroom for bursts. On Strix Halo the per-layer cost and tensor throughput are measured (vLLM ROCm), but more than four sequences and short prompts run up to 3× slower; on other AMD, Apple and unified-memory devices, and for MoE prompt reading, speeds remain upper bounds.
How the plan is worked out
One replica is a vLLM instance laid out as the VRAM calculator lays one out: one GPU, a tensor-parallel group, pipeline stages of such groups, or for a MoE data-parallel ranks with its experts spread over them. At a given request rate, it spends part of its time reading prompts. The prompt time is the prompt's FLOPs divided by the attainable dense tensor throughput of the GPU (90% of the peak on GeForce cards, measured; 60% on other cards, estimated), or the time required to read the weights once if that is longer. The rest of its time goes to decode steps, so prompt and decode times stretch by the same factor. The step time at a given batch size comes from the decode model: the longer of the time required to read the weights and KV cache and the time required for the matrix multiplications. The planner finds the highest rate that keeps the time to first token, the time per output token and the KV cache within bounds, and divides the load by it.
- prefill share f = requests per second per replica × prefill time
- time per output token = decode step at the batch ÷ (1 − f)
- sequences in decode = requests per second × output tokens × time per output token
- GPUs = ⌈requests per second ÷ one replica's capacity⌉ × GPUs in a replica
It tries every such layout on up to 8 GPUs that one request at the max context fits, and on the calculator's fewest when the model needs more, and lists them cheapest first. Pipeline stages each run a batch, so a replica of them is as fast as that many replicas of one stage would be; they are what makes a model fit that a tensor-parallel group cannot hold. A shared prefix is cached once and not recomputed. Arrivals are assumed to be steady, and prompt and output lengths are represented by their averages, so real traffic with bursts and long tails needs additional headroom.
Questions
- Why can a model that fits one GPU need several?
- Memory says whether a model runs; the load says how many copies it takes. Every request first has its prompt read, which is compute-bound, and then holds a slot in every decode step until its reply ends. Llama 3.1 8B fits one H100 with room to spare, yet at 2,048-token prompts and 256-token replies with a 50 ms per-token target one H100 carries about 10 requests a second, so 15 a second take two.
- Why does tensor parallelism sometimes cost no more GPUs?
- Splitting a model across GPUs splits the weights each step reads and the work each prompt needs, so each step is shorter and each prompt is read sooner. When a latency target is what limits a replica, two GPUs as one replica can carry more than twice what one does; when the KV cache or raw throughput limits it, separate replicas are as good and simpler.
- How far can I trust the figures?
- They are first-order. Memory comes from vLLM's own accounting, checked against real startups. The speeds use vendor peaks with stated efficiencies, and the queueing model assumes steady arrivals. Treat the GPU count as the floor for your average load and add 20 to 30% for bursts, then confirm with a benchmark on one replica.