LLMScalelive Hugging Face data

vLLM parameter helper — a tuned vllm serve command

Generate a tuned `vllm serve` command. Pick a model, what you're optimizing for, and your workload — LLMScale recommends flags for the best performance, accuracy and reliability, with a copyable command.

1

Choose a model

built-in DBgatedllama8.03B params·32L · 4096d · 32/8 headsGQA
2

Optimize for

Type exact numbers instead of picking from the lists. Remembered in this browser.
vLLM's defaults with what this model and card need.The most tokens a second across all users: big batches, more of the card, FP8 KV cache for long contexts.The fastest reply for each user: small batches and fewer sequences at once.Nothing that trades quality: 16-bit KV cache and BF16 weights.Fits the most on a small card: more of its memory, FP8 KV cache, fewer sequences.
3

Workload

A conversational assistant.Long prompts with retrieved passages.Code generation and completion.Function calling: adds the tool-call parser for this model.Replies that must follow a JSON schema: adds the structured-output backend.Images in the prompt: limits images per request.A thinking model: separates its reasoning from the answer.
ⓘHow many GPUs vLLM runs on. The page lays them out as the calculator does: tensor parallel where the heads allow, pipeline stages, replicas, and for a MoE expert parallelism.
ⓘThe longest request vLLM accepts, prompt and reply together.model max 131,072

Recommended command

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 8192
modelmeta-llama/Llama-3.1-8B-Instruct
What each flag does
  • --max-model-len 8192Caps the context window — frees KV cache vs the model's full max.

Check against a vLLM log

Paste what vllm serve printed during startup. LLMScale reads vLLM's own figures and places them beside its estimate for the model, card and settings above. The log stays in your browser.

How the vllm serve flags are chosen

Pick a model, a GPU and how many of them, a context length, a priority (balanced, throughput, low latency, max accuracy or memory-tight) and a task (chat, RAG, code, tool use, structured JSON, vision or reasoning). The helper turns that into a vllm serve command with a one-line reason for every flag, and the same flags as Kubernetes args.

  • --max-model-len = the context you set
  • --tensor-parallel-size, --pipeline-parallel-size, --data-parallel-size and --enable-expert-parallel = the calculator's layout on that many GPUs: of the ways vLLM can run the model on them that hold the requests at the context, the fastest (of those within 10% of it, the one that holds the most; for the throughput and memory priorities, the one that holds the most), with pipeline stages only when no other layout holds them (or the one a link or the layout picker names)
  • --gpu-memory-utilization = 0.95 when optimizing for throughput or memory-tight; otherwise vLLM's default, 0.92, is left as it is
  • --kv-cache-dtype fp8 = memory-tight or throughput priority, with 32k+ context or a RAG task; always for DeepSeek V4, with --block-size 256, as its vLLM recipe sets them
  • --max-num-seqs = 512 (throughput), 32 (low latency) or 64 (memory-tight), capped at the full-length sequences the KV cache holds once each GPU has its share of the weights and vLLM's overhead
  • Prefix caching and chunked prefill are on by default in vLLM 0.31, so the command leaves them out

Tool use adds --enable-auto-tool-choice with the tool-call parser for the model's family, structured JSON adds the xgrammar backend, and vision limits images per prompt. Pre-quantized repositories (AWQ, GPTQ, FP8, NVFP4) need no --quantization flag, and the page says so. A memory check — the VRAM calculator's own sizing of one full-length request on each GPU of the layout — warns when the model will not fit, and gives the calculator's count of these GPUs and how to lay them out. Apple devices are left out because vLLM has no Metal backend.

Questions

What does --gpu-memory-utilization do?
It is the fraction of each GPU's memory vLLM may take for model weights, activations and KV cache. What the weights do not use becomes KV cache, so a higher value fits more concurrent sequences and leaves less headroom for anything else on the card.
Why is --max-num-seqs lower than I expected?
The helper caps it at what the KV cache can actually hold at the context length you chose. Setting it higher does not add capacity: requests past that point wait in the queue.
Should I use an FP8 KV cache?
It halves the KV cache, so roughly twice as many sequences fit, but it can cost accuracy on long-context work. That is why the helper only suggests it when memory or throughput is the priority and the context is long, and asks you to validate it first.
How many GPUs do I need, and how are they split?
Enough for the weights and your workload's KV cache to fit. When one does not hold it, the memory check gives the VRAM calculator's count and layout: tensor parallel where the attention heads divide, pipeline stages when they do not or when the cache cannot be split further, and for a MoE its experts spread over data-parallel ranks. The capacity planner sizes the same layouts for a load.

Go deeper