Serving brief — what a model's repository says about how to serve it
A model repository holds more than config.json. The chat template decides tool calling and thinking, generation_config.json the sampling defaults and when a reply ends, and the tensor names show what is really quantized and whether the multi-token-prediction layers shipped. Pick a model: this page reads those files in your browser, flags what would trip a deployment, and writes the vLLM and SGLang flags they call for.
Pick a model to read its repository
Or start from one of these:
Once a model is picked, this page answers:
- What is in this repository?
- How does it chat — tools, thinking, images?
- Which sampling settings does it ship?
- How are the weights stored?
- Can it draft its own tokens (MTP)?
- How long a context is it configured for?
- What does loading it need?
How the repository is read
The page reads a repository the way an engine does before it loads one: the file list and sizes from the Hugging Face API; config.json, generation_config.json, tokenizer_config.json or chat_template.jinja, hf_quant_config.json, adapter_config.json and params.json where the repository has them; and the tensor names, from model.safetensors.index.json or, for a single-file model, from the safetensors header, which takes two small ranged requests rather than a download. All of it runs in your browser.
A flag is written only when vLLM 0.31.0 or SGLang 0.5.21 names it, in its docs or its parser registry; a model family neither names gets no flag rather than a guess. MTP support comes from the code each engine uses to build its draft model. Memory, parallelism and batching are left to the sizing and vLLM pages.
Questions
- What is MTP, and why check the tensors?
- Multi-token-prediction layers are extra layers trained to predict a token further ahead. vLLM and SGLang can use them as a built-in draft model for speculative decoding. config.json only says how many there should be, and quantized copies sometimes drop them: DeepSeek-V3's AWQ copy declares one MTP layer and ships none of the 1,564 tensors the original has for it. So the page checks the tensor names.
- Why does generation_config.json matter?
- vLLM and SGLang take their default temperature, top_p and top_k from it, and the tokens that end a reply. Qwen3's lists two end tokens where its config.json lists one, and gpt-oss's adds two. vLLM also turns a max_new_tokens there into a server-wide cap.
- Which tool-call parser does a model need?
- The one that reads the format its chat template writes: JSON in <tool_call> blocks for Qwen, harmony channels for gpt-oss, <function=…> XML for Qwen3-Coder. The page reads the template and the model type, and names the parser each engine's docs give for that family. The names differ between engines: Qwen is hermes in vLLM and qwen in SGLang.
- Can I trust the quantization a repository declares?
- Usually, but the tensor names are the evidence. FP8 with block scales leaves weight_scale_inv tensors, NVFP4 weight_scale_2, MXFP4 _blocks and _scales, AWQ and GPTQ qweight and qzeros. NVIDIA's NVFP4 checkpoints declare theirs in hf_quant_config.json rather than config.json. The page reads both, and says when the names and the declaration disagree.