vLLM's specialized decode path speeds Kimi K3 up to 1.9x on NVIDIA GB300
vLLM added a specialized Kimi K3 decode path for NVIDIA GB300. It uses agent-assisted CuTe DSL kernels that fuse attention, MoE and communication across 8 or 16 GPUs in one NVLink domain. Per-layer decode time falls from 130 to 69 microseconds.
Without speculation, throughput rises 1.62–1.92x on TP16 and 1.31–1.58x on TP8 at batch 1–64. With DSpark (synthetic AL=6), batch 1 on TP16 hits 842 tokens/s, against 488 on stock vLLM. For long contexts, decode context parallelism splits the KV cache across GPUs.
Intel has named its Crescent Island GPU the D-Series. The D270 has 32 Xe3P cores, 256 XMX engines and 32 MB of L2. It is rated at up to 1.3 PFLOPS FP8 and supports formats from FP4 to FP64, on an air-cooled 350 W PCIe Gen5 x16 card. Intel's own version has 160 GB of LPDDR5X, ODM designs go up to 480 GB, and bandwidth is rated at 1.5 TB/s.
That capacity could hold large weights plus long-context KV cache on one card. However, 1.5 TB/s limits decode speed, which is bandwidth-bound. No pricing yet.
vLLM's prefill-decode disaggregated serving can now send KV cache back from the decode node to the prefill node. Later turns of an agent conversation reuse earlier context instead of recomputing it. On a 1P1D EC2 P5en setup, with prefix caching off on the prefill node, TTFT at long prompts fell by up to 1.6x for Qwen3-32B and up to 3.3x for GLM-5-FP8.
The cost falls on decode VRAM. Finished KV blocks are kept for decoder_kv_blocks_ttl (default 480 s), so idle sessions hold memory until the timer expires. Remainders below kv_recompute_threshold (default 64 tokens) are recomputed instead.
Ollama v0.40.0 makes MLX the default runtime on Apple Silicon. Model architectures that MLX supports now run on it automatically. Listed models include qwen3.8, gemma4, qwen3.6 and qwen3.5, plus the embedding model embeddinggemma-2. More models will be enabled after testing.
On Macs this swaps the inference backend with no configuration change. The release notes give no speed or memory figures, so re-measure throughput and memory use on your own hardware.
Mistral released a preview of Mistral Large 4 via its API: an MoE model with 1T total and 49B active parameters, trained on 3,800 NVIDIA Grace Blackwell GPUs. Open weights are promised for the end of this month. It scores 38 on Artificial Analysis, up from 9 for Mistral Large 3.
For self-hosting, per-token compute follows the 49B active parameters, but all 1T weights must be held in memory, so VRAM rather than compute sets the hardware floor. No memory estimate yet: the model config is not public.
inclusionAI has released Ling-3.1-flash, a hybrid-linear MoE with 560B total and 25B active parameters and a stated 1M-token context. Its 72 layers split into 63 KDA and 9 Gated MLA layers, so only a small share of layers hold a conventional KV cache.
LLMScale estimates for one user at 8K context: 537.3 GiB at FP8 (4× B200) and 287.8 GiB at INT4 (fits a 512GB M3 Ultra). The card's SGLang recipe uses 8× H200 or 4× GB300. vLLM needs a dedicated branch.
#moe #kv-cache #long-context #sglang
Ling-3.1-flash · 572.12B parameters · 8k context, one user
FP16 KV cache and vLLM's overhead on each device at its defaults. “Fits on”: the smallest device that holds it alone, or how many of the largest it takes. Change the context, users or GPU in the calculator
LLMScale