LLMScale — LLM Feed

inclusionAI releases Ling-3.1-flash, a 560B hybrid-linear MoE with 25B active

Tue, 20 Oct 2026 05:42:40 GMT

inclusionAI has released Ling-3.1-flash, a hybrid-linear MoE with 560B total and 25B active parameters and a stated 1M-token context. Its 72 layers split into 63 KDA and 9 Gated MLA layers, so only a small share of layers hold a conventional KV cache.

LLMScale estimates for one user at 8K context: 537.3 GiB at FP8 (4× B200) and 287.8 GiB at INT4 (fits a 512GB M3 Ultra). The card's SGLang recipe uses 8× H200 or 4× GB300. vLLM needs a dedicated branch.

Sources: Hugging Face · GitHub

Liquid AI releases LFM2.5-24B-A2B, a 24B MoE with 2B active and 131K context

Tue, 20 Oct 2026 05:42:15 GMT

Liquid AI released LFM2.5-24B-A2B, an open-weight reasoning MoE with 24B parameters, about 2B active per token and 64 experts with top-4 routing. Context grows from 32,768 to 131,072 tokens, on a hybrid of gated short convolutions and GQA. Liquid reports 184 tokens/s decode on an Apple M5 Max.

LLMScale estimates for one user at 8,192 tokens of context: 45.97 GiB in BF16 (A100 80GB), 23.89 GiB in FP8 (RTX 5090) and 13.54 GiB in INT4 (RTX 4060 Ti 16GB).

Sources: Liquid AI · Hugging Face

NVIDIA releases Nemotron 3.5 Super VL, a 120B-A12B multimodal MoE in NVFP4

Tue, 20 Oct 2026 05:39:42 GMT

NVIDIA has released Nemotron 3.5 Super VL, an open multimodal MoE model with 120B parameters, 12B active per token. One model covers text, image, document and video input. The checkpoint uses NVFP4: 4-bit weights and activations on Blackwell, 4-bit weights with 16-bit activations on other GPUs.

All 120B weights must fit in memory even though only 12B run per token. NVIDIA says LiLiCorr speculative decoding speeds generation up to 2.3x on DGX Spark.

Sources: NVIDIA Developer Blog · Hugging Face

vLLM adds day-0 support for Muse Spark Open, an 816B-A42B MoE with 1M context

Tue, 20 Oct 2026 05:39:30 GMT

vLLM has day-0 support for Meta's Muse Spark Open, an open-weight MoE with 816B total and 42B active parameters and a 1M-token context. Weights come in BF16 and NVFP4, plus MXFP4 from AMD.

46 of its 62 layers use a 2048-token sliding window, so KV cache at 1M tokens is just over a quarter of all-full attention. The example serves NVFP4 at TP8 with expert parallelism and a DSpark draft model for speculative decoding.

Sources: vLLM · GitHub

NVIDIA releases Dory1.0-4B, a looped 4B model with 256K context

Tue, 20 Oct 2026 05:39:30 GMT

NVIDIA released Dory1.0-4B, a 4B looped language model: a shared 20-layer block repeats between 8 input and 8 output layers, with interleaved sliding-window and full attention and up to 256K context. Loop count is set at inference; SWE-bench Verified goes from 50.3 at 1 loop to 61.8 at 4.

Per the model card, last-loop KV caching means extra loops add compute but no KV cache over a non-looped model. LLMScale estimates 11.12 GiB in BF16 and 6.60 GiB in INT4 for one user at 8K context.

Sources: Hugging Face · Reddit r/LocalLLaMA

NVIDIA releases Nemotron-Labs-Dory1.0-4B, a looped model with 256K context

Tue, 20 Oct 2026 05:39:10 GMT

NVIDIA released Nemotron-Labs-Dory1.0-4B, an open 4B looped model. A shared 20-layer block sits between 8-layer input and output blocks and runs repeatedly. It mixes sliding-window and full attention and supports up to 256K context.

Per-loop caching grows the KV cache with each loop; last-loop mode keeps it at one-loop size. LLMScale estimates 11.12 GiB in BF16 and 8.05 GiB in FP8 for one user at 8K context. vLLM support is still an open PR.

Sources: Hugging Face · GitHub · GitHub

Unsloth opens GGUF repo for DeepSeek-V4.1-Flash-1017; quants not yet posted

Tue, 20 Oct 2026 05:38:45 GMT

Unsloth has opened a Hugging Face repo for GGUF builds of DeepSeek-V4.1-Flash-1017, an MIT-licensed minor update to DeepSeek-V4.1-Flash with reported gains in reasoning, coding and agentic tasks. The page says the quants are not posted yet.

LLMScale estimates 748.49B parameters, about 173.27B active per token (MoE). For one user at 8,192 tokens of context: 701.1 GiB at FP8 (8× B200) and 374.9 GiB at INT4, enough for a 512GB M3 Ultra. Recommended context is 1M tokens.

Sources: Hugging Face · Reddit r/LocalLLaMA

Amazon releases ALoDLM-8B, a looped diffusion language model built on Qwen3-8B

Tue, 20 Oct 2026 05:38:45 GMT

Amazon has released ALoDLM-8B, a diffusion language model initialized from Qwen3-8B. It reuses shared transformer layers in a loop and gives unresolved tokens extra passes. A Reddit post claims about 3x Qwen3-8B's speed. The model card measures throughput single-stream on one B200, in a parallel-commit mode separate from its quality runs. The license is CC BY-NC 4.0.

LLMScale estimates for one user at 8,192 tokens of context: 18.37 GiB in BF16, 11.90 GiB in FP8 and 8.87 GiB in INT4.

Sources: Hugging Face · Reddit r/LocalLLaMA

vLLM on eight MI355X GPUs tops HGX B300 per-GPU throughput on Kimi K3

Tue, 20 Oct 2026 05:38:34 GMT

vLLM reports that an eight-GPU AMD Instinct MI355X system serving Kimi K3 reaches about 7,567 total tokens/s per GPU on the AgentX coding-agent benchmark. That is 24.4% more than NVIDIA HGX B300 (6,083) at 50 tokens/s/user P90. Both GPUs have 288 GB HBM3E.

The biggest gains came from cache handling. DCP8 splits the MLA KV history across GPUs, cutting a 142K-token conversation from about 1.96 GB to 245 MB per GPU. LLMScale estimates 1458.6 GiB at INT4 for one user at 8K context.

Sources: vLLM · Hugging Face · vLLM

Four open-weight releases in 72 hours; only NVIDIA's Nemotron 3.5 Bolt fits 24GB

Tue, 20 Oct 2026 05:38:34 GMT

A r/LocalLLaMA recap lists four open-weight releases in 72 hours: Meta's Muse Spark Open (816B total, 42B active), Reflection's Beam (501B, 23B active), DeepSeek V4.1 Flash 1017 (510GB repo) and NVIDIA's Nemotron 3.5 Bolt (74B, 9B active).

Only Bolt targets a 24GB GPU: quantization-aware training to 2 bit puts weights at 22GB plus a 1.7GB vision projector, leaving little room for KV cache. Beam's smallest official checkpoint is 294GB in NVFP4. Sizes are from the post, not LLMScale estimates.

Sources: Reddit r/LocalLLaMA

vLLM's specialized decode path speeds Kimi K3 up to 1.9x on NVIDIA GB300

Tue, 20 Oct 2026 05:38:19 GMT

vLLM added a specialized Kimi K3 decode path for NVIDIA GB300. It uses agent-assisted CuTe DSL kernels that fuse attention, MoE and communication across 8 or 16 GPUs in one NVLink domain. Per-layer decode time falls from 130 to 69 microseconds.

Without speculation, throughput rises 1.62–1.92x on TP16 and 1.31–1.58x on TP8 at batch 1–64. With DSpark (synthetic AL=6), batch 1 on TP16 hits 842 tokens/s, against 488 on stock vLLM. For long contexts, decode context parallelism splits the KV cache across GPUs.

Sources: vLLM · GitHub

Intel names Crescent Island GPU D-Series, with up to 480 GB of LPDDR5X

Tue, 13 Oct 2026 17:14:17 GMT

Intel has named its Crescent Island GPU the D-Series. The D270 has 32 Xe3P cores, 256 XMX engines and 32 MB of L2. It is rated at up to 1.3 PFLOPS FP8 and supports formats from FP4 to FP64, on an air-cooled 350 W PCIe Gen5 x16 card. Intel's own version has 160 GB of LPDDR5X, ODM designs go up to 480 GB, and bandwidth is rated at 1.5 TB/s.

That capacity could hold large weights plus long-context KV cache on one card. However, 1.5 TB/s limits decode speed, which is bandwidth-bound. No pricing yet.

Sources: Reddit r/LocalLLaMA

vLLM adds bi-directional KV transfer, cutting multi-turn TTFT up to 3.3x

Tue, 13 Oct 2026 17:13:50 GMT

vLLM's prefill-decode disaggregated serving can now send KV cache back from the decode node to the prefill node. Later turns of an agent conversation reuse earlier context instead of recomputing it. On a 1P1D EC2 P5en setup, with prefix caching off on the prefill node, TTFT at long prompts fell by up to 1.6x for Qwen3-32B and up to 3.3x for GLM-5-FP8.

The cost falls on decode VRAM. Finished KV blocks are kept for decoder_kv_blocks_ttl (default 480 s), so idle sessions hold memory until the timer expires. Remainders below kv_recompute_threshold (default 64 tokens) are recomputed instead.

Sources: vLLM · vLLM · GitHub

Ollama v0.40.0 makes MLX the default runtime on Apple Silicon

Sun, 11 Oct 2026 15:11:00 GMT

Ollama v0.40.0 makes MLX the default runtime on Apple Silicon. Model architectures that MLX supports now run on it automatically. Listed models include qwen3.8, gemma4, qwen3.6 and qwen3.5, plus the embedding model embeddinggemma-2. More models will be enabled after testing.

On Macs this swaps the inference backend with no configuration change. The release notes give no speed or memory figures, so re-measure throughput and memory use on your own hardware.

Sources: GitHub

Mistral Large 4 preview: 1T-parameter MoE with 49B active, open weights due end of month

Sat, 10 Oct 2026 09:41:50 GMT

Mistral released a preview of Mistral Large 4 via its API: an MoE model with 1T total and 49B active parameters, trained on 3,800 NVIDIA Grace Blackwell GPUs. Open weights are promised for the end of this month. It scores 38 on Artificial Analysis, up from 9 for Mistral Large 3.

For self-hosting, per-token compute follows the 49B active parameters, but all 1T weights must be held in memory, so VRAM rather than compute sets the hardware floor. No memory estimate yet: the model config is not public.

Sources: Mistral AI · Simon Willison's Weblog