NVIDIA releases Dory1.0-4B, a looped 4B model with 256K context
NVIDIA released Dory1.0-4B, a 4B looped language model: a shared 20-layer block repeats between 8 input and 8 output layers, with interleaved sliding-window and full attention and up to 256K context. Loop count is set at inference; SWE-bench Verified goes from 50.3 at 1 loop to 61.8 at 4.
Per the model card, last-loop KV caching means extra loops add compute but no KV cache over a non-looped model. LLMScale estimates 11.12 GiB in BF16 and 6.60 GiB in INT4 for one user at 8K context.
#looped-model #kv-cache #nvidia #long-context
NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 · 3.98B parameters · 8k context, one user
FP16 KV cache and vLLM's overhead on each device at its defaults. “Fits on”: the smallest device that holds it alone, or how many of the largest it takes. Change the context, users or GPU in the calculator
NVIDIA released Nemotron-Labs-Dory1.0-4B, an open 4B looped model. A shared 20-layer block sits between 8-layer input and output blocks and runs repeatedly. It mixes sliding-window and full attention and supports up to 256K context.
Per-loop caching grows the KV cache with each loop; last-loop mode keeps it at one-loop size. LLMScale estimates 11.12 GiB in BF16 and 8.05 GiB in FP8 for one user at 8K context. vLLM support is still an open PR.
#looped-models #kv-cache #long-context #vllm
NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 · 3.98B parameters · 8k context, one user
FP16 KV cache and vLLM's overhead on each device at its defaults. “Fits on”: the smallest device that holds it alone, or how many of the largest it takes. Change the context, users or GPU in the calculator
Unsloth has opened a Hugging Face repo for GGUF builds of DeepSeek-V4.1-Flash-1017, an MIT-licensed minor update to DeepSeek-V4.1-Flash with reported gains in reasoning, coding and agentic tasks. The page says the quants are not posted yet.
LLMScale estimates 748.49B parameters, about 173.27B active per token (MoE). For one user at 8,192 tokens of context: 701.1 GiB at FP8 (8× B200) and 374.9 GiB at INT4, enough for a 512GB M3 Ultra. Recommended context is 1M tokens.
#deepseek #gguf #moe #quantization
DeepSeek-V4.1-Flash-1017-GGUF · 748.49B parameters · 8k context, one user
FP16 KV cache and vLLM's overhead on each device at its defaults. “Fits on”: the smallest device that holds it alone, or how many of the largest it takes. Change the context, users or GPU in the calculator
Amazon has released ALoDLM-8B, a diffusion language model initialized from Qwen3-8B. It reuses shared transformer layers in a loop and gives unresolved tokens extra passes. A Reddit post claims about 3x Qwen3-8B's speed. The model card measures throughput single-stream on one B200, in a parallel-commit mode separate from its quality runs. The license is CC BY-NC 4.0.
LLMScale estimates for one user at 8,192 tokens of context: 18.37 GiB in BF16, 11.90 GiB in FP8 and 8.87 GiB in INT4.
FP16 KV cache and vLLM's overhead on each device at its defaults. “Fits on”: the smallest device that holds it alone, or how many of the largest it takes. Change the context, users or GPU in the calculator
vLLM reports that an eight-GPU AMD Instinct MI355X system serving Kimi K3 reaches about 7,567 total tokens/s per GPU on the AgentX coding-agent benchmark. That is 24.4% more than NVIDIA HGX B300 (6,083) at 50 tokens/s/user P90. Both GPUs have 288 GB HBM3E.
The biggest gains came from cache handling. DCP8 splits the MLA KV history across GPUs, cutting a 142K-token conversation from about 1.96 GB to 245 MB per GPU. LLMScale estimates 1458.6 GiB at INT4 for one user at 8K context.
FP16 KV cache and vLLM's overhead on each device at its defaults. “Fits on”: the smallest device that holds it alone, or how many of the largest it takes. Change the context, users or GPU in the calculator
A r/LocalLLaMA recap lists four open-weight releases in 72 hours: Meta's Muse Spark Open (816B total, 42B active), Reflection's Beam (501B, 23B active), DeepSeek V4.1 Flash 1017 (510GB repo) and NVIDIA's Nemotron 3.5 Bolt (74B, 9B active).
Only Bolt targets a 24GB GPU: quantization-aware training to 2 bit puts weights at 22GB plus a 1.7GB vision projector, leaving little room for KV cache. Beam's smallest official checkpoint is 294GB in NVFP4. Sizes are from the post, not LLMScale estimates.
LLMScale