Ollama v0.40.0 makes MLX the default runtime on Apple Silicon
Ollama v0.40.0 makes MLX the default runtime on Apple Silicon. Model architectures that MLX supports now run on it automatically. Listed models include qwen3.8, gemma4, qwen3.6 and qwen3.5, plus the embedding model embeddinggemma-2. More models will be enabled after testing.
On Macs this swaps the inference backend with no configuration change. The release notes give no speed or memory figures, so re-measure throughput and memory use on your own hardware.
Mistral released a preview of Mistral Large 4 via its API: an MoE model with 1T total and 49B active parameters, trained on 3,800 NVIDIA Grace Blackwell GPUs. Open weights are promised for the end of this month. It scores 38 on Artificial Analysis, up from 9 for Mistral Large 3.
For self-hosting, per-token compute follows the 49B active parameters, but all 1T weights must be held in memory, so VRAM rather than compute sets the hardware floor. No memory estimate yet: the model config is not public.
inclusionAI has released Ling-3.1-flash, a hybrid-linear MoE with 560B total and 25B active parameters and a stated 1M-token context. Its 72 layers split into 63 KDA and 9 Gated MLA layers, so only a small share of layers hold a conventional KV cache.
LLMScale estimates for one user at 8K context: 537.3 GiB at FP8 (4× B200) and 287.8 GiB at INT4 (fits a 512GB M3 Ultra). The card's SGLang recipe uses 8× H200 or 4× GB300. vLLM needs a dedicated branch.
#moe #kv-cache #long-context #sglang
Ling-3.1-flash · 572.12B parameters · 8k context, one user
FP16 KV cache and vLLM's overhead on each device at its defaults. “Fits on”: the smallest device that holds it alone, or how many of the largest it takes. Change the context, users or GPU in the calculator
Liquid AI released LFM2.5-24B-A2B, an open-weight reasoning MoE with 24B parameters, about 2B active per token and 64 experts with top-4 routing. Context grows from 32,768 to 131,072 tokens, on a hybrid of gated short convolutions and GQA. Liquid reports 184 tokens/s decode on an Apple M5 Max.
LLMScale estimates for one user at 8,192 tokens of context: 45.97 GiB in BF16 (A100 80GB), 23.89 GiB in FP8 (RTX 5090) and 13.54 GiB in INT4 (RTX 4060 Ti 16GB).
#moe #on-device #long-context #quantization
LFM2.5-24B-A2B · 23.97B parameters · 8k context, one user
FP16 KV cache and vLLM's overhead on each device at its defaults. “Fits on”: the smallest device that holds it alone, or how many of the largest it takes. Change the context, users or GPU in the calculator
NVIDIA has released Nemotron 3.5 Super VL, an open multimodal MoE model with 120B parameters, 12B active per token. One model covers text, image, document and video input. The checkpoint uses NVFP4: 4-bit weights and activations on Blackwell, 4-bit weights with 16-bit activations on other GPUs.
All 120B weights must fit in memory even though only 12B run per token. NVIDIA says LiLiCorr speculative decoding speeds generation up to 2.3x on DGX Spark.
vLLM has day-0 support for Meta's Muse Spark Open, an open-weight MoE with 816B total and 42B active parameters and a 1M-token context. Weights come in BF16 and NVFP4, plus MXFP4 from AMD.
46 of its 62 layers use a 2048-token sliding window, so KV cache at 1M tokens is just over a quarter of all-full attention. The example serves NVFP4 at TP8 with expert parallelism and a DSpark draft model for speculative decoding.
LLMScale