This Week's Picks
Release
llama.cpp 1.0 Ships with a Stability Promise
Semver commitment lands alongside FP8 KV and an MoE expert cache.
- Why it matters:
- llama.cpp and ggml released 1.0 on Nov 13: llama_set_warmup and 35 deprecated functions plus defrag_thold are gone, the NextN speculative API moves from llama-ext.h into the public llama.h, OCP FP8 KV caches arrive (-ctk/-ctv f8_e4m3 with optional static per-tensor scales), and a host-memory MoE expert cache (moe_cache_size, moe_cache_cpu_share, LRFU replacement) lets small GPUs run large MoE models. New architectures include Kimi-K3.1-Pro, Ripple-65B-Preview, xing4_0, drama-3 TTS, and two vision models, bringing the total past 150. The team commits to bumping the major version for any incompatible public C API or ABI change. llama-server security defaults tighten: tools or MCP on a non-loopback address now require an API key (or explicit --allow-unauthenticated-tools), HF_TOKEN is restricted to huggingface.co, and DNS rebinding plus SSRF holes are fixed.
- Relevance to edge / on-device deployment:
- Directly relevant: FP8 KV saves memory for long context on-device, the expert cache helps small-VRAM devices run large MoE models, and the stricter server defaults affect every self-hoster.
Source: GitHub llama.cpp Releases
Release
DeepSeek at 58× Throughput on Vera Rubin NVL72
1.7M tok/s per GPU on NVL72; 387 tok/s per-user P90.
- Why it matters:
- The vLLM blog (Nov 12) reports DeepSeek-V4.1-Flash serving on NVIDIA's new Vera Rubin NVL72: 58× the throughput of GB300 NVL72 on the SemiAnalysis AgentX benchmark, reaching 1.7M tok/s per GPU; on the interactivity metric, per-user P90 hits 387 tok/s. It is official same-week adaptation data for new silicon, and another case of per-user speed replacing per-GPU throughput as the headline metric.
- Relevance to edge / on-device deployment:
- No edge angle per se, but the 387 tok/s per-user interactivity bar is becoming the consensus serving target and will shape speed expectations for on-device products too.
Source: vLLM Blog
Release
vLLM Lands Encoder Disaggregation (EPD)
Multimodal encoders become standalone services with cross-encoder caching.
- Why it matters:
- The vLLM blog (Nov 11) introduces encoder disaggregation: vision and audio encoders run as separate services decoupled from LLM decoding, with metadata-only multimodal requests, remote embedding transfer, cross-encoder caching, and lower-overhead EPD routing, aiming to take multimodal scaling concerns out of the decoding engine.
- Relevance to edge / on-device deployment:
- On-device multimodal devices can borrow the same split: keep the encoder resident and load the decoder on demand, easing deployment on small-memory hardware.
Source: vLLM Blog
Tool
UD V3.1 Extends to the Qwen3.8 Family
Qwen3.8-Flash-Next 1-bit hits 54.0% on DeepSWE.
- Why it matters:
- Unsloth's Dynamic V3.1 quantization (docs updated Nov 11) now covers Qwen3.8-27B, DeepSeek-V4.1-Flash-1017, and Qwen3.8-Flash-Next, claiming 10–15% higher accuracy and 50% fewer tokens than same-size alternatives. The Qwen3.8-Flash-Next 1-bit build scores 54.0% on DeepSWE v1.1 and 68.3% on Terminal Bench 2.1; DeepSeek-V4.1-Flash-1017 IQ1_S reaches 65.8% (DeepSWE) and 69.5% (TB). On last week's reproducibility challenge, the ISTA/GSQ-RCO camp has still not responded publicly as of this week.
- Relevance to edge / on-device deployment:
- Directly relevant: a 1-bit GGUF of Qwen3.8-Flash-Next makes a flagship-class agent model runnable on a single 12GB card, a concrete selection datapoint for local agent setups.
Source: Unsloth Docs
Paper
FlashOffloading: Host-Memory KV Offload
Full KV offload to host DRAM under PD; 1.38–1.91× throughput.
- Why it matters:
- Long contexts fill HBM and cap concurrency, while naive host-memory offloading adds management overhead. FlashOffloading is a phase-specialized architecture: prefill uses a four-buffer rolling HBM cache pipeline overlapping load, compute, and writeback; decode uses a hot cache with a reverse token-to-slot sparse index that adds zero extra miss-lookup cost, plus sparse-attention kernels overlapping H2D transfers. Deployed with GLM-5.2 and MTP3 speculative decoding on an Ascend 910C PD-disaggregated platform under a 30ms TPOT constraint, it improves prefill throughput by up to 1.38× and decode by up to 1.91×, turning 512K–1M-token contexts from 'does not fit' into servable.
- Relevance to edge / on-device deployment:
- Ascend is server silicon, but treating host memory as a KV extension layer overlapped with compute is isomorphic to unified-memory device management and worth borrowing for on-device long context.
Source: arXiv 2611.07542
This issue is curated from public sources; every link points to the original publisher. If information has changed after publication, the source page takes precedence.