Infrastructure
29 stories.
llama.cpp adds Q1_0, Q2_0 quant support for Hexagon NPUs
Snapdragon devices get another notch of compression for on-device inference
llama.cpp adds in-app model discovery and download resume
A built-in Hub browser, resumable downloads, and an HTML sanitization fix
llama.cpp patches Hexagon NPU attention bug in b11696
A single fix for Qualcomm's HMX flash-attention sinks ships in the latest build
llama.cpp b11690 trims memory copies in speculative decoding cache
A single-line fix, but it is the kind that adds up at scale
Meta details how it made AMD GPUs production-ready for training
Porting recommendation models from Nvidia to AMD took more than a driver swap
llama.cpp v0.7.0 adds MoE GPU cache, new default server port
A new expert cache and port change reshape how MoE models get served
llama.cpp adds BF16 flash attention kernel for Metal backend
A small patch note for a release that still runs on every Apple GPU
AWS details MoE kernel fixes for SageMaker, upstreams them to vLLM
Three upstreamed vLLM fixes squeeze more throughput from Hopper GPUs
llama.cpp adds image-embedding support for Qwen3.8 Flash Next
A single pull request, the release notes offer little else
Study finds weight order matters more than value in LLM compression
Researchers compress models to 16–64 shared values and find rank is king
llama.cpp adds runtime support for Prism Bonsai 2 27B
A pre-release build lets the open-source engine load the Hadamard-based model
llama.cpp patches server bug that corrupted busy slot's context
A slot-pinning bug could silently swap context mid-generation
Paper: KV-cache tool gives LLMs a 50M-token memory on disk
galahad-kv reloads stored key-value state instead of recomputing it, cutting cost on one H100.
llama.cpp tunes CUDA kernel for Gated DeltaNet state updates
Release b11494 defaults GDN state updates to four columns per warp, a CUDA scheduling change in PR #30087.
llama.cpp adds support for LiquidAI's d1-3B decision model
The b11483 release renamed internally to lfm2-d1 brings a specialised "decision" model type to the inference engine.
Quantization's effect on agent tool-failure recovery flips with eval design
A single-author study finds 8-bit vs 4-bit comparisons reverse depending on prompt, task set and parsing strictness.
llama.cpp extends Metal fast-path matmul to more quant types
The b11476 release widens a row-batched kernel to BF16, Q2_K, MXFP4 and IQ types.
llama.cpp adds BF16 support for XIELU on CUDA
Release b11457 closes a gap in the CUDA kernel, letting XIELU activations run in bfloat16 on Nvidia GPUs.
llama.cpp adds support for K2 Horizon dense and MoVA models
Release b11454 brings a new converter, tokenizer and chat template fixes for the K2 Horizon model family.
llama.cpp patches data race in shared k-pool sequences
Release b11435 fixes a CPU-backend race affecting hybrid models with shared sequence caches.
llama.cpp v0.6.0 adds new batch API and five decision models
A server endpoint now runs dedicated pipelines for decision-style models.
New method quantizes Llama 3.2 11B Vision to 2.7 bits for phones
Researchers compress an 11B-parameter vision-language model to 3.7GB for Arm CPUs without needing the original training setup.
llama.cpp's WebGPU backend adds quantised matrix-vector support
Release b11406 extends WebGPU's MMVQ path to six more quantisation formats, including MXFP4.
vLLM 0.31.0 adds fast-restart weight caching and new DeepSeek optimisations
The release adds a preload daemon that keeps quantized weights resident across restarts, plus deep DeepSeek-V4.1-Flash tuning.
New method cuts memory for hybrid-attention model states by 69%
DAMP quantises recurrent states in Gated DeltaNet and Kimi Delta Attention. Plain INT4 wrecks their accuracy; this doesn't.
Researchers make layer-skipping LLMs actually serve faster
vSkipper plugs dynamic layer skippers into SGLang, turning skipped compute into real gains.
Study: TPU speed gains for AlphaFold2 vanish without proper batching
JAX's default path leaves seven of eight TPU chips idle. That erases the hardware's cost edge over a GPU.
Study maps when KV cache, not model weights, bottlenecks LLM inference
A paper derives exact context lengths where cache traffic overtakes weight traffic on H100, B200 and MI300X.
Hugging Face rewrote tokenizers to stop the CPU starving the GPU
Tokenisation was once rounding error. It is not any more.