llama.cpp extends Metal fast-path matmul to more quant types
The b11476 release widens a row-batched kernel to BF16, Q2_K, MXFP4 and IQ types.
The facts
- Release b11476 adds a generic few-row MMA (matrix multiply-accumulate) kernel for any type with a 16-weight dequantizer, covering BF16, Q1_0, Q2_0, MXFP4, Q2_K, Q3_K, TQ2_0 and IQ types (PR #30065)
- Each type switches to the new kernel at the row count where it starts beating the existing kernel on an M3 Ultra: 5 rows for TQ2_0, 4 for BF16, 3 for MXFP4/Q2_0/Q2_K/IQ4_NL, and 2 for the rest
- test-backend-ops perf benchmarks (MUL_MAT, m=4096, k=14336) on M3 Ultra show the new kernel running at 0.23 to 0.98 of master's time from the threshold row count up to 8 rows, 0.24 to 0.33 at 9-16 rows, and roughly parity (0.99-1.01) at 1 and 512 rows
- Build includes binaries for macOS, Linux (CPU, Vulkan, CUDA 12/13, ROCm, OpenVINO, SYCL), Android, Windows and openEuler
llama.cpp's latest build, tagged b11476, extends a matrix-multiply kernel on Apple's Metal backend. It now covers far more quantisation formats than before.
The change comes from pull request #30065. It generalises a 'few-row' MMA (matrix multiply-accumulate) kernel that once served a narrower set of types. The kernel now also handles BF16, Q1_0, Q2_0, MXFP4, Q2_K, Q3_K, TQ2_0 and the IQ family. Any format with a 16-weight dequantizer qualifies.
The kernel only kicks in once a matrix multiply has enough rows to make it worthwhile. That threshold varies by type. On an M3 Ultra, TQ2_0 needs 5 rows before the new path wins, and BF16 needs 4. MXFP4, Q2_0, Q2_K and IQ4_NL need 3 rows, and the remaining types need just 2. Below those thresholds, the existing kernels still win.
The project's own test-backend-ops benchmark, run on the MUL_MAT operation with m=4096 and k=14336, puts the gains in context. From the row threshold up to 8 rows, the new kernel runs in 0.23 to 0.98 of the time master took. At 9 to 16 rows it drops to 0.24 to 0.33. At the extremes — 1 row or 512 rows — it is essentially a wash, at 0.99 to 1.01.
The release ships binaries across macOS, Linux, Android, Windows and openEuler. CUDA 12 and 13, ROCm, Vulkan, SYCL and OpenVINO variants ship alongside the Metal build.
Why it matters
For anyone self-hosting quantised models on Apple Silicon, this narrows the gap between low-batch and mid-batch inference. Workloads with a handful of concurrent sequences on formats like Q2_K or MXFP4 now get a three- to four-fold speed-up on matrix multiplies. No change to weights or configuration is needed. Kernel selection happens automatically by row count.
Primary source
llama.cpp releases — github.comWritten from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.