llama.cpp adds Q1_0, Q2_0 quant support for Hexagon NPUs
Snapdragon devices get another notch of compression for on-device inference
The facts
- Release b11716 of llama.cpp adds support for Q1_0 and Q2_0 quantisation formats on Qualcomm Hexagon NPUs (pull request #30122)
- Changes include enabling Q1_0 weights in the 2D HMX MUL_MAT path and adding Q2_0 support with cleanup of stale asserts and special cases
- Work includes fixes to the MUL_MAT_NX tiling assumption, since Q1/Q2 formats use non-square tiles rather than the square tiling assumed elsewhere
- Multiple commits optimise Q1_0 and Q2_0 vec_dots and layout to reduce unpack and vrmpy overhead
- Co-authored by Max Krasnyansky of Qualcomm; builds are available for Linux arm64 and Android arm64 targeting Snapdragon CPU, Adreno GPU and Hexagon NPU
The latest llama.cpp pre-release, tagged b11716, adds support for two low-bit quantisation formats on Qualcomm's Hexagon neural processing unit. Q1_0 and Q2_0 compress model weights to roughly one or two bits each. The change, merged via pull request #30122, lets the Hexagon backend run these compressed models through the existing 2D HMX matrix-multiply path used for tiled tensor formats.
Getting there required more than flipping a flag. Q1_0 and Q2_0 weights use non-square tiles, which broke an assumption baked into llama.cpp's MUL_MAT_NX routine: that tiling would always be square. The patch generalises that path instead of special-casing it. Several follow-up commits strip out stale asserts and special cases left over from earlier work. They also tighten the vec_dot routines and memory layout for both formats, cutting the overhead from unpacking and vector multiply-accumulate (vrmpy) operations.
The work was co-authored by Max Krasnyansky of Qualcomm. That underscores the chipmaker's continued hand in llama.cpp's Hexagon backend. Build artefacts for this release include a dedicated Linux arm64 target and an Android arm64 target. Both cover Snapdragon CPU, Adreno GPU and Hexagon NPU execution. The usual Linux, Windows, macOS and openEuler builds remain too, across CUDA, Vulkan, ROCm, SYCL and OpenVINO backends.
Why it matters
Sub-2-bit quantisation on Hexagon NPUs means developers can push larger models onto Snapdragon phones and laptops without falling back to the CPU or GPU path. Anyone targeting on-device inference on Snapdragon hardware should re-test memory and latency against this build before locking in a quant format.
Primary source
llama.cpp releases — github.comWritten from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.