llama.cpp adds BF16 flash attention kernel for Metal backend
A small patch note for a release that still runs on every Apple GPU
The facts
- Release b11645 of llama.cpp adds a BF16 variant of the tensor API flash attention kernel for the Metal backend (PR #30491)
- The change also moves tensor flash-attention kernels into a new file, kernels/fa_tensor.metal
- The release ships prebuilt binaries across macOS, iOS, Linux, Android, Windows and openEuler, covering CPU, CUDA 12/13, ROCm, Vulkan, SYCL, OpenVINO and Snapdragon NPU/GPU targets
- Tagged as a pre-release and signed with GitHub's verified commit signature
Llama.cpp's latest pre-release, tagged b11645, adds a BF16 variant of the tensor API flash attention kernel used by the project's Metal backend. Metal is the code that runs the inference engine on Apple Silicon GPUs. Flash attention computes the attention mechanism in transformer models without materialising the full attention matrix in memory. Running it in BF16, a 16-bit floating point format, trades a little numerical range for lower memory bandwidth and faster execution than full 32-bit precision. The pull request, numbered 30491, also reorganises the code by moving the tensor flash-attention kernels into a new file, kernels/fa_tensor.metal.
The release itself is modest in scope but wide in reach. Llama.cpp ships prebuilt binaries for this tag across macOS and iOS, and across several Linux distributions including Ubuntu on x64, arm64 and s390x. It also covers Android, Windows on x64 and arm64, and openEuler. Backend variants include CUDA 12 and 13, ROCm, Vulkan, SYCL, OpenVINO and Qualcomm Snapdragon's CPU, Adreno GPU and Hexagon NPU. The commit was signed with GitHub's verified signature system.
No benchmark figures accompany the change. The release notes give no indication of expected speedup or memory savings from the new kernel.
Why it matters
For anyone serving Llama-family or other GGUF models on Apple hardware, a BF16 flash attention path on Metal should lower memory bandwidth pressure during inference. It avoids the extra precision loss of lower-bit kernels. The change is narrow. But it is the kind of incremental backend work that keeps llama.cpp competitive against heavier, GPU-bound serving stacks.
Primary source
llama.cpp releases — github.comWritten from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.