llama.cpp fixes a silent NEON fp32-to-fp16 bug on ARM CPUs
A hardware conversion instruction sat unused while software quietly took over
The facts
- Release b11761 of llama.cpp merges PR #30505, fixing ggml-cpu's NEON fp32->fp16 conversion selection
- The __ARM_NEON branch defined both GGML_CPU_COMPUTE macros but only selected FP16_TO_FP32, leaving FP32_TO_FP16 falling through to a slower portable software conversion
- The NEON hardware routine neon_compute_fp32_to_fp16, a single fcvt instruction, was defined but never referenced
- The patch adds a compile-time #error pair that blocks any branch from defining a COMPUTE macro without selecting it, and updates a stale comment that only named NEON and POWER9
The latest llama.cpp release, tagged b11761, fixes a bug in the project's CPU backend. ARM's NEON vector instructions were not actually being used for one direction of floating-point conversion. The __ARM_NEON code branch defined both GGML_CPU_COMPUTE macros, which select hardware-accelerated routines for converting between 32-bit and 16-bit floats. But only FP16_TO_FP32 was ever wired up.
The FP32_TO_FP16 path silently fell through to a portable software implementation instead of the NEON hardware routine sitting right next to it. That routine, called neon_compute_fp32_to_fp16, is a single fcvt instruction that nothing in the code ever called.
The release notes explain why this went unnoticed for so long. The software fallback was correct, just slower, so there was no crash or wrong output to flag the problem. The only symptom was a hardware instruction that existed in the codebase but was never reached. POWER9_VECTOR and the RISC-V __riscv_zfhmin paths, by contrast, selected both macros correctly. That made the NEON branch an outlier rather than the pattern.
To stop the same mistake recurring elsewhere, the patch adds a pair of compile-time #error checks. These refuse to build any branch that defines a COMPUTE macro without also selecting it. It leaves one carve-out: the FP16_TO_FP32 direction keeps a !defined(F16C) exception. An earlier change, PR #30343, deliberately routes that conversion through a lookup table instead of hardware on certain x86 builds. The commit also replaces the comment above the fallback, since it previously singled out NEON and POWER9 by name even though the selection logic now happens inside each branch itself.
This is a small fix with an outsized lesson for anyone running quantised inference on ARM hardware. That includes Apple Silicon Macs and Snapdragon-based Linux and Android builds listed among this release's binaries. Silent fallbacks that produce correct results but skip the fast path are exactly the kind of regression that benchmarks catch late and code review catches never. That is the real reason llama.cpp added a compiler guard rather than just the fix itself.
Why it matters
Anyone benchmarking llama.cpp CPU inference on ARM should re-test fp32-to-fp16 conversion throughput against this build. The bigger value is the new #error guard, which should catch similar silent fallbacks across other architectures before they ship unnoticed.
Primary source
llama.cpp releases — github.comWritten from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.