Study finds mobile NPUs waste most of their edge on LLMs
Framework choice, not chip design, decides who wins the phone AI race
The facts
- Researchers built PowerBench, a profiling tool giving the first backend-specific energy attribution for mobile LLM inference, tested across five frameworks and three hardware backends (CPU, GPU, NPU)
- Framework-induced performance gaps on NPUs reach up to 10x, caused by differences in offloading strategies, underutilised operators and activation quantization
- NPUs excel at compute-bound prefilling while CPUs win at memory-bound decoding, a 'phase split' caused by NPUs favouring large, fixed-shape workloads unsuited to decoding's small, dynamic kernels
- Tuning CPU frequency alone cut energy use by up to 39.7%, and an energy-oriented configuration could cut NPU backend energy consumption by up to 53.2% across three datasets
- Paper accepted by ACM SIGMETRICS 2027, authored by Guanyu Cai and six co-authors, first submitted 6 July 2026 and revised 16 October 2026
Mobile NPUs are badly underused when running large language models, a new measurement study says. The software stack matters more than the silicon itself, according to the paper's authors.
Researchers behind the paper, titled 'Is Your NPU Ready for LLMs?', built a profiling tool called PowerBench. It measures throughput and energy use across five mainstream inference frameworks and three hardware backends: CPU, GPU and NPU. The authors say it is the first cross-layer study to attribute energy consumption to specific backends during mobile LLM inference, rather than measuring the whole device at once.
The headline finding: identical NPU hardware can perform up to ten times worse depending purely on which framework drives it. The authors trace this gap to differences in offloading strategy, operators that fail to use the hardware fully, and how activations are quantized.
The paper also documents what it calls a 'phase split'. NPUs suit the compute-heavy prefilling stage of inference, where a prompt is processed in one large batch. But CPUs beat every other backend at decoding, the memory-bound process of generating tokens one at a time. NPUs are built for large, fixed-shape workloads, the authors note, which conflicts with decoding's small, constantly changing kernels.
Beyond the hardware mismatch, the researchers say scheduling leaves significant performance on the table. Suboptimal thread configurations, uncoordinated NPU sleep latencies and CPU polling intervals waste energy that better tuning could recover. Adjusting CPU frequency alone cut energy consumption by up to 39.7%, the authors found. Applying an energy-oriented configuration across the full pipeline, they estimate, can cut NPU-backend energy use by up to 53.2% across three datasets. The paper has been accepted by ACM SIGMETRICS 2027.
The study implies that benchmarking a phone's NPU in isolation, as vendors typically do, tells a developer little about real LLM performance. Anyone building on-device inference should treat framework selection and backend scheduling as first-order engineering problems. Chip choice alone is not the answer.
Why it matters
The paper suggests on-device LLM performance is a software problem dressed up as a hardware one. Teams shipping mobile inference should benchmark their chosen framework against alternatives before assuming the NPU on their target chip is the bottleneck. Chip vendors' isolated NPU benchmarks may be telling buyers very little about real-world LLM throughput.
Primary source
arXiv cs.AI — arxiv.orgWritten from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.