On-device AI
7 stories. An explainer is being written.
On-device AI in the news
·llama.cpp releases
llama.cpp adds Q1_0, Q2_0 quant support for Hexagon NPUs
Snapdragon devices get another notch of compression for on-device inference
·llama.cpp releases
llama.cpp patches Hexagon NPU attention bug in b11696
A single fix for Qualcomm's HMX flash-attention sinks ships in the latest build
·llama.cpp releases
llama.cpp b11690 trims memory copies in speculative decoding cache
A single-line fix, but it is the kind that adds up at scale
·llama.cpp releases
llama.cpp adds BF16 flash attention kernel for Metal backend
A small patch note for a release that still runs on every Apple GPU
·arXiv cs.CL
Study finds LLaMA-2 safety hinges on 0.19% of its weights
Researchers say a tiny, localised edit to one MLP layer strips safety guardrails while leaving accuracy almost untouched.
·Google DeepMind
Google releases EmbeddingGemma 2, a 740M multimodal embedding model
Apache 2.0 model unifies text, image, audio and video for on-device search and RAG
·arXiv cs.LG
New method quantizes Llama 3.2 11B Vision to 2.7 bits for phones
Researchers compress an 11B-parameter vision-language model to 3.7GB for Arm CPUs without needing the original training setup.