New method cuts memory for hybrid-attention model states by 69%
DAMP quantises recurrent states in Gated DeltaNet and Kimi Delta Attention. Plain INT4 wrecks their accuracy; this doesn't.
The facts
- Researchers find uniform INT8/FP8 quantization degrades complex reasoning accuracy, while INT4 and NVFP4 collapse it to near zero, in hybrid models using Gated DeltaNet (GDN) or Kimi Delta Attention (KDA)
- DAMP keeps high-risk key channels (identified via quantization error plus decay-based error retention) in FP16 and stores the rest in INT8, under a fixed storage budget
- Tested on Qwen3.6-35B, Kimi-Linear-48B and Kimi-K3 across six reasoning and code benchmarks, DAMP holds average accuracy close to FP32 at 9.9 bits per state value
- In SGLang, DAMP cuts recurrent-state storage by 69.1%, speeds the recurrent-state update kernel by up to 2.59x, and lowers full-model time per output token by up to 19.0%
Hybrid models mix standard softmax attention with linear-attention variants like Gated DeltaNet or Kimi Delta Attention. They promise cheaper long-context inference. Their recurrent states stay a fixed size instead of growing with the key-value cache.
But those states are usually kept in FP32. They are large and bandwidth-hungry enough to blunt the savings. A new paper from authors at Qwen, Kimi and collaborating institutions shows why naive quantisation doesn't fix this. INT8 and FP8 quietly degrade complex reasoning. INT4 or NVFP4 wreck it entirely.
The authors trace the failure to outliers concentrated in specific key channels and value dimensions within GDN and KDA states. Learned decay rates also determine how much quantisation error survives over time. They report that largely the same heads and key channels show slow decay across different tasks. That means the risky channels can be identified once, offline, rather than per task. Their method, DAMP, uses that analysis to pick high-risk channels and keep them in FP16, compressing the rest to INT8 under a fixed total storage budget.
On Qwen3.6-35B, Kimi-Linear-48B and Kimi-K3, across six reasoning and code generation benchmarks, the authors report DAMP holds average accuracy close to the FP32 baseline. It does so at 9.9 bits per state value, under half the bits of the original format. Implemented in the SGLang serving framework, they report a 69.1% reduction in recurrent-state storage. They also report up to a 2.59x speedup on the recurrent-state update kernel, and up to a 19.0% cut in full-model time per output token.
No code release or licence is mentioned in the paper. The gains come from the authors' own SGLang implementation, not an independent benchmark.
Why it matters
Anyone serving hybrid-attention models like Kimi-Linear or Qwen's linear-attention variants gets a concrete lever for cutting GPU memory and per-token latency without retraining. That holds provided they can reproduce the channel-selection step. The finding that naive INT4 quantisation collapses reasoning accuracy is also a warning: standard KV-cache compression tricks don't just transfer to recurrent-state models.
Primary source
arXiv cs.LG — arxiv.orgWritten from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.