Study maps when KV cache, not model weights, bottlenecks LLM inference
A paper derives exact context lengths where cache traffic overtakes weight traffic on H100, B200 and MI300X.
The facts
- For Llama-3-70B in BF16, the 140GB weight footprint exceeds a single accelerator's 80GB of HBM, and one 128k-token sequence adds 42GB of KV cache
- The paper derives closed-form arithmetic intensity as a function of context length, parameterized for NVIDIA H100, NVIDIA B200 and AMD MI300X, including hardware-specific crossover lengths where KV traffic overtakes weight traffic
- It classifies existing KV cache methods into five domains -- quantization, token eviction, KV paging, prefix caching, and heterogeneous tiering -- and evaluates one method per domain at 128k context under a single protocol
- Findings: below the hardware-specific crossover point, KV compression yields negligible speedup; beyond it, quality degrades sharply below 4-bit quantization and turns discontinuous for eviction on position-sensitive tasks; tiering shifts the bottleneck to PCIe or NVLink interconnects rather than HBM bandwidth
A new systematization-of-knowledge paper argues that KV cache compression research has been comparing apples to oranges. Papers report speedups using inconsistent workloads, hardware and quality metrics. Cross-paper comparison is, as a result, close to impossible.
The author, Tejinder Singh, instead derives the problem analytically. He builds a closed-form model of arithmetic intensity -- the ratio of compute to memory traffic -- as a function of context length. The model is parameterized for three specific accelerators: NVIDIA's H100 and B200, and AMD's MI300X.
The concrete numbers show why this matters. Llama-3-70B in BF16 (16-bit) precision has a 140GB weight footprint, already too large for a single accelerator's 80GB of HBM (high-bandwidth memory). A single 128,000-token sequence adds another 42GB of KV cache on top. The paper's central claim is a three-regime structure: below a hardware-specific crossover length, weight traffic dominates and KV compression buys almost nothing. Beyond that crossover, KV traffic dominates, and every compression technique trades quality for bandwidth savings that approach a theoretical roofline bound.
Singh sorts the existing literature into five domains: quantization, token eviction, KV paging, prefix caching and heterogeneous tiering. He evaluates one representative method from each at 128k context under a single testing protocol. Paging and prefix caching turn out to be lossless, but they solve a capacity problem, not a bandwidth one. Quantization and eviction cut bandwidth directly, but degradation accelerates sharply below 4-bit precision. Eviction's quality loss becomes discontinuous -- a cliff rather than a slope -- on tasks sensitive to token position. Tiering, meanwhile, doesn't eliminate the bottleneck so much as relocate it: once KV state moves off a single die, the constraint becomes PCIe or NVLink interconnect bandwidth rather than HBM.
The paper closes with design rules for choosing a compression domain given a target hardware platform, context length and acceptable quality loss.
Why it matters
For teams serving long-context models, the practical upshot is stark. Below the crossover length for their specific accelerator, KV cache tricks are wasted engineering effort, since the bottleneck is still weight bandwidth. Above it, the choice of compression domain is a genuine quality-versus-bandwidth trade governed by hardware topology, not a free lunch. Anyone quantizing KV state below 4 bits should expect a steep quality cliff, not a gradual slope.
Primary source
arXiv cs.LG — arxiv.orgWritten from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.