Paper: KV-cache tool gives LLMs a 50M-token memory on disk

galahad-kv reloads stored key-value state instead of recomputing it, cutting cost on one H100.

A new arXiv paper from Sietse Schelpe tests a memory layer called galahad-kv. It stores a large language model's key-value (KV) state — the internal representation it builds while processing a prompt — to encrypted local NVMe disk. Later it reloads that state byte-exact, instead of recomputing it.

The author ran it across 50,000,000 tokens of real public text. The test used vLLM on a single NVIDIA H100 GPU, with Gemma 4 12B and Gemma 4 31B.

Every block the paper probed, 100 out of 100, came back from the encrypted store with zero recompute. That held at depths ranging from the start of the stream to the full 50 million tokens, on both model sizes. Loading a stored block was 2.8x to 4.3x faster than recomputing it, the paper reports, and used 8.8x to 12.3x less GPU energy. GPU memory stayed flat across the entire run.

Asked about facts planted millions of tokens earlier, the 12B model answered correctly 82 times out of 100. The 31B model answered correctly 98 times out of 100. Neither model fabricated an answer, according to the paper.

The author is explicit about the limits. This is reuse of previously stored state, not a genuinely wider attention window: only one block loads at a time, and answer quality still depends on the underlying model. Writing the memory in the first place is a one-time cost. The resulting store runs to terabytes of local NVMe disk.

The paper includes a single-GPU reproduction built on public software, with the package released under a free licence. It describes a test protocol designed to resist common ways of gaming long-context benchmarks.

Written from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.