AWS adds bi-directional KV transfer to vLLM for agentic chat

Decode nodes now hand KV cache back to prefill, cutting repeat computation

Amazon Web Services has contributed a bi-directional key-value cache transfer mechanism to vLLM. It targets multi-turn agentic workloads where clients resend full conversation history on every turn.

In prefill-decode disaggregated serving, a prefill node builds the KV cache for a prompt and a decode node generates the response. Until now, KV only flowed one way, from prefill to decode. That meant decode nodes held onto cache for prompts and generated tokens that prefill nodes could never retrieve. Prefill was forced to recompute KV projections for prior turns, raising time to first token (TTFT) as conversations grew.

The new mechanism lets decode nodes return previously computed KV blocks to prefill nodes on later turns. This is coordinated by what AWS calls a KV-cache-aware router, which tracks conversation_id to kv_transfer_params mappings and reads block-location metadata from the response stream rather than querying decode separately. A decoder_kv_blocks_ttl setting, defaulting to 480 seconds, keeps decode-side blocks alive long enough to be reused on a subsequent turn. A kv_recompute_threshold, defaulting to 64 tokens, decides whether a short reusable prefix is cheaper to recompute locally or fetch over the network.

AWS tested the feature on a single-prefill, single-decode deployment using EC2 P5en GPU instances linked by its EFA networking. Prefix caching was disabled on the prefill node to simulate cache misses. With the feature off, AWS reports TTFT for Qwen3-32B climbed from 158 milliseconds at a 2,000-token prompt to 778 milliseconds at 24,000 tokens. GLM-5-FP8 rose from 500 milliseconds to 2,334 milliseconds across 2,000 to 20,000 tokens. With the feature on, Qwen3-32B rose only to 460 milliseconds and GLM-5-FP8 to 702 milliseconds over the same ranges. AWS calculates this as roughly 1.6x and 3.3x reductions respectively at the longest prompts, since the added KV-transfer time of 29 to 130 milliseconds was outweighed by the recompute it avoided.

The feature was originally built and validated on AWS Trainium instance clusters connected with EFA. AWS says it was then contributed as an accelerator-agnostic capability, and the company says the GPU results show the same benefits carry over to Nvidia hardware.

Written from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.