AWS adds push-based KV transfer to vLLM for faster inference
A new connector cuts time-to-first-token by moving setup work off the critical path
The facts
- AWS built a push-based NIXL connector for vLLM's disaggregated prefill-decode inference, contributed as an accelerator-agnostic feature after validating it on AWS Trainium clusters
- In a benchmark on a 1P1D deployment using Amazon EC2 P5en instances with AWS EFA, push mode gave up to 3x better time-to-first-token and up to 30% better time-per-output-token versus pull mode, for the Qwen3-32B model
- At 8 queries per second and 16k input tokens, pull mode TTFT reached about 1,744 ms versus roughly 540 ms for push mode
- The push connector registers destination KV blocks on the decode instance in advance and uses a background thread on the prefill worker to prepare and submit the NIXL WRITE transfer, rather than requiring the decode instance to pull blocks after prefill completes
- Users enable it by configuring both prefill and decode servers to use NixlPushConnector instead of NixlConnector
AWS has published a push-based key-value (KV) transfer connector for vLLM, the open-source LLM serving engine. It targets disaggregated inference setups that split prompt processing, or prefill, from token generation, or decode, across separate instances.
The default NIXL connector in vLLM works in pull mode. The decode instance waits for the router to relay prefill metadata, then allocates blocks and issues a read to fetch the KV cache. All of that sits squarely on the path to the first generated token. AWS's new NixlPushConnector flips the sequence instead. The decode instance registers its destination blocks while prefill is still running. A background thread on the prefill worker handles the transfer setup and write once both sides are ready.
AWS tested the connector on Qwen3-32B, a large language model, in a single-prefill, single-decode deployment. The setup used Amazon EC2 P5en instances connected over AWS Elastic Fabric Adapter. AWS reports push mode delivered up to 3x better time-to-first-token and up to 30% better time-per-output-token than pull mode. The gap widened under load. At 8 queries per second with 16,000-token inputs, pull mode's TTFT reached about 1,744 milliseconds, against roughly 540 milliseconds for push mode. TPOT rose to about 10.4 milliseconds for pull, versus close to 7 milliseconds for push. At low request rates and short prompts, the two modes performed almost identically, AWS says. The divergence appears specifically as concurrency and context length grow.
The company says it built and validated the feature on AWS Trainium instance clusters before contributing it to vLLM as an accelerator-agnostic feature. It argues the published results from the GPU-based P5en cluster show the gains carry over across hardware. There is a tradeoff in routing flexibility. Pull mode lets the router pick a decode instance only after prefill finishes, which suits load-aware routing. Push mode commits to a decode target earlier so that setup can overlap with prefill. AWS notes results depend on workload and configuration. It provides a reproduction guide covering environment setup and benchmark collection for teams that want to verify the numbers on their own stack.
Why it matters
Anyone running disaggregated vLLM inference at scale on AWS now has a concrete lever for TTFT under load, not just a theoretical one. The catch is reduced routing flexibility. Push mode commits to a decode instance before prefill completes, trading quality-of-service control for latency.
Primary source
AWS Machine Learning — aws.amazon.comWritten from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.