AWS contributes bi-directional KV cache transfer to vLLM, reporting up to 3.3x lower TTFT on multi-turn workloads

AWS contributed an extension to vLLM's NIXL KV connector so that prefill nodes can pull previously computed KV cache blocks back from decode nodes on later turns of a conversation, instead of recomputing them. A KV-cache-aware router maps each conversation_id to the right decode node. Decode nodes keep the blocks for a configurable time (decoder_kv_blocks_ttl, default 480 seconds), and a kv_recompute_threshold (default 64 tokens) decides whether to pull or recompute. In AWS's own tests on a 1P1D EC2 P5en setup, time to first token (TTFT) fell by up to 1.6x for Qwen3-32B and up to 3.3x for GLM-5-FP8. AWS says it built the feature on Trainium with EFA and contributed it as accelerator-agnostic.

This matters mainly to teams that already run disaggregated prefill and decode serving in vLLM for multi-turn or agentic workloads, where re-processing a growing conversation wastes accelerator time. If the gains hold, they could cut latency or serve more conversations on the same hardware. For everyone else, including most users of hosted APIs or single-node vLLM, the effect is small or indirect. The 480 second retention also means decode nodes hold memory for idle conversations, so the cost trade-off depends on how your traffic behaves.

The 1.6x and 3.3x figures are "up to" results from AWS's own benchmarks on one small 1P1D layout with two models, so they are best cases and not independent proof of typical gains. The "accelerator-agnostic" claim is also unverified, since it was built on Trainium and EFA and tested on P5en. The extraction found no link to the vLLM pull request, so it is unclear whether the code is merged or still under review.

1. 01Whether the vLLM pull request or merged code appears with a release number, and whether maintainers raise concerns in review about memory use or correctness.
2. 02Independent benchmarks on other hardware (NVIDIA-only clusters, AMD) and on larger prefill/decode ratios than 1P1D, ideally reporting throughput and tail latency (p95 or p99), not just TTFT.
3. 03Adoption signals: other serving stacks or cloud providers using conversation-aware KV routing, and users reporting the benefit or memory cost of the 480 second retention in production.