Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs

TL;DR - This paper characterizes when external KV caching improves vLLM inference and introduces py-kvcache, an NVMe-backed connector that overlaps asynchronous disk reads with computation. It shows that caching should be enabled selectively based on workload, hardware, and prefix length rather than raw storage bandwidth alone.

• At 80K tokens, py-kvcache loads KV states from disk 2.0× faster than LMCache; scheduler-aware preloading contributes a 1.34× speedup.
• With GPU, CPU, and disk caching enabled, it is 1.23× faster than LMCache and within roughly 4% of native vLLM KV Offload.
• Performance depends on transfer granularity, staging-memory use, and scheduling, with benefits extending to irregular prefix chains and multi-turn workloads.
• Trace replays improved time to first token on a weaker GPU, but not on an H100 where requests were often below the caching break-even point and GPU memory retained enough prefixes.