Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs
Ranking
Overall
78
Content
95
Popularity
37
Observed public metrics from 1 member.
Merged summary
TL;DR - This paper characterizes when external KV caching improves vLLM inference and introduces py-kvcache, an NVMe-backed connector that overlaps asynchronous disk reads with computation. It shows that caching should be enabled selectively based on workload, hardware, and prefix length rather than raw storage bandwidth alone.
- At 80K tokens, py-kvcache loads KV states from disk 2.0× faster than LMCache; scheduler-aware preloading contributes a 1.34× speedup.
- With GPU, CPU, and disk caching enabled, it is 1.23× faster than LMCache and within roughly 4% of native vLLM KV Offload.
- Performance depends on transfer granularity, staging-memory use, and scheduling, with benefits extending to irregular prefix chains and multi-turn workloads.
- Trace replays improved time to first token on a weaker GPU, but not on an H100 where requests were often below the caching break-even point and GPU memory retained enough prefixes.
Sources (1)
Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - This paper characterizes when external KV caching improves vLLM inference and introduces py-kvcache, an NVMe-backed connector that overlaps asynchronous disk reads with computation. It shows that caching should be enabled selectively based on workload, hardware, and prefix length rather than raw storage bandwidth alone.
- At 80K tokens, py-kvcache loads KV states from disk 2.0× faster than LMCache; scheduler-aware preloading contributes a 1.34× speedup.
- With GPU, CPU, and disk caching enabled, it is 1.23× faster than LMCache and within roughly 4% of native vLLM KV Offload.
- Performance depends on transfer granularity, staging-memory use, and scheduling, with benefits extending to irregular prefix chains and multi-turn workloads.
- Trace replays improved time to first token on a weaker GPU, but not on an H100 where requests were often below the caching break-even point and GPU memory retained enough prefixes.