vToken: Token-Level Virtualization for Reclaimable KV Caches

TL;DR - vToken virtualizes KV-cache management at token granularity, enabling serving systems to reclaim memory stranded within fixed-size blocks. Its vLLM implementation improves memory utilization, throughput, and concurrency without changing PagedAttention kernels.

• Decouples logical token liveness from physical placement using token-table indirection.
• Asynchronously repacks live tokens while preserving CUDA Graph compatibility.
• Reduces retained KV blocks per request by 27.2%–72.3% and improves SLA-constrained throughput by up to 1.37×.
• Supports up to 2× more concurrency under constrained active-KV budgets and reduces policy integration from 500+ lines to under 50.