vToken: Token-Level Virtualization for Reclaimable KV Caches
Ranking
Overall
80
Content
100
Popularity
34
Observed public metrics from 1 member.
Merged summary
TL;DR - vToken virtualizes KV-cache management at token granularity, enabling serving systems to reclaim memory stranded within fixed-size blocks. Its vLLM implementation improves memory utilization, throughput, and concurrency without changing PagedAttention kernels.
- Decouples logical token liveness from physical placement using token-table indirection.
- Asynchronously repacks live tokens while preserving CUDA Graph compatibility.
- Reduces retained KV blocks per request by 27.2%–72.3% and improves SLA-constrained throughput by up to 1.37×.
- Supports up to 2× more concurrency under constrained active-KV budgets and reduces policy integration from 500+ lines to under 50.
Sources (1)
vToken: Token-Level Virtualization for Reclaimable KV Caches
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - vToken virtualizes KV-cache management at token granularity, enabling serving systems to reclaim memory stranded within fixed-size blocks. Its vLLM implementation improves memory utilization, throughput, and concurrency without changing PagedAttention kernels.
- Decouples logical token liveness from physical placement using token-table indirection.
- Asynchronously repacks live tokens while preserving CUDA Graph compatibility.
- Reduces retained KV blocks per request by 27.2%–72.3% and improves SLA-constrained throughput by up to 1.37×.
- Supports up to 2× more concurrency under constrained active-KV budgets and reduces policy integration from 500+ lines to under 50.