GraniKV: Asymmetric Granularity KV-Cache Paging for Multi-Agent Systems with Long Shared Prefix
Ranking
Overall
79
Content
95
Popularity
41
Observed public metrics from 1 member.
Merged summary
TL;DR - GraniKV improves multi-agent LLM serving by storing long shared prefixes contiguously while allocating request-specific suffixes at token granularity. It achieves up to 2.16× higher output-token throughput than a production paged-serving baseline.
- Uses separate HOT and COLD pools for shared prefixes and per-request suffixes.
- Dispatches each decoding step across dual backends based on compute, memory, or communication bottlenecks.
- Delivers 1.57–2.16× throughput gains across tested Llama and Qwen configurations.
- Sustains 1.95× throughput with heterogeneous prompts, where batch-global cascade attention falls to baseline performance.
Sources (1)
GraniKV: Asymmetric Granularity KV-Cache Paging for Multi-Agent Systems with Long Shared Prefix
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - GraniKV improves multi-agent LLM serving by storing long shared prefixes contiguously while allocating request-specific suffixes at token granularity. It achieves up to 2.16× higher output-token throughput than a production paged-serving baseline.
- Uses separate HOT and COLD pools for shared prefixes and per-request suffixes.
- Dispatches each decoding step across dual backends based on compute, memory, or communication bottlenecks.
- Delivers 1.57–2.16× throughput gains across tested Llama and Qwen configurations.
- Sustains 1.95× throughput with heterogeneous prompts, where batch-global cascade attention falls to baseline performance.