GraniKV: Asymmetric Granularity KV-Cache Paging for Multi-Agent Systems with Long Shared Prefix

TL;DR - GraniKV improves multi-agent LLM serving by storing long shared prefixes contiguously while allocating request-specific suffixes at token granularity. It achieves up to 2.16× higher output-token throughput than a production paged-serving baseline.

• Uses separate HOT and COLD pools for shared prefixes and per-request suffixes.
• Dispatches each decoding step across dual backends based on compute, memory, or communication bottlenecks.
• Delivers 1.57–2.16× throughput gains across tested Llama and Qwen configurations.
• Sustains 1.95× throughput with heterogeneous prompts, where batch-global cascade attention falls to baseline performance.