CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving

TL;DR - CacheRoute periodically plans prefix-affinity routing so repeated LLM requests reach servers holding reusable KV caches without severely skewing load. On Llama-3.3-70B across 60 H100 GPUs, it delivered 2.3× the throughput of the strongest baseline under a 3.5-second p99 latency SLO.

• CacheRoute assigns high-rate keys to stable warm server sets and places them according to expected load; exceptionally hot keys may use multiple destinations.
• It sustained 176±11 QPS and increased served KV-cache hit rate from 64.1±1.3% with cache-blind balancing to 93.2±0.5%.
• Experiments isolate gains from both request affinity and load-aware placement across multiple model sizes and traffic patterns.
• Two 32B workloads showed that insufficient KV reuse can let residual load skew reduce or eliminate gains, motivating shadow-replay validation before deployment.