CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving
Ranking
Overall
88
Content
95
Popularity
71
Observed public metrics from 1 member.
Merged summary
TL;DR - CacheRoute periodically plans prefix-affinity routing so repeated LLM requests reach servers holding reusable KV caches without severely skewing load. On Llama-3.3-70B across 60 H100 GPUs, it delivered 2.3× the throughput of the strongest baseline under a 3.5-second p99 latency SLO.
- CacheRoute assigns high-rate keys to stable warm server sets and places them according to expected load; exceptionally hot keys may use multiple destinations.
- It sustained 176±11 QPS and increased served KV-cache hit rate from 64.1±1.3% with cache-blind balancing to 93.2±0.5%.
- Experiments isolate gains from both request affinity and load-aware placement across multiple model sizes and traffic patterns.
- Two 32B workloads showed that insufficient KV reuse can let residual load skew reduce or eliminate gains, motivating shadow-replay validation before deployment.
Sources (1)
CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving
Public signals
Semantic Scholar citations 1 · Semantic Scholar influential citations 0
TL;DR - CacheRoute periodically plans prefix-affinity routing so repeated LLM requests reach servers holding reusable KV caches without severely skewing load. On Llama-3.3-70B across 60 H100 GPUs, it delivered 2.3× the throughput of the strongest baseline under a 3.5-second p99 latency SLO.
- CacheRoute assigns high-rate keys to stable warm server sets and places them according to expected load; exceptionally hot keys may use multiple destinations.
- It sustained 176±11 QPS and increased served KV-cache hit rate from 64.1±1.3% with cache-blind balancing to 93.2±0.5%.
- Experiments isolate gains from both request affinity and load-aware placement across multiple model sizes and traffic patterns.
- Two 32B workloads showed that insufficient KV reuse can let residual load skew reduce or eliminate gains, motivating shadow-replay validation before deployment.