InferScale: GPU-Native KV Injection for Personalized LLM Serving
Ranking
Overall
80
Content
95
Popularity
44
Observed public metrics from 1 member.
Merged summary
TL;DR - InferScale accelerates personalized LLM serving by caching memory facts as reusable GPU-resident KV states and injecting them directly into vLLM’s paged cache. This largely decouples time-to-first-token from retrieved-context size while retaining most baseline accuracy.
- Chunked RoPE assigns serving-time positions to independently cached memory facts.
- Context-Window Encoding preserves some cross-fact context without caching unrelated tokens.
- At 50 retrieved facts, InferScale reduced TTFT by 72–79% (3.6–4.8×).
- It achieved 60.3% accuracy versus Mem0’s 63.3% and delivered 3.7–4.5× concurrent throughput.
Sources (1)
InferScale: GPU-Native KV Injection for Personalized LLM Serving
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - InferScale accelerates personalized LLM serving by caching memory facts as reusable GPU-resident KV states and injecting them directly into vLLM’s paged cache. This largely decouples time-to-first-token from retrieved-context size while retaining most baseline accuracy.
- Chunked RoPE assigns serving-time positions to independently cached memory facts.
- Context-Window Encoding preserves some cross-fact context without caching unrelated tokens.
- At 50 retrieved facts, InferScale reduced TTFT by 72–79% (3.6–4.8×).
- It achieved 60.3% accuracy versus Mem0’s 63.3% and delivered 3.7–4.5× concurrent throughput.