InferScale: GPU-Native KV Injection for Personalized LLM Serving

TL;DR - InferScale accelerates personalized LLM serving by caching memory facts as reusable GPU-resident KV states and injecting them directly into vLLM’s paged cache. This largely decouples time-to-first-token from retrieved-context size while retaining most baseline accuracy.

• Chunked RoPE assigns serving-time positions to independently cached memory facts.
• Context-Window Encoding preserves some cross-fact context without caching unrelated tokens.
• At 50 retrieved facts, InferScale reduced TTFT by 72–79% (3.6–4.8×).
• It achieved 60.3% accuracy versus Mem0’s 63.3% and delivered 3.7–4.5× concurrent throughput.