把记忆交给CPU，大模型会变快
Ranking
No observed public metrics; popularity remains neutral/archived.
Merged summary
TL;DR - Intel proposes shifting reusable LLM KV caches from scarce GPU memory into a CPU-managed storage hierarchy, using QAT hardware compression to reduce recomputation and improve serving capacity. This is particularly relevant for long-running agents, whose accumulated context can otherwise increase time to first token and GPU costs.
- KV Shrink combines GPU-to-CPU/SSD cache offloading, hot/cold scheduling APIs, and lossless QAT compression; data-layout changes reportedly reduce KV-cache storage by roughly 20%–30%.
- In Intel’s Qwen3-32B tests at an 80% cache hit rate, KV Shrink delivered up to about 5× faster TTFT than native vLLM without tiered offloading, while QAT compression achieved roughly twice the performance of CPU software compression.
- A separate Coding Agent test against LMCache reported average TTFT reductions of about 12.1% at single-stream load and 4.6% at eight-way concurrency.
- Intel’s broader KV Fuse, KV Cascade, and KV Infinity efforts target cache composition, context filtering, and on-demand loading; actual benefits depend on workload, cache-hit rate, transfer costs, and system configuration.
Sources (1)
把记忆交给CPU，大模型会变快
TL;DR - Intel proposes shifting reusable LLM KV caches from scarce GPU memory into a CPU-managed storage hierarchy, using QAT hardware compression to reduce recomputation and improve serving capacity. This is particularly relevant for long-running agents, whose accumulated context can otherwise increase time to first token and GPU costs.
- KV Shrink combines GPU-to-CPU/SSD cache offloading, hot/cold scheduling APIs, and lossless QAT compression; data-layout changes reportedly reduce KV-cache storage by roughly 20%–30%.
- In Intel’s Qwen3-32B tests at an 80% cache hit rate, KV Shrink delivered up to about 5× faster TTFT than native vLLM without tiered offloading, while QAT compression achieved roughly twice the performance of CPU software compression.
- A separate Coding Agent test against LMCache reported average TTFT reductions of about 12.1% at single-stream load and 4.6% at eight-way concurrency.
- Intel’s broader KV Fuse, KV Cascade, and KV Infinity efforts target cache composition, context filtering, and on-demand loading; actual benefits depend on workload, cache-hit rate, transfer costs, and system configuration.