What It Costs to Compose, Rebuild, and Correct Precomputed Memory

TL;DR - This paper evaluates when reusable precomputed LLM memory remains correct and how much it costs to update. It finds that composing memories from separate parts and applying real-time corrections can be unreliable, while rebuilding often costs a substantial fraction of the original preparation.

• Experiments use Llama-3.1-8B-Instruct with saved key-value caches and trained cache compressions.
• Precomputed memory quality degrades when independently prepared components are assembled.
• Keeping memory current generally requires rebuilding it at the cadence of changes to its source material.
• Warm rebuilds and carefully phrased side-channel updates show promise, with the latter serving as an interim measure between rebuilds.