Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection
Ranking
Overall
67
Content
80
Popularity
36
Observed public metrics from 1 member.
Merged summary
TL;DR - PRECOG pre-encodes document corpora offline into State-Space Model hidden states and injects the best-matching state at query time, collapsing RAG prefill cost from O(L_context) to O(1). It matters because it turns retrieval-augmented generation on edge hardware from unusable (~27 s) into interactive (<6 ms).
- Exploits an SSM-specific property: the fixed-size, position-agnostic recurrent hidden state fully summarizes everything read, so retrieved context can be swapped in directly instead of re-ingested in-context.
- Adds SMC (Structured Memory Consolidation): hierarchical persistent memory with cognitive-domain clustering, an adjustable fidelity-vs-storage dial, O(1) session init, and fusion of episodic, semantic, and retrieved corpus states.
- Demonstrated on TENNs-LLM, a 1.2B gated-SSM model with a 192 KB hidden state; reported ~4500Ă— prefill speedup while matching in-context RAG answer quality.
- Authors argue the mechanism is architecturally impossible for Transformer KV-caches, which are position-entangled and grow linearly with context.
Sources (1)
Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - PRECOG pre-encodes document corpora offline into State-Space Model hidden states and injects the best-matching state at query time, collapsing RAG prefill cost from O(L_context) to O(1). It matters because it turns retrieval-augmented generation on edge hardware from unusable (~27 s) into interactive (<6 ms).
- Exploits an SSM-specific property: the fixed-size, position-agnostic recurrent hidden state fully summarizes everything read, so retrieved context can be swapped in directly instead of re-ingested in-context.
- Adds SMC (Structured Memory Consolidation): hierarchical persistent memory with cognitive-domain clustering, an adjustable fidelity-vs-storage dial, O(1) session init, and fusion of episodic, semantic, and retrieved corpus states.
- Demonstrated on TENNs-LLM, a 1.2B gated-SSM model with a 192 KB hidden state; reported ~4500Ă— prefill speedup while matching in-context RAG answer quality.
- Authors argue the mechanism is architecturally impossible for Transformer KV-caches, which are position-entangled and grow linearly with context.