On-Demand Attention: Language Models Know When to Recall

TL;DR - On-Demand Attention (ODA) lets language models selectively recall their full context only when decoding states predict it will help. This reduces costly global attention reads while retaining most of their performance benefits and enabling practical long-context speedups.

• A lightweight recall head decides during generation when to switch from local to global attention.
• Only the recall head is trained; pretrained model weights remain unchanged, and the full historical KV cache stays available.
• GPU-side conditional execution in vLLM converts fewer global reads into decoding speedups at long context lengths.
• Experiments on Qwen and Gemma models, including hybrid-attention architectures, show selective recall recovers most performance lost with local-only attention.