Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge

TL;DR - Long-context training can weaken an LLM’s parametric knowledge by encouraging it to retrieve information from context rather than internalize it. Performance peaks at an intermediate context length, challenging the assumption that longer training contexts are always beneficial.

• Longer pretraining contexts initially improve language modeling, understanding, and closed-book MCQA, but performance declines beyond an intermediate optimum.
• Fine-tuning with abundant relevant context improves supported-context performance while reducing robustness when test-time context is absent or misleading.
• Informative contexts shift gradient pressure from feed-forward networks toward attention modules.
• Causal interventions indicate that this architectural shift increases contextual reliance during inference.