Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge
Ranking
Overall
86
Content
95
Popularity
65
Observed public metrics from 1 member.
Merged summary
TL;DR - Long-context training can weaken an LLM’s parametric knowledge by encouraging it to retrieve information from context rather than internalize it. Performance peaks at an intermediate context length, challenging the assumption that longer training contexts are always beneficial.
- Longer pretraining contexts initially improve language modeling, understanding, and closed-book MCQA, but performance declines beyond an intermediate optimum.
- Fine-tuning with abundant relevant context improves supported-context performance while reducing robustness when test-time context is absent or misleading.
- Informative contexts shift gradient pressure from feed-forward networks toward attention modules.
- Causal interventions indicate that this architectural shift increases contextual reliance during inference.
Sources (1)
Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge
Public signals
Hugging Face upvotes 6
TL;DR - Long-context training can weaken an LLM’s parametric knowledge by encouraging it to retrieve information from context rather than internalize it. Performance peaks at an intermediate context length, challenging the assumption that longer training contexts are always beneficial.
- Longer pretraining contexts initially improve language modeling, understanding, and closed-book MCQA, but performance declines beyond an intermediate optimum.
- Fine-tuning with abundant relevant context improves supported-context performance while reducing robustness when test-time context is absent or misleading.
- Informative contexts shift gradient pressure from feed-forward networks toward attention modules.
- Causal interventions indicate that this architectural shift increases contextual reliance during inference.