OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling

TL;DR - OctoLong is a context-engineering pipeline that mines dependency-rich, cross-repository code contexts (up to millions of tokens) to build long-context training data, plus OctoLong-Instruct, a suite of open long-context LMs trained on it. It matters because existing long-context corpora are finite and weak in long-distance dependencies, and this offers a scalable substitute.

• The pipeline combines an AST parser, a language server backend, and a package manager to recursively retrieve code references, assembling contexts with genuine long-range dependencies rather than concatenated text.
• Training recipe: context-extension mid-training on a ~50B-token mixture including ~6.2B tokens of OctoLong code contexts, followed by ~10B tokens of instruction tuning, across base models from 600M to 14B parameters.
• Replacing just 12% of traditional context-extension corpora with OctoLong data reportedly improves long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks.
• Gains also carry over to short-context coding (API usage); claims are evaluated via ablations and comparison against 18 state-of-the-art open-weight long-context LMs, though no specific numbers are given in the abstract.