OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - OctoLong is a context-engineering pipeline that mines dependency-rich, cross-repository code contexts (up to millions of tokens) to build long-context training data, plus OctoLong-Instruct, a suite of open long-context LMs trained on it. It matters because existing long-context corpora are finite and weak in long-distance dependencies, and this offers a scalable substitute.
- The pipeline combines an AST parser, a language server backend, and a package manager to recursively retrieve code references, assembling contexts with genuine long-range dependencies rather than concatenated text.
- Training recipe: context-extension mid-training on a ~50B-token mixture including ~6.2B tokens of OctoLong code contexts, followed by ~10B tokens of instruction tuning, across base models from 600M to 14B parameters.
- Replacing just 12% of traditional context-extension corpora with OctoLong data reportedly improves long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks.
- Gains also carry over to short-context coding (API usage); claims are evaluated via ablations and comparison against 18 state-of-the-art open-weight long-context LMs, though no specific numbers are given in the abstract.
Sources (1)
OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling
TL;DR - OctoLong is a context-engineering pipeline that mines dependency-rich, cross-repository code contexts (up to millions of tokens) to build long-context training data, plus OctoLong-Instruct, a suite of open long-context LMs trained on it. It matters because existing long-context corpora are finite and weak in long-distance dependencies, and this offers a scalable substitute.
- The pipeline combines an AST parser, a language server backend, and a package manager to recursively retrieve code references, assembling contexts with genuine long-range dependencies rather than concatenated text.
- Training recipe: context-extension mid-training on a ~50B-token mixture including ~6.2B tokens of OctoLong code contexts, followed by ~10B tokens of instruction tuning, across base models from 600M to 14B parameters.
- Replacing just 12% of traditional context-extension corpora with OctoLong data reportedly improves long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks.
- Gains also carry over to short-context coding (API usage); claims are evaluated via ablations and comparison against 18 state-of-the-art open-weight long-context LMs, though no specific numbers are given in the abstract.