How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents

TL;DR - This paper shows that transformer model growth, recursive depth, and boundary operators can improve pre-training scaling exponents rather than merely deliver fixed efficiency gains. A 7.4B growth architecture matches GPT-3 13B on CORE with roughly 20× less compute, with gains increasing at larger scales.

• Growing models during training, with or without shared weights, produces the largest changes in scaling exponents.
• Boundary operators that normalize and inject an earlier block also yield increasing compute-efficiency gains, though smaller than model growth.
• In data-constrained multi-epoch training, looping acts as a regularizer, and the compute-optimal loop count increases with scale.
• The results suggest that increasing usable computational depth within a fixed compute budget can improve scaling behavior.