How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents
Ranking
Overall
84
Content
95
Popularity
59
Observed public metrics from 1 member.
Merged summary
TL;DR - This paper shows that transformer model growth, recursive depth, and boundary operators can improve pre-training scaling exponents rather than merely deliver fixed efficiency gains. A 7.4B growth architecture matches GPT-3 13B on CORE with roughly 20Ă— less compute, with gains increasing at larger scales.
- Growing models during training, with or without shared weights, produces the largest changes in scaling exponents.
- Boundary operators that normalize and inject an earlier block also yield increasing compute-efficiency gains, though smaller than model growth.
- In data-constrained multi-epoch training, looping acts as a regularizer, and the compute-optimal loop count increases with scale.
- The results suggest that increasing usable computational depth within a fixed compute budget can improve scaling behavior.
Sources (1)
How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents
Public signals
Hugging Face upvotes 1
TL;DR - This paper shows that transformer model growth, recursive depth, and boundary operators can improve pre-training scaling exponents rather than merely deliver fixed efficiency gains. A 7.4B growth architecture matches GPT-3 13B on CORE with roughly 20Ă— less compute, with gains increasing at larger scales.
- Growing models during training, with or without shared weights, produces the largest changes in scaling exponents.
- Boundary operators that normalize and inject an earlier block also yield increasing compute-efficiency gains, though smaller than model growth.
- In data-constrained multi-epoch training, looping acts as a regularizer, and the compute-optimal loop count increases with scale.
- The results suggest that increasing usable computational depth within a fixed compute budget can improve scaling behavior.