Squeezing More from Limited Data with Recursive Transformers

TL;DR - This paper studies language-model pre-training with fixed budgets of 10M–100M words, where larger standard Transformers eventually overfit. Recursive Transformers with shared blocks and factorized embeddings use abundant compute more effectively, outperforming standard Transformers at 10M and 100M words.

• Optimal model size depends strongly on both the available data and the downstream task.
• Standard Transformers scale down poorly because embeddings consume much of the parameter budget and per-token compute is coupled to representational capacity.
• Recursive Transformers reuse one block across depth, increasing computation without proportionally increasing parameters.
• Three recursive models remained competitive with BabyLM Challenge 2025 winners.