Loop the Loopies!
Merged summary
TL;DR - Loopie is a series of looped Mixture-of-Experts Transformers that reportedly outperforms standard Transformers under equal compute budgets. Its post-training pipeline also yields tool-free gold-medal performance on the 2025 IMO and IPhO.
- Includes 20B/2B-active and 6B/0.6B-active MoE models.
- Reuses Transformer layers through looping rather than simply scaling parameter count.
- Ablations include comparisons against a vanilla 30B-A3B model.
- Claims to overcome the historical compute-efficiency disadvantage of looped Transformers.
Sources (1)
Loop the Loopies!
TL;DR - Loopie is a series of looped Mixture-of-Experts Transformers that reportedly outperforms standard Transformers under equal compute budgets. Its post-training pipeline also yields tool-free gold-medal performance on the 2025 IMO and IPhO.
- Includes 20B/2B-active and 6B/0.6B-active MoE models.
- Reuses Transformer layers through looping rather than simply scaling parameter count.
- Ablations include comparisons against a vanilla 30B-A3B model.
- Claims to overcome the historical compute-efficiency disadvantage of looped Transformers.