‹ back to 2026-09-02

SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

arXiv cs.LG LLMs & Foundation Models Shaowen Wang, Ge Zhang, Kairong Luo, Yuhao Wu, Shaofan Liu, Jiaheng Liu, Wenhao Huang, Shen Yan, Jian Li 2026-09-01

TL;DR - SMELT is a compute-matched Mixture-of-Experts Transformer that reuses its middle layers to gain effective depth without increasing parameter, FLOP, or KV-cache budgets. Its scaling laws indicate 6.8–18.0% training-FLOP savings on the compute-optimal frontier, with especially strong downstream gains for code.

SMELT loops the middle half of its layers twice while matching an unlooped baseline on per-token FLOPs, non-embedding parameters, and KV cache.
Scaling experiments cover four model sizes up to 54B non-embedding parameters, with separate Chinchilla-style scaling laws fitted for looped and baseline architectures.
Benefits grow with sample length and the number of in-context examples, and downstream improvements exceed what validation loss alone predicts.
Mechanistic analysis suggests the second pass reduces attention sinks and shifts attention toward content-relevant tokens.

view merged work →