🛰️ Daily AI Frontier
‹ back to 2026-09-02

SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

Research LLMs & Foundation Models

Ranking

Overall 91
Content 100
Popularity 70

Observed public metrics from 1 member.

Merged summary

TL;DR - SMELT is a compute-matched Mixture-of-Experts Transformer that reuses its middle layers to gain effective depth without increasing parameter, FLOP, or KV-cache budgets. Its scaling laws indicate 6.8–18.0% training-FLOP savings on the compute-optimal frontier, with especially strong downstream gains for code.

  • SMELT loops the middle half of its layers twice while matching an unlooped baseline on per-token FLOPs, non-embedding parameters, and KV cache.
  • Scaling experiments cover four model sizes up to 54B non-embedding parameters, with separate Chinchilla-style scaling laws fitted for looped and baseline architectures.
  • Benefits grow with sample length and the number of in-context examples, and downstream improvements exceed what validation loss alone predicts.
  • Mechanistic analysis suggests the second pass reduces attention sinks and shifts attention toward content-relevant tokens.

Sources (1)

SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

arXiv cs.LG Shaowen Wang, Ge Zhang, Kairong Luo, Yuhao Wu, Shaofan Liu, Jiaheng Liu, Wenhao Huang, Shen Yan, Jian Li 2026-09-01 arXiv:2609.01343
Public signals Hugging Face upvotes 90 · Semantic Scholar citations 4 · Semantic Scholar influential citations 0
Providers: Hugging Face · Upvotes 90 OpenAlex · N/A Publisher · N/A Semantic Scholar · Citations 4 · Influential citations 0 X · N/A Fetched 2026-09-25 14:24:57.071033 UTC

TL;DR - SMELT is a compute-matched Mixture-of-Experts Transformer that reuses its middle layers to gain effective depth without increasing parameter, FLOP, or KV-cache budgets. Its scaling laws indicate 6.8–18.0% training-FLOP savings on the compute-optimal frontier, with especially strong downstream gains for code.

  • SMELT loops the middle half of its layers twice while matching an unlooped baseline on per-token FLOPs, non-embedding parameters, and KV cache.
  • Scaling experiments cover four model sizes up to 54B non-embedding parameters, with separate Chinchilla-style scaling laws fitted for looped and baseline architectures.
  • Benefits grow with sample length and the number of in-context examples, and downstream improvements exceed what validation loss alone predicts.
  • Mechanistic analysis suggests the second pass reduces attention sinks and shifts attention toward content-relevant tokens.
item →