🛰️ Daily AI Frontier
‹ back to 2026-08-10

Modular TTT: Rethinking Test-Time Training as Composable Modules

Research LLMs & Foundation Models

Ranking

Overall 70
Content 70
Popularity 69

Observed public metrics from 1 member.

Representative image for Modular TTT: Rethinking Test-Time Training as Composable Modules

Merged summary

TL;DR - Modular TTT is a framework that represents test-time training's inner learner as a directed acyclic graph, exposing fast-weight network, loss, learning rate, weight decay, and normalization as explicit, composable design dimensions. It matters because it turns the proliferation of hard-coded TTT variants into a searchable design space where each component's contribution can be isolated.

  • The framework auto-composes primitive-level train-view forward, train-view backward, and causal query-view rules into the full graph-level TTT computation, including the fast-weight state transition.
  • Systematic ablations found that small learning-rate initialization, weight decay, and a single-layer nonlinearity help; MSE and inner-product losses perform about the same.
  • Deeper fast-weight networks and normalization hurt performance, attributed to excessively large activations; residual connections and gating gave little measurable benefit.
  • The best resulting variant, trained at 410M and 1.45B parameters on 100B tokens, matched Gated DeltaNet on training loss and benchmark performance.

Sources (1)

Modular TTT: Rethinking Test-Time Training as Composable Modules

arXiv cs.LG Bohao Tang, Zhen Qin, Yuqi Pan, Zheng Li, Pengfei Liu, Ya Zhang 2026-08-07 arXiv:2608.07110
Public signals Hugging Face upvotes 9
Providers: Hugging Face · Upvotes 9 OpenAlex · N/A Publisher · N/A Semantic Scholar · N/A X · N/A Fetched 2026-09-09 08:18:31.506519 UTC

TL;DR - Modular TTT is a framework that represents test-time training's inner learner as a directed acyclic graph, exposing fast-weight network, loss, learning rate, weight decay, and normalization as explicit, composable design dimensions. It matters because it turns the proliferation of hard-coded TTT variants into a searchable design space where each component's contribution can be isolated.

  • The framework auto-composes primitive-level train-view forward, train-view backward, and causal query-view rules into the full graph-level TTT computation, including the fast-weight state transition.
  • Systematic ablations found that small learning-rate initialization, weight decay, and a single-layer nonlinearity help; MSE and inner-product losses perform about the same.
  • Deeper fast-weight networks and normalization hurt performance, attributed to excessively large activations; residual connections and gating gave little measurable benefit.
  • The best resulting variant, trained at 410M and 1.45B parameters on 100B tokens, matched Gated DeltaNet on training loss and benchmark performance.
item →