Modular TTT: Rethinking Test-Time Training as Composable Modules

TL;DR - Modular TTT is a framework that represents test-time training's inner learner as a directed acyclic graph, exposing fast-weight network, loss, learning rate, weight decay, and normalization as explicit, composable design dimensions. It matters because it turns the proliferation of hard-coded TTT variants into a searchable design space where each component's contribution can be isolated.

• The framework auto-composes primitive-level train-view forward, train-view backward, and causal query-view rules into the full graph-level TTT computation, including the fast-weight state transition.
• Systematic ablations found that small learning-rate initialization, weight decay, and a single-layer nonlinearity help; MSE and inner-product losses perform about the same.
• Deeper fast-weight networks and normalization hurt performance, attributed to excessively large activations; residual connections and gating gave little measurable benefit.
• The best resulting variant, trained at 410M and 1.45B parameters on 100B tokens, matched Gated DeltaNet on training loss and benchmark performance.