Rethinking Expressivity and Efficiency in Test-Time Training

TL;DR - E²-TTT is a test-time training method that parallelizes chunk-level computation while preserving the fast-weight and momentum states of per-token updates. It combines the throughput of chunk-wise methods with stronger long-context retrieval and length extrapolation.

• Derives a closed-form chunk transition that exactly reproduces per-token recurrence states under the chunk-start gradient approximation.
• Trains models up to 1.3B parameters from scratch and matches prior TTT and hybrid-attention baselines in language modeling.
• Outperforms those baselines on in-context retrieval.
• Retains over 90% passkey accuracy at eight times its training context length while matching efficient chunk-wise training throughput.