Rethinking Expressivity and Efficiency in Test-Time Training
Ranking
Overall
90
Content
100
Popularity
66
Observed public metrics from 1 member.
Merged summary
TL;DR - E²-TTT is a test-time training method that parallelizes chunk-level computation while preserving the fast-weight and momentum states of per-token updates. It combines the throughput of chunk-wise methods with stronger long-context retrieval and length extrapolation.
- Derives a closed-form chunk transition that exactly reproduces per-token recurrence states under the chunk-start gradient approximation.
- Trains models up to 1.3B parameters from scratch and matches prior TTT and hybrid-attention baselines in language modeling.
- Outperforms those baselines on in-context retrieval.
- Retains over 90% passkey accuracy at eight times its training context length while matching efficient chunk-wise training throughput.
Sources (1)
Rethinking Expressivity and Efficiency in Test-Time Training
Public signals
Hugging Face upvotes 1
TL;DR - E²-TTT is a test-time training method that parallelizes chunk-level computation while preserving the fast-weight and momentum states of per-token updates. It combines the throughput of chunk-wise methods with stronger long-context retrieval and length extrapolation.
- Derives a closed-form chunk transition that exactly reproduces per-token recurrence states under the chunk-start gradient approximation.
- Trains models up to 1.3B parameters from scratch and matches prior TTT and hybrid-attention baselines in language modeling.
- Outperforms those baselines on in-context retrieval.
- Retains over 90% passkey accuracy at eight times its training context length while matching efficient chunk-wise training throughput.