Indexing: the Beginning and the End
Ranking
Overall
72
Content
85
Popularity
41
Observed public metrics from 1 member.
Merged summary
TL;DR - This paper uses the indexing primitive to expose information bottlenecks across sequence-model architectures. It proves that input ordering, masking, and attention type fundamentally affect whether indexing can be solved with constant depth.
- Low-causal-complexity masked architectures cannot solve end-position indexing in any constant number of layers.
- Small softmax transformers solve end-position indexing in one layer; non-masked linear attention requires two.
- With the index first, small RNNs need one layer, while the other studied architectures need two.
- The unconditional lower bounds allow infinite-precision arithmetic, and experiments up to sequence length 64 qualitatively support the theory.
Sources (1)
Indexing: the Beginning and the End
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - This paper uses the indexing primitive to expose information bottlenecks across sequence-model architectures. It proves that input ordering, masking, and attention type fundamentally affect whether indexing can be solved with constant depth.
- Low-causal-complexity masked architectures cannot solve end-position indexing in any constant number of layers.
- Small softmax transformers solve end-position indexing in one layer; non-masked linear attention requires two.
- With the index first, small RNNs need one layer, while the other studied architectures need two.
- The unconditional lower bounds allow infinite-precision arithmetic, and experiments up to sequence length 64 qualitatively support the theory.