Algebraic Decomposition Theory for Transformer Length Generalization
Ranking
Overall
87
Content
100
Popularity
56
Observed public metrics from 1 member.
Merged summary
TL;DR - This paper gives the first complete characterization of regular languages on which transformers can generalize beyond their training sequence lengths. It also provides a polynomial-time decision algorithm based on a new algebraic decomposition theory.
- Characterizes transformer length generalization through the C-RASP formalism.
- Extends classical finite-semigroup decomposition theory using iterated wreath products of the additive integer group.
- Decides regular-language membership in polynomial time relative to the syntactic monoid’s size.
- Experiments show the theory predicts transformer length-generalization behavior better than existing classifications.
Sources (1)
Algebraic Decomposition Theory for Transformer Length Generalization
Public signals
Semantic Scholar citations 1 · Semantic Scholar influential citations 0
TL;DR - This paper gives the first complete characterization of regular languages on which transformers can generalize beyond their training sequence lengths. It also provides a polynomial-time decision algorithm based on a new algebraic decomposition theory.
- Characterizes transformer length generalization through the C-RASP formalism.
- Extends classical finite-semigroup decomposition theory using iterated wreath products of the additive integer group.
- Decides regular-language membership in polynomial time relative to the syntactic monoid’s size.
- Experiments show the theory predicts transformer length-generalization behavior better than existing classifications.