A Compositional Theory of Causally Masked Transformers
Ranking
Overall
76
Content
90
Popularity
44
Observed public metrics from 1 member.
Merged summary
TL;DR - This paper develops an algebraic framework for characterizing what finite-precision, causally masked transformers can compute on arbitrary-length inputs. It connects attention mechanisms and numerical semantics to distinct classes of finite-state memory and expressivity.
- The framework models attention’s prefix summary as finite internal memory that future queries can access.
- Independent per-head updates compose across layers, enabling systematic derivation of expressivity bounds.
- Without positional embeddings, sliding-window, modified soft, combined, and standard floating-point soft attention yield progressively richer memory operations.
- These cases correspond to definite, R-trivial, locally R-trivial, and aperiodic semigroups; the bounds are tight under an explicit free-wiring assumption.
Sources (1)
A Compositional Theory of Causally Masked Transformers
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - This paper develops an algebraic framework for characterizing what finite-precision, causally masked transformers can compute on arbitrary-length inputs. It connects attention mechanisms and numerical semantics to distinct classes of finite-state memory and expressivity.
- The framework models attention’s prefix summary as finite internal memory that future queries can access.
- Independent per-head updates compose across layers, enabling systematic derivation of expressivity bounds.
- Without positional embeddings, sliding-window, modified soft, combined, and standard floating-point soft attention yield progressively richer memory operations.
- These cases correspond to definite, R-trivial, locally R-trivial, and aperiodic semigroups; the bounds are tight under an explicit free-wiring assumption.