ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers
Ranking
Overall
76
Content
90
Popularity
43
Observed public metrics from 1 member.
Merged summary
TL;DR - ELSAA approximates Transformer attention with complementary sparse and low-rank branches, avoiding construction of the full quadratic score matrix. It aims to support longer-context training while retaining both sharp token-level and broad global interactions.
- Applies approximation to the attention score operator after dense (Q,K,V) projections, not to learned projection or output matrices.
- Uses sparse attention for selected high-similarity interactions and a low-rank branch for diffuse global context.
- Introduces denominator-aware fusion to balance branches whose normalization supports contain different attention mass.
- Avoids materializing the full (N \times N) attention matrix.
Sources (1)
ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - ELSAA approximates Transformer attention with complementary sparse and low-rank branches, avoiding construction of the full quadratic score matrix. It aims to support longer-context training while retaining both sharp token-level and broad global interactions.
- Applies approximation to the attention score operator after dense (Q,K,V) projections, not to learned projection or output matrices.
- Uses sparse attention for selected high-similarity interactions and a low-rank branch for diffuse global context.
- Introduces denominator-aware fusion to balance branches whose normalization supports contain different attention mass.
- Avoids materializing the full (N \times N) attention matrix.