ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers

TL;DR - ELSAA approximates Transformer attention with complementary sparse and low-rank branches, avoiding construction of the full quadratic score matrix. It aims to support longer-context training while retaining both sharp token-level and broad global interactions.

• Applies approximation to the attention score operator after dense (Q,K,V) projections, not to learned projection or output matrices.
• Uses sparse attention for selected high-similarity interactions and a low-rank branch for diffuse global context.
• Introduces denominator-aware fusion to balance branches whose normalization supports contain different attention mass.
• Avoids materializing the full (N \times N) attention matrix.