Linearized 2-Simplicial Attention

TL;DR - A method that linearizes 2-simplicial (trilinear) attention by recasting the score as a composite query–key inner product, then approximating it with positive random features to get linear-time cost with global context. It matters because it enables higher-order attention without the quadratic (or windowed, locality-limited) cost that has kept 2-simplicial attention impractical.

• The trilinear score is rewritten so summation over one token axis matches ordinary softmax attention form; positive random features compress the entire past into a fixed-size state, while the second axis stays explicit over a short recent-token window.
• Result is linear cost in sequence length plus global reach, which windowed 2-simplicial attention lacks.
• Implemented with custom Triton kernels and combined with Kimi Delta Attention (KDA) to produce a model containing no softmax attention at all.
• Under matched compute it reports the highest mean downstream accuracy among compared architectures; at 16k context it beats a KDA hybrid on mean accuracy and cuts LAMBADA perplexity from 715.6 to 602.6.