Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention

TL;DR - This paper argues that value-pathway gates improve softmax attention by adding two missing capabilities: abstaining from producing an output and filtering noisy residual-stream features. Experiments across 10M–350M parameter models show that combining both mechanisms consistently performs best with negligible parameter overhead.

• A learned per-head sink logit enables abstention, while per-value gating suppresses interference from superposed features.
• Abstention provides most of the gating benefit at 10M parameters, but its impact declines as model size increases.
• Noise filtering becomes more beneficial with scale and accounts for most of the improvement at 350M parameters.
• Controlled interference experiments confirm the filtering effect and reveal distinct blind spots in each gate type; both remain compatible with key-value caching.