Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention
Ranking
No observed public metrics; popularity remains neutral/archived.
Merged summary
TL;DR - This paper argues that value-pathway gates improve softmax attention by adding two missing capabilities: abstaining from producing an output and filtering noisy residual-stream features. Experiments across 10M–350M parameter models show that combining both mechanisms consistently performs best with negligible parameter overhead.
- A learned per-head sink logit enables abstention, while per-value gating suppresses interference from superposed features.
- Abstention provides most of the gating benefit at 10M parameters, but its impact declines as model size increases.
- Noise filtering becomes more beneficial with scale and accounts for most of the improvement at 350M parameters.
- Controlled interference experiments confirm the filtering effect and reveal distinct blind spots in each gate type; both remain compatible with key-value caching.
Sources (1)
Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention
TL;DR - This paper argues that value-pathway gates improve softmax attention by adding two missing capabilities: abstaining from producing an output and filtering noisy residual-stream features. Experiments across 10M–350M parameter models show that combining both mechanisms consistently performs best with negligible parameter overhead.
- A learned per-head sink logit enables abstention, while per-value gating suppresses interference from superposed features.
- Abstention provides most of the gating benefit at 10M parameters, but its impact declines as model size increases.
- Noise filtering becomes more beneficial with scale and accounts for most of the improvement at 350M parameters.
- Controlled interference experiments confirm the filtering effect and reveal distinct blind spots in each gate type; both remain compatible with key-value caching.