‹ back to 2026-08-05

When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings

arXiv cs.CL LLMs & Foundation Models Christopher Schröder, Lukas Gienapp, Ferdinand Schlatt, Martin Potthast, Gerhard Heyer 2026-08-04

TL;DR - ALiBi’s linear positional biases can underflow floating-point precision, zeroing attention weights and making some heads partially blind. This can seriously harm long-context token retrieval despite barely affecting standard decoder benchmarks.

The failure occurs in state-of-the-art pretrained ALiBi models and is distinct from ordinary out-of-context degradation.
Experiments with 148M-parameter decoders show substantial retrieval degradation but only minor benchmark impact.
Among four training-time mitigations, log-scaled distances most consistently improve passkey retrieval.
Default ALiBi slopes remain competitive for needle-in-a-haystack retrieval despite the numerical issue.

view merged work →