One QK Channel, Many Sources: Guarding Low-Precision Attention Collapse
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - An arXiv preprint showing that abrupt bfloat16 transformer training collapse, though triggered by different low-precision error sources, always propagates through a single shared query-key (QK) spectral runaway channel, and proposes QK-Guard to block it. It matters because it replaces per-source numerical patches with one targeted intervention for stable low-precision training.
- Authors reproduced a GPT-2-class collapse traced to the streaming-softmax accumulator (fp32 accumulation fixes it), then used it as an assay: errors injected outside attention still cause the same QK spectral runaway, and correcting only QK keeps training stable even with the source fault active — "fault source is not failure channel."
- A causal probe projecting updates off the leading three singular directions of current QK weights gives a query-projection top singular value of 11.1, versus 237 when equal energy is removed elsewhere, indicating QK drives rather than merely tracks the early runaway.
- Collapse entry depends on temporal sign-coherence of updates across steps rather than aggregate deviation magnitude.
- QK-Guard is a dormant controller that activates parameter-free QK normalization at the onset of attention-logit saturation; it contained every tested runaway and matched always-on QK normalization over 60k steps, while non-QK actions at the same trigger failed. Results held across tested architectures, scales, and a second GPU architecture.
Sources (1)
One QK Channel, Many Sources: Guarding Low-Precision Attention Collapse
TL;DR - An arXiv preprint showing that abrupt bfloat16 transformer training collapse, though triggered by different low-precision error sources, always propagates through a single shared query-key (QK) spectral runaway channel, and proposes QK-Guard to block it. It matters because it replaces per-source numerical patches with one targeted intervention for stable low-precision training.
- Authors reproduced a GPT-2-class collapse traced to the streaming-softmax accumulator (fp32 accumulation fixes it), then used it as an assay: errors injected outside attention still cause the same QK spectral runaway, and correcting only QK keeps training stable even with the source fault active — "fault source is not failure channel."
- A causal probe projecting updates off the leading three singular directions of current QK weights gives a query-projection top singular value of 11.1, versus 237 when equal energy is removed elsewhere, indicating QK drives rather than merely tracks the early runaway.
- Collapse entry depends on temporal sign-coherence of updates across steps rather than aggregate deviation magnitude.
- QK-Guard is a dormant controller that activates parameter-free QK normalization at the onset of attention-logit saturation; it contained every tested runaway and matched always-on QK normalization over 60k steps, while non-QK actions at the same trigger failed. Results held across tested architectures, scales, and a second GPU architecture.