「用初中数学讲明白AI」第3章：注意力——一个加权平均如何改变世界
Ranking
Overall
33
Content
25
Popularity
N/A
No observed public metrics; popularity remains neutral/archived.
Merged summary
TL;DR - Chapter 3 of a popular-science series ("Explaining AI with middle-school math") that reduces the Transformer attention mechanism to a weighted average, walking through a hand-computable toy example. It matters as an accessible conceptual explainer rather than new research.
- Core claim: a word's contextual meaning comes from a weighted average over other words' vectors; worked example uses "苹果 很 甜" with 2-D vectors and a 3×3 weight matrix whose rows sum to 1 (e.g. 苹果 → [0.8, 0.9]).
- Q/K/V are framed as three "personas" produced by multiplying a token's embedding by three learned matrices — Q asks, K labels (library book spine), V supplies content; scores come from Q·K dot products, scaled, then softmax (exponential sharpening beats plain normalization).
- Causal attention is enforced by a triangular mask setting disallowed positions to −∞ so softmax zeroes them, which is what enables left-to-right generation; the first token has no context.
- Multi-head attention runs parallel Q/K/V sets (GPT-2: 12 heads; GPT-3: 96 layers × 96 heads ≈ 9216), with heads empirically specializing (syntax, long-range dependencies, punctuation/structure, coreference) without being told to; outputs are concatenated and projected back.
Sources (1)
「用初中数学讲明白AI」第3章：注意力——一个加权平均如何改变世界
Public signals
N/A
TL;DR - Chapter 3 of a popular-science series ("Explaining AI with middle-school math") that reduces the Transformer attention mechanism to a weighted average, walking through a hand-computable toy example. It matters as an accessible conceptual explainer rather than new research.
- Core claim: a word's contextual meaning comes from a weighted average over other words' vectors; worked example uses "苹果 很 甜" with 2-D vectors and a 3×3 weight matrix whose rows sum to 1 (e.g. 苹果 → [0.8, 0.9]).
- Q/K/V are framed as three "personas" produced by multiplying a token's embedding by three learned matrices — Q asks, K labels (library book spine), V supplies content; scores come from Q·K dot products, scaled, then softmax (exponential sharpening beats plain normalization).
- Causal attention is enforced by a triangular mask setting disallowed positions to −∞ so softmax zeroes them, which is what enables left-to-right generation; the first token has no context.
- Multi-head attention runs parallel Q/K/V sets (GPT-2: 12 heads; GPT-3: 96 layers × 96 heads ≈ 9216), with heads empirically specializing (syntax, long-range dependencies, punctuation/structure, coreference) without being told to; outputs are concatenated and projected back.