苏剑林：浅谈K3，模型架构的下一步，可能不在颠覆而在最小改动

TL;DR - Su Jianlin, from the Kimi team, walks through the architecture of the newly released open-source model K3 (KDA + MLA + Stable LatentMoE + AttnRes, trained with a per-head Muon optimizer), arguing that the next step in model architecture is incremental refinement of validated components rather than wholesale redesign.

• Stable LatentMoE: LatentMoE (down-project → 2n-choose-2k routing → up-project) gains accuracy at similar cost but chains four matrices and destabilizes training. Fixes: replace SiLU with SiTU (sigmoid-tanh, β=4) plus softcap on the linear branch (β₁=4, β₂=25) to suppress O(‖x‖⁴) outliers — softcap beat the hard clipping used in GPT-OSS/DSV4 — and add a single RMSNorm before up-projection, which also improved benchmarks beyond stability.
• Load balancing: with experts going from 448-choose-8 to 896-choose-16, the SignSGD-style Loss-Free update became unstable, so K3 switched to Quantile Balancing (no extra hyperparameters), computing global quantiles via 1000-bin histogram approximation (10k bins gave no gain) that aggregates cheaply across devices and gradient accumulation.
• Attention: K3 keeps MLA despite DSV4 dropping it, since MLA remains near-optimal for fixed training cost and KV cache; alternatives (GQA8, MFA) either lose accuracy or raise training/prefill cost. The author reads DSV4's head_dims=512 K=V MQA plus sparsity/compression as pushing MLA's decoding form to an extreme rather than abandoning it.
• NoPE: RoPE is removed because the hybrid KDA+MLA design implicitly supplies generalized positional encoding (via the DeltaNet/PaTH equivalence); adding RoPE back changed nothing measurable, though a pure-MLA model like K2 still needs it.