苏神复盘 Kimi K3：896 个专家背后，藏着哪些关键技术取舍？
Ranking
Overall
68
Content
75
Popularity
N/A
No observed public metrics; popularity remains neutral/archived.
Merged summary
TL;DR - A media recap of Kimi researcher Su Jianlin's post breaking down the architectural trade-offs behind Kimi K3, a 2.8T-parameter MoE model with 896 routed experts. It matters because it shows trillion-scale scaling is now driven by communication, numerical-stability, and engineering constraints rather than raw capacity.
- LatentMoE: tokens are compressed from the 7168-dim main hidden state into a 3584-dim latent space before routing, cutting per-expert compute and cross-GPU traffic; the saved budget funds a bigger pool (896 routed experts, 16 activated, vs. a 448/8 alternative) plus 2 always-on shared experts to offset the latent bottleneck.
- Stable LatentMoE: RMSNorm after expert aggregation and before up-projection normalizes branch scale; SiTU-GLU soft-caps SwiGLU outliers that would otherwise blow up BF16/FP8 dynamic range; Quantile Balancing replaces K2's SignSGD-style fixed-step bias updates by solving directly for per-expert Top-K thresholds.
- Attention: roughly 3 KDA layers per 1 Gated MLA layer — KDA carries continuous fixed-size state (recurrent, order-dependent), MLA does periodic global retrieval, and a gate filters what MLA writes back; because KDA's sequential state updates encode position, RoPE is dropped from MLA ("generalized RoPE").
- Engineering pragmatism: the now-unused 64-dim RoPE branch is kept to avoid disturbing KV-cache layout, attention kernels, and inference stacks; Per-Head Muon decouples optimizer normalization across heads. The article notes open questions — latent-space information loss, KDA forgetting, and reliance on specialized kernels/comms.
Sources (1)
苏神复盘 Kimi K3：896 个专家背后，藏着哪些关键技术取舍？
Public signals
N/A
TL;DR - A media recap of Kimi researcher Su Jianlin's post breaking down the architectural trade-offs behind Kimi K3, a 2.8T-parameter MoE model with 896 routed experts. It matters because it shows trillion-scale scaling is now driven by communication, numerical-stability, and engineering constraints rather than raw capacity.
- LatentMoE: tokens are compressed from the 7168-dim main hidden state into a 3584-dim latent space before routing, cutting per-expert compute and cross-GPU traffic; the saved budget funds a bigger pool (896 routed experts, 16 activated, vs. a 448/8 alternative) plus 2 always-on shared experts to offset the latent bottleneck.
- Stable LatentMoE: RMSNorm after expert aggregation and before up-projection normalizes branch scale; SiTU-GLU soft-caps SwiGLU outliers that would otherwise blow up BF16/FP8 dynamic range; Quantile Balancing replaces K2's SignSGD-style fixed-step bias updates by solving directly for per-expert Top-K thresholds.
- Attention: roughly 3 KDA layers per 1 Gated MLA layer — KDA carries continuous fixed-size state (recurrent, order-dependent), MLA does periodic global retrieval, and a gate filters what MLA writes back; because KDA's sequential state updates encode position, RoPE is dropped from MLA ("generalized RoPE").
- Engineering pragmatism: the now-unused 64-dim RoPE branch is kept to avoid disturbing KV-cache layout, attention kernels, and inference stacks; Per-Head Muon decouples optimizer normalization across heads. The article notes open questions — latent-space information loss, KDA forgetting, and reliance on specialized kernels/comms.