From Expert Reduction to Behavioral Divergence: Tracing Numerical State through Sparse MoE Inference
Ranking
Overall
78
Content
95
Popularity
39
Observed public metrics from 1 member.
Merged summary
TL;DR - This study shows that mathematically equivalent expert-reduction orders can alter sparse-MoE routing and generated text because of floating-point effects. It argues that operand conversion, accumulator precision, and reduction order must be treated as a runtime and hardware compatibility contract.
- Controlled DeepSeek-V4-Flash experiments produced multiple continuation basins from different reduction orders, including sharply different semantic outcomes.
- Numerical divergence persisted across token boundaries even when emitted tokens initially matched, demonstrating that identical text does not imply identical autoregressive state.
- Exact state reconstruction reproduced downstream routes, predictions, and text, identifying post-mHC and full persistent state as key causal boundaries.
- One tested aggregation scheme preserved native routes, tokens, text, and internal states bitwise, though its order invariance was only established for the evaluated configurations.
Sources (1)
From Expert Reduction to Behavioral Divergence: Tracing Numerical State through Sparse MoE Inference
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - This study shows that mathematically equivalent expert-reduction orders can alter sparse-MoE routing and generated text because of floating-point effects. It argues that operand conversion, accumulator precision, and reduction order must be treated as a runtime and hardware compatibility contract.
- Controlled DeepSeek-V4-Flash experiments produced multiple continuation basins from different reduction orders, including sharply different semantic outcomes.
- Numerical divergence persisted across token boundaries even when emitted tokens initially matched, demonstrating that identical text does not imply identical autoregressive state.
- Exact state reconstruction reproduced downstream routes, predictions, and text, identifying post-mHC and full persistent state as key causal boundaries.
- One tested aggregation scheme preserved native routes, tokens, text, and internal states bitwise, though its order invariance was only established for the evaluated configurations.