From Expert Reduction to Behavioral Divergence: Tracing Numerical State through Sparse MoE Inference

TL;DR - This study shows that mathematically equivalent expert-reduction orders can alter sparse-MoE routing and generated text because of floating-point effects. It argues that operand conversion, accumulator precision, and reduction order must be treated as a runtime and hardware compatibility contract.

• Controlled DeepSeek-V4-Flash experiments produced multiple continuation basins from different reduction orders, including sharply different semantic outcomes.
• Numerical divergence persisted across token boundaries even when emitted tokens initially matched, demonstrating that identical text does not imply identical autoregressive state.
• Exact state reconstruction reproduced downstream routes, predictions, and text, identifying post-mHC and full persistent state as key causal boundaries.
• One tested aggregation scheme preserved native routes, tokens, text, and internal states bitwise, though its order invariance was only established for the evaluated configurations.