Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Experts

TL;DR - This paper finds that sparse mixture-of-experts routers across layers share a common geometry and reusable state-transition dynamics once their layer-specific coordinate systems are aligned. The result could enable more accurate prediction or reuse of routing decisions across model depth.

• Generalized orthogonal Procrustes analysis aligns each router’s control subspace into a shared canonical representation.
• A single linear transition achieves (R^2=0.39)–(0.71), retaining 79–90% of the predictive power of separate layer-specific dynamics.
• Router-control states preserve expert choices more faithfully than matched-rank residual representations, distinguishing routing-specific information from generic cross-layer predictability.
• Learned state evolution improves over simple persistence, reducing (\Delta\mathrm{NLL}) by 15.7% on OLMoE and 6.2% across a 10-router horizon on Phi.