Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Experts
Ranking
Overall
88
Content
100
Popularity
61
Observed public metrics from 1 member.
Merged summary
TL;DR - This paper finds that sparse mixture-of-experts routers across layers share a common geometry and reusable state-transition dynamics once their layer-specific coordinate systems are aligned. The result could enable more accurate prediction or reuse of routing decisions across model depth.
- Generalized orthogonal Procrustes analysis aligns each router’s control subspace into a shared canonical representation.
- A single linear transition achieves (R^2=0.39)–(0.71), retaining 79–90% of the predictive power of separate layer-specific dynamics.
- Router-control states preserve expert choices more faithfully than matched-rank residual representations, distinguishing routing-specific information from generic cross-layer predictability.
- Learned state evolution improves over simple persistence, reducing (\Delta\mathrm{NLL}) by 15.7% on OLMoE and 6.2% across a 10-router horizon on Phi.
Sources (1)
Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Experts
Public signals
Semantic Scholar citations 1 · Semantic Scholar influential citations 0
TL;DR - This paper finds that sparse mixture-of-experts routers across layers share a common geometry and reusable state-transition dynamics once their layer-specific coordinate systems are aligned. The result could enable more accurate prediction or reuse of routing decisions across model depth.
- Generalized orthogonal Procrustes analysis aligns each router’s control subspace into a shared canonical representation.
- A single linear transition achieves (R^2=0.39)–(0.71), retaining 79–90% of the predictive power of separate layer-specific dynamics.
- Router-control states preserve expert choices more faithfully than matched-rank residual representations, distinguishing routing-specific information from generic cross-layer predictability.
- Learned state evolution improves over simple persistence, reducing (\Delta\mathrm{NLL}) by 15.7% on OLMoE and 6.2% across a 10-router horizon on Phi.