Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration

TL;DR - CE-MoE restructures mixture-of-experts language models to use fewer routed expert layers while preserving depth with additional token-mixing and dense feed-forward layers. This reduces costly all-to-all communication without sacrificing model quality.

• Evaluated across models ranging from 2B to 31.5B total parameters.
• Matches full-MoE baselines on validation loss and downstream benchmarks under equal total and activated parameter budgets.
• At 31.5B parameters, CE-MoE requires 33.3% fewer GPU-hours.
• The largest model also improves average downstream score and inference throughput.