Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon

TL;DR - Spectral analysis suggests Muon accelerates LLM pretraining by allocating updates more effectively than Adam across singular directions. The proposed Spectral-Aware Muon further exploits tolerant directions, reducing the training tokens needed to reach a target validation loss.

• Transformer loss landscapes exhibit a stable anisotropic profile: a volatile spectral head requires small steps, while the tolerant bulk supports much larger ones.
• Muon’s uniform scaling explains its advantage over Adam and SGD but still underuses the bulk directions.
• SAMuon amplifies the bulk using a static spectral prior; SAMuon-lite approximates this with rank-one power iteration and near-zero wall-clock overhead.
• Across 124M–1B parameter models, SAMuon used 13.3%–24.0% fewer tokens than Muon to reach the same validation loss.