On the Principles Behind Neural Network Optimizers

TL;DR - This thesis develops a theoretical foundation for Adam, explaining its convergence behavior and effectiveness on Transformers. It also introduces Adam-mini, which matches Adam’s performance while reducing optimizer memory by 50%.

• Adam exhibits a problem-dependent convergence phase transition governed partly by batch-size-dependent hyperparameters and β₂.
• Transformer Hessians evolve toward near-block-diagonal, highly heterogeneous structures that favor Adam’s diagonal preconditioning.
• Random matrix theory links this Hessian structure to repeated multiplication of large matrix variables.
• The findings inform other neural-network optimizers, including Muon.