On the Principles Behind Neural Network Optimizers
Ranking
Overall
82
Content
95
Popularity
N/A
No observed public metrics; popularity remains neutral/archived.
Merged summary
TL;DR - This thesis develops a theoretical foundation for Adam, explaining its convergence behavior and effectiveness on Transformers. It also introduces Adam-mini, which matches Adam’s performance while reducing optimizer memory by 50%.
- Adam exhibits a problem-dependent convergence phase transition governed partly by batch-size-dependent hyperparameters and β₂.
- Transformer Hessians evolve toward near-block-diagonal, highly heterogeneous structures that favor Adam’s diagonal preconditioning.
- Random matrix theory links this Hessian structure to repeated multiplication of large matrix variables.
- The findings inform other neural-network optimizers, including Muon.
Sources (1)
On the Principles Behind Neural Network Optimizers
Public signals
N/A
TL;DR - This thesis develops a theoretical foundation for Adam, explaining its convergence behavior and effectiveness on Transformers. It also introduces Adam-mini, which matches Adam’s performance while reducing optimizer memory by 50%.
- Adam exhibits a problem-dependent convergence phase transition governed partly by batch-size-dependent hyperparameters and β₂.
- Transformer Hessians evolve toward near-block-diagonal, highly heterogeneous structures that favor Adam’s diagonal preconditioning.
- Random matrix theory links this Hessian structure to repeated multiplication of large matrix variables.
- The findings inform other neural-network optimizers, including Muon.