Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining

TL;DR - This paper finds that language-model pretraining loss dynamics are governed mainly by the effective learning rate (ELR), defined as the learning rate relative to parameter norm. Using ELR as a shared coordinate could make scaling laws and training schedules more transferable across optimization and norm-control methods.

• Matching ELR produces nearly identical loss trajectories despite substantially different raw learning rates and parameter norms.
• The effect holds across optimizers, architectures, datasets, and model scales, with typical mean collapse errors of a few × 10^-3.
• Normalization design and the timescale of learning-rate and norm variation determine how precisely trajectories collapse.
• ELR explains how weight decay and Hyperball shape loss dynamics and enables a fitted scaling law to transfer across norm-control methods, including delayed acceleration.