The Loss Does Not See the Basis, but Adam Does

TL;DR - An arXiv paper showing that the implicit low-rank bias of gradient descent on factored models $W=UV^\top$ survives only under optimizers that are gauge-equivariant (invariant to $(U,V)\mapsto(UQ,VQ)$), which coordinate-wise methods like Adam and RMSProp are not. It reframes basis/preconditioner choice as determining which interpolant the optimizer selects, not a mere tuning detail.

• Gradient descent, momentum, "shared-scalar" Adam, Muon, and Shampoo are gauge-equivariant; Adam, RMSProp, and other coordinate-wise rules are not. Equivariance is necessary but not sufficient for low-rank recovery.
• Theory contributions: a structure theorem identifying memoryless equivariant rules as exactly the Gram-determined left preconditioners, plus a transfer theorem carrying gradient flow's pathwise properties to common-scalar flows.
• Empirically, nine update rules are ranked on underdetermined matrix sensing; a one-parameter interpolation from coordinate-wise to shared-scalar preconditioning restores the low-rank bias monotonically, isolating anisotropy as the cause. A "spectral schedule" explains conflicting Muon reports: equal-rate updates recover exactly low-rank targets but lose their edge as the spectral tail grows.
• In transformers, Adam splits two gauge-equivalent initializations at the first step (equivariant optimizers stay at float precision), ending with per-head invariants $W_Q^\top W_K$ 56% apart in relative Frobenius distance. On two hyperspectral datasets at matched training loss, GD cuts held-out error 43–44% at the lowest sampling density and at lower effective rank.