Double Descent and Malign Overfitting in Diffusion Models

TL;DR - Diffusion models can exhibit malign rather than benign overfitting: test loss begins worsening when parameters (p\sim n), long before the interpolation peak at (p\sim nm). Regularization restores the benefits of larger models and can outperform all unregularized configurations.

• Experiments on CelebA U-Nets and a random-features model with closed-form learning curves support the same behavior.
• With (m) noise realizations per sample, the interpolation peak shifts from the standard-regression scale (p\sim n) to (p\sim nm), while overfitting still starts near (p\sim n).
• Bias grows as the learned model approaches the memorizing empirical score; beyond the peak, variance falls but bias continues increasing.
• Ridge regularization in theory and early stopping in U-Net experiments make overparameterization beneficial.