Double Descent and Malign Overfitting in Diffusion Models
Ranking
Overall
79
Content
100
Popularity
29
Observed public metrics from 1 member.
Merged summary
TL;DR - Diffusion models can exhibit malign rather than benign overfitting: test loss begins worsening when parameters (p\sim n), long before the interpolation peak at (p\sim nm). Regularization restores the benefits of larger models and can outperform all unregularized configurations.
- Experiments on CelebA U-Nets and a random-features model with closed-form learning curves support the same behavior.
- With (m) noise realizations per sample, the interpolation peak shifts from the standard-regression scale (p\sim n) to (p\sim nm), while overfitting still starts near (p\sim n).
- Bias grows as the learned model approaches the memorizing empirical score; beyond the peak, variance falls but bias continues increasing.
- Ridge regularization in theory and early stopping in U-Net experiments make overparameterization beneficial.
Sources (1)
Double Descent and Malign Overfitting in Diffusion Models
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - Diffusion models can exhibit malign rather than benign overfitting: test loss begins worsening when parameters (p\sim n), long before the interpolation peak at (p\sim nm). Regularization restores the benefits of larger models and can outperform all unregularized configurations.
- Experiments on CelebA U-Nets and a random-features model with closed-form learning curves support the same behavior.
- With (m) noise realizations per sample, the interpolation peak shifts from the standard-regression scale (p\sim n) to (p\sim nm), while overfitting still starts near (p\sim n).
- Bias grows as the learned model approaches the memorizing empirical score; beyond the peak, variance falls but bias continues increasing.
- Ridge regularization in theory and early stopping in U-Net experiments make overparameterization beneficial.