Disentangling Optimization Scale from Preference Scale in DPO
Ranking
Overall
83
Content
100
Popularity
42
Observed public metrics from 1 member.
Merged summary
TL;DR - This paper shows that DPO’s β parameter conflates preference-noise scaling with optimization step scaling, making policy deviation and loss comparisons difficult to interpret. It proposes a centered-softplus reformulation that separates these effects for more controllable alignment training.
- At a fixed learning rate, policy KL divergence from the reference is non-monotonic in β: near zero in a small-β dead zone, maximal at an intermediate β, then lower again.
- Similar DPO loss curves across different β values can correspond to several-fold differences in policy KL divergence.
- The proposed objective is argmin-equivalent to DPO for β > 0 while allowing preference-noise scale and learning-rate effects to be tuned independently.
- Its normalized form has a continuous β → 0 limit that becomes a linear preference-margin objective.
Sources (1)
Disentangling Optimization Scale from Preference Scale in DPO
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - This paper shows that DPO’s β parameter conflates preference-noise scaling with optimization step scaling, making policy deviation and loss comparisons difficult to interpret. It proposes a centered-softplus reformulation that separates these effects for more controllable alignment training.
- At a fixed learning rate, policy KL divergence from the reference is non-monotonic in β: near zero in a small-β dead zone, maximal at an intermediate β, then lower again.
- Similar DPO loss curves across different β values can correspond to several-fold differences in policy KL divergence.
- The proposed objective is argmin-equivalent to DPO for β > 0 while allowing preference-noise scale and learning-rate effects to be tuned independently.
- Its normalized form has a continuous β → 0 limit that becomes a linear preference-margin objective.