Disentangling Optimization Scale from Preference Scale in DPO

TL;DR - This paper shows that DPO’s β parameter conflates preference-noise scaling with optimization step scaling, making policy deviation and loss comparisons difficult to interpret. It proposes a centered-softplus reformulation that separates these effects for more controllable alignment training.

• At a fixed learning rate, policy KL divergence from the reference is non-monotonic in β: near zero in a small-β dead zone, maximal at an intermediate β, then lower again.
• Similar DPO loss curves across different β values can correspond to several-fold differences in policy KL divergence.
• The proposed objective is argmin-equivalent to DPO for β > 0 while allowing preference-noise scale and learning-rate effects to be tuned independently.
• Its normalized form has a continuous β → 0 limit that becomes a linear preference-margin objective.