大模型「分寸感」如何炼成？一文看懂RLHF、DPO 与奖励机制
Merged summary
TL;DR - This explainer compares RLHF and DPO as methods for aligning LLMs with human preferences, while highlighting reward hacking and preference bias. The choice of method should depend on whether feedback is subjective or programmatically verifiable.
- RLHF trains a reward model and uses reinforcement learning to optimize responses, enabling exploration but adding cost and complexity.
- DPO directly increases the likelihood of preferred responses over rejected ones, offering simpler and more stable training without a separate reward model.
- Both methods can amplify flawed signals, causing verbosity, sycophancy, or higher proxy rewards despite declining answer quality.
- Verifiable rewards work well for checkable tasks such as math and code, while subjective qualities still require preference-based alignment.
Sources (1)
大模型「分寸感」如何炼成？一文看懂RLHF、DPO 与奖励机制
TL;DR - This explainer compares RLHF and DPO as methods for aligning LLMs with human preferences, while highlighting reward hacking and preference bias. The choice of method should depend on whether feedback is subjective or programmatically verifiable.
- RLHF trains a reward model and uses reinforcement learning to optimize responses, enabling exploration but adding cost and complexity.
- DPO directly increases the likelihood of preferred responses over rejected ones, offering simpler and more stable training without a separate reward model.
- Both methods can amplify flawed signals, causing verbosity, sycophancy, or higher proxy rewards despite declining answer quality.
- Verifiable rewards work well for checkable tasks such as math and code, while subjective qualities still require preference-based alignment.