大模型RL为何越训越窄？ACL 2026杰出论文揭秘熵坍缩真相
Merged summary
TL;DR - An ACL 2026 Outstanding Paper analyzes token-level entropy collapse during reinforcement learning with verifiable rewards (RLVR) and introduces STEER, a reweighting method that stabilizes exploration. It improves mathematical reasoning and coding performance across multiple models and RL algorithms.
- Entropy changes depend primarily on token advantage and generation probability, explaining how RL updates either expand or prune reasoning paths.
- Existing interventions address only parts of this dynamic and may inadvertently amplify entropy-reducing updates.
- STEER downweights tokens predicted to cause extreme entropy changes rather than indiscriminately maximizing entropy.
- On Qwen2.5 models, STEER raised average math scores from 44.2 to 48.6 and improved code editing from 42.6 to 45.1.
Sources (1)
大模型RL为何越训越窄？ACL 2026杰出论文揭秘熵坍缩真相
TL;DR - An ACL 2026 Outstanding Paper analyzes token-level entropy collapse during reinforcement learning with verifiable rewards (RLVR) and introduces STEER, a reweighting method that stabilizes exploration. It improves mathematical reasoning and coding performance across multiple models and RL algorithms.
- Entropy changes depend primarily on token advantage and generation probability, explaining how RL updates either expand or prune reasoning paths.
- Existing interventions address only parts of this dynamic and may inadvertently amplify entropy-reducing updates.
- STEER downweights tokens predicted to cause extreme entropy changes rather than indiscriminately maximizing entropy.
- On Qwen2.5 models, STEER raised average math scores from 44.2 to 48.6 and improved code editing from 42.6 to 45.1.