Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL
Ranking
Overall
68
Content
80
Popularity
40
Observed public metrics from 1 member.
Merged summary
TL;DR - PRISM is a multi-reward RL framework for LLM post-training that decomposes optimization into separate per-reward policies instead of blending reward signals, reducing objective conflict and adding inference-time controllability.
- Frames the core problem as a worsened "alignment tax": combining multiple reward objectives causes trade-offs/conflicts and unstable, inefficient post-training.
- Instead of compositing rewards, PRISM trains a set of standalone positive policies plus one global negative policy, then composes them in policy space.
- Policy composition at inference enables flexible preference control without retraining.
- Reported to consistently beat existing multi-reward RL baselines on scientific reasoning, tool-use reasoning, and helpfulness-safety alignment.
Sources (1)
Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - PRISM is a multi-reward RL framework for LLM post-training that decomposes optimization into separate per-reward policies instead of blending reward signals, reducing objective conflict and adding inference-time controllability.
- Frames the core problem as a worsened "alignment tax": combining multiple reward objectives causes trade-offs/conflicts and unstable, inefficient post-training.
- Instead of compositing rewards, PRISM trains a set of standalone positive policies plus one global negative policy, then composes them in policy space.
- Policy composition at inference enables flexible preference control without retraining.
- Reported to consistently beat existing multi-reward RL baselines on scientific reasoning, tool-use reasoning, and helpfulness-safety alignment.