Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL

TL;DR - PRISM is a multi-reward RL framework for LLM post-training that decomposes optimization into separate per-reward policies instead of blending reward signals, reducing objective conflict and adding inference-time controllability.

• Frames the core problem as a worsened "alignment tax": combining multiple reward objectives causes trade-offs/conflicts and unstable, inefficient post-training.
• Instead of compositing rewards, PRISM trains a set of standalone positive policies plus one global negative policy, then composes them in policy space.
• Policy composition at inference enables flexible preference control without retraining.
• Reported to consistently beat existing multi-reward RL baselines on scientific reasoning, tool-use reasoning, and helpfulness-safety alignment.