PoEM: Predicting RL Outcomes from Existing Policies

TL;DR - PoEM predicts the policy that reinforcement learning would produce for a new reward function by combining policies already trained on other rewards. It could reduce the cost and instability of rerunning post-training whenever rewards change or are combined.

• If a new reward is a linear combination of existing rewards, its policy’s log-probabilities can likewise be expressed as a linear combination of existing log-policies.
• The authors observe that log-policies often occupy an approximately low-rank subspace even when rewards are not linearly related.
• Combination weights can be estimated from reward values or basis-policy outputs on samples, without additional RL training.
• Experiments cover synthetic and real rewards across text and image modalities.