PoEM: Predicting RL Outcomes from Existing Policies
Ranking
Overall
71
Content
90
Popularity
27
Observed public metrics from 1 member.
Merged summary
TL;DR - PoEM predicts the policy that reinforcement learning would produce for a new reward function by combining policies already trained on other rewards. It could reduce the cost and instability of rerunning post-training whenever rewards change or are combined.
- If a new reward is a linear combination of existing rewards, its policy’s log-probabilities can likewise be expressed as a linear combination of existing log-policies.
- The authors observe that log-policies often occupy an approximately low-rank subspace even when rewards are not linearly related.
- Combination weights can be estimated from reward values or basis-policy outputs on samples, without additional RL training.
- Experiments cover synthetic and real rewards across text and image modalities.
Sources (1)
PoEM: Predicting RL Outcomes from Existing Policies
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - PoEM predicts the policy that reinforcement learning would produce for a new reward function by combining policies already trained on other rewards. It could reduce the cost and instability of rerunning post-training whenever rewards change or are combined.
- If a new reward is a linear combination of existing rewards, its policy’s log-probabilities can likewise be expressed as a linear combination of existing log-policies.
- The authors observe that log-policies often occupy an approximately low-rank subspace even when rewards are not linearly related.
- Combination weights can be estimated from reward values or basis-policy outputs on samples, without additional RL training.
- Experiments cover synthetic and real rewards across text and image modalities.