Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - TRIAL is an agentic RL framework that converts sparse outcome rewards into dense, turn-aligned hindsight supervision, deciding per-turn credit from how much a hindsight-conditioned context changes the model's log-probability of its own response. It matters because it addresses the open problem of allocating multiple hindsight signals across turns of a completed rollout.
- For each decision turn, TRIAL builds an outcome view of that decision's realized consequence and scores the same response under both ordinary and hindsight-conditioned contexts; the signed log-probability gap sets the direction and local strength of token-level supervision.
- Turn-level magnitudes are normalized jointly across the realized trajectory, yielding allocation multipliers with an eligible-token-weighted mean of one — supervision is redistributed across turns while the average multiplier stays fixed.
- On WebShop and ALFWorld across multiple backbones, TRIAL beats GRPO in all eight backbone/environment/metric combinations and is best or tied-best among six methods on six of them; WebShop with Qwen3-1.7B goes from 56.4% to 75.2% success and 78.7% to 85.7% task score.
- Ablations attribute substantial gains specifically to trajectory-relative turn allocation, beyond what dense hindsight distillation alone provides.
Sources (1)
Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning
TL;DR - TRIAL is an agentic RL framework that converts sparse outcome rewards into dense, turn-aligned hindsight supervision, deciding per-turn credit from how much a hindsight-conditioned context changes the model's log-probability of its own response. It matters because it addresses the open problem of allocating multiple hindsight signals across turns of a completed rollout.
- For each decision turn, TRIAL builds an outcome view of that decision's realized consequence and scores the same response under both ordinary and hindsight-conditioned contexts; the signed log-probability gap sets the direction and local strength of token-level supervision.
- Turn-level magnitudes are normalized jointly across the realized trajectory, yielding allocation multipliers with an eligible-token-weighted mean of one — supervision is redistributed across turns while the average multiplier stays fixed.
- On WebShop and ALFWorld across multiple backbones, TRIAL beats GRPO in all eight backbone/environment/metric combinations and is best or tied-best among six methods on six of them; WebShop with Qwen3-1.7B goes from 56.4% to 75.2% success and 78.7% to 85.7% task score.
- Ablations attribute substantial gains specifically to trajectory-relative turn allocation, beyond what dense hindsight distillation alone provides.