How Much, Then Where: Credit-Conserving Action-to-Token Allocation for Multi-Turn Agent Reinforcement Learning

TL;DR - FACTOR is a reinforcement learning method for multi-turn LLM agents that splits credit assignment into two separate decisions: how much credit each action gets, and how that credit is spread across the action's tokens. It matters because long-horizon agent training is bottlenecked by noisy credit assignment, and FACTOR reports consistent wins across standard agent benchmarks.

• Per-action credit comes from checkpoint-calibrated TD residuals that telescope to the trajectory advantage; token-level allocation uses feedback-conditioned teacher-student likelihood gaps (hindsight allocation).
• Per-action normalization preserves the action-average coefficient and prevents token-level sign flips; an action-mean reduction removes the surrogate weight's implicit dependence on action token length, so at the behavior policy (pre-clipping) each action's inner action-mean surrogate equals its TD credit.
• Reported to beat competitive baselines on ALFWorld, WebShop, and ScienceWorld in every environment-seed comparison, with the largest gains on the longest-horizon environment, and hyperparameters transferring untuned to a larger backbone and a different model family.
• Ablations attribute most of the improvement to TD action credit, with hindsight token allocation adding complementary gains.