TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents

TL;DR — TRACE is a dense credit-assignment method for agentic RL that gives per-tool-call rewards to long-horizon multi-turn agents, addressing the sparse, high-variance, and misleading signal from outcome-only rewards.

• Represents rollouts as state transitions at tool-call boundaries; derives per-action rewards as Temporal-Difference changes in log-ratio state values computed from gold-answer log-probabilities under a frozen reference model.
• Requires no extra critic, process-label training, cold-start SFT, agentic mid-training, or live-web data; its one-step log-ratio TD component telescopes across redundant tool calls.
• On closed-web BrowseComp-Plus, raises Qwen3-4B from 7.2 to 35.6 and Qwen3-30B-A3B from 8.4 to 42.6; learned search behavior transfers to open-web benchmarks with faster RL convergence.