DiDPO: Diff-in-Diff Policy Optimization for Coding Agent Training

TL;DR - DiDPO is a critic-free reinforcement learning method that derives fine-grained credit units directly from the structure of code diffs, addressing the fact that a single coding action bundles multiple independent edits whose individual contributions outcome- or step-level RLVR rewards cannot separate.

• Organizes multi-turn coding interactions into thought–action steps, then discovers and compares code diffs across sampled trajectories rather than scoring whole trajectories or whole steps.
• Splits each whole diff into sub-diffs using a "groupability score" that balances an anchor's semantic scope against the group mass it can form; highly similar sub-diffs are aggregated into anchors.
• Anchors form advantage groups, and the diff-level advantage is projected back onto individual response tokens for training.
• Reported to beat strong agentic RL baselines on long-horizon coding and reasoning benchmarks, exceeding comparable methods by over 10% on Qwen2.5-7B-Coder; the authors also open-source verl-code, an agentic RL codebase.