Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View

TL;DR - This paper unifies reverse-trajectory and forward-matching reinforcement learning methods for diffusion models under one path-space policy-gradient framework. It argues that their empirical differences primarily arise from variance reduction and derives a recipe that improves prior diffusion-RL baselines.

• Derives an explicit trajectory-space policy gradient using importance sampling between sampling SDEs.
• Connects Flow-GRPO-style updates with the forward-matching structure of AWM and DiffusionNFT.
• Organizes diffusion-RL design around value-gradient estimation, weighting functions, and sampling choices.
• Proposes a rollout-reusing KDE value-gradient estimator and scale-bounded weights, validated on SD3.5-M and Qwen-Image.