Self-OPD: On-Policy Distillation for Flow Matching Models without Teacher

TL;DR - Self-OPD is a teacher-free on-policy distillation framework that uses a flow matching model’s own stochastic exploration to generate step-wise supervision. It aims to reduce teacher-training costs and distribution mismatch while improving single- and multi-objective alignment.

• At each timestep, the method creates multiple stochastic SDE branches, rolls them out with an ODE sampler, and evaluates their rewards against a deterministic self-reference.
• A pull-push objective attracts the model toward high-advantage branches and repels it from low-advantage ones, with direction-aware attenuation and variance normalization.
• Multi-objective alignment combines normalized scores at the reward level to avoid direct gradient conflicts.
• Reported experiments show improvements over prior reinforcement-learning and OPD methods without task-specific teachers.