Beyond the Mean: Multi-Moment Policy Optimization for LLM Reasoning

TL;DR - An arXiv preprint that reframes RL for LLM reasoning as optimizing the moments of the failure-probability distribution across problems, proposing Multi-Moment Policy Optimization (MMPO) to jointly minimize several moments instead of just one. It matters because it exposes a hidden design axis in existing RL objectives and reports consistent gains on math reasoning benchmarks.

• Treats the failure probability of a randomly sampled problem as a random variable; existing methods are shown to optimize only a single moment of that distribution, ignoring its broader shape.
• MMPO jointly minimizes multiple moments and has an operational reading: minimizing the expected truncated time to obtain the first successful response.
• Adds a general moment-transformation framework that induces different moment profiles, giving a unified view over a wider family of policy optimization objectives.
• Evaluated on five mathematical reasoning benchmarks across model scales, reported to consistently outperform strong baselines (no specific numbers given in the abstract).