Beyond the Mean: Multi-Moment Policy Optimization for LLM Reasoning
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - An arXiv preprint that reframes RL for LLM reasoning as optimizing the moments of the failure-probability distribution across problems, proposing Multi-Moment Policy Optimization (MMPO) to jointly minimize several moments instead of just one. It matters because it exposes a hidden design axis in existing RL objectives and reports consistent gains on math reasoning benchmarks.
- Treats the failure probability of a randomly sampled problem as a random variable; existing methods are shown to optimize only a single moment of that distribution, ignoring its broader shape.
- MMPO jointly minimizes multiple moments and has an operational reading: minimizing the expected truncated time to obtain the first successful response.
- Adds a general moment-transformation framework that induces different moment profiles, giving a unified view over a wider family of policy optimization objectives.
- Evaluated on five mathematical reasoning benchmarks across model scales, reported to consistently outperform strong baselines (no specific numbers given in the abstract).
Sources (1)
Beyond the Mean: Multi-Moment Policy Optimization for LLM Reasoning
TL;DR - An arXiv preprint that reframes RL for LLM reasoning as optimizing the moments of the failure-probability distribution across problems, proposing Multi-Moment Policy Optimization (MMPO) to jointly minimize several moments instead of just one. It matters because it exposes a hidden design axis in existing RL objectives and reports consistent gains on math reasoning benchmarks.
- Treats the failure probability of a randomly sampled problem as a random variable; existing methods are shown to optimize only a single moment of that distribution, ignoring its broader shape.
- MMPO jointly minimizes multiple moments and has an operational reading: minimizing the expected truncated time to obtain the first successful response.
- Adds a general moment-transformation framework that induces different moment profiles, giving a unified view over a wider family of policy optimization objectives.
- Evaluated on five mathematical reasoning benchmarks across model scales, reported to consistently outperform strong baselines (no specific numbers given in the abstract).