SoftmaxGRPO: Learning to Reason using Softmax Advantage Group Estimation

TL;DR - SoftmaxGRPO is a drop-in replacement for GRPO's z-score group advantage normalization, using temperature-scaled softmax advantages to keep per-prompt weights bounded and avoid wasting learning signal on near-solved prompts. It matters because it fixes a structural flaw in the dominant RL objective used for LLM reasoning training.

• Under binary rewards, GRPO's group normalization produces divergent weighting on easy prompts; softmax advantages remain bounded regardless of prompt difficulty.
• Theory: the exact finite-group population objective is derived for binary rewards, with MaxRL identified as its low-temperature limit; for bounded scalar rewards the large-group update exactly optimizes a log-moment-generating-function objective.
• A negative result is included: no universal finite-group scalar objective exists without extra assumptions on the reward distribution.
• Empirically it reallocates measured gradient budget away from near-solved prompts, reaching 51.8% on DeepMath with verifiable rewards and lifting a 1.5B instruction-tuned model from 35.0% to 68.0% on Poetry using lightweight text-similarity rewards.