SoftmaxGRPO: Learning to Reason using Softmax Advantage Group Estimation
Ranking
Overall
69
Content
80
Popularity
43
Observed public metrics from 1 member.
Merged summary
TL;DR - SoftmaxGRPO is a drop-in replacement for GRPO's z-score group advantage normalization, using temperature-scaled softmax advantages to keep per-prompt weights bounded and avoid wasting learning signal on near-solved prompts. It matters because it fixes a structural flaw in the dominant RL objective used for LLM reasoning training.
- Under binary rewards, GRPO's group normalization produces divergent weighting on easy prompts; softmax advantages remain bounded regardless of prompt difficulty.
- Theory: the exact finite-group population objective is derived for binary rewards, with MaxRL identified as its low-temperature limit; for bounded scalar rewards the large-group update exactly optimizes a log-moment-generating-function objective.
- A negative result is included: no universal finite-group scalar objective exists without extra assumptions on the reward distribution.
- Empirically it reallocates measured gradient budget away from near-solved prompts, reaching 51.8% on DeepMath with verifiable rewards and lifting a 1.5B instruction-tuned model from 35.0% to 68.0% on Poetry using lightweight text-similarity rewards.
Sources (1)
SoftmaxGRPO: Learning to Reason using Softmax Advantage Group Estimation
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - SoftmaxGRPO is a drop-in replacement for GRPO's z-score group advantage normalization, using temperature-scaled softmax advantages to keep per-prompt weights bounded and avoid wasting learning signal on near-solved prompts. It matters because it fixes a structural flaw in the dominant RL objective used for LLM reasoning training.
- Under binary rewards, GRPO's group normalization produces divergent weighting on easy prompts; softmax advantages remain bounded regardless of prompt difficulty.
- Theory: the exact finite-group population objective is derived for binary rewards, with MaxRL identified as its low-temperature limit; for bounded scalar rewards the large-group update exactly optimizes a log-moment-generating-function objective.
- A negative result is included: no universal finite-group scalar objective exists without extra assumptions on the reward distribution.
- Empirically it reallocates measured gradient budget away from near-solved prompts, reaching 51.8% on DeepMath with verifiable rewards and lifting a 1.5B instruction-tuned model from 35.0% to 68.0% on Poetry using lightweight text-similarity rewards.