Start Classifying: Categorical Critics for LLM Reinforcement Learning
Ranking
Overall
67
Content
80
Popularity
36
Observed public metrics from 1 member.
Merged summary
TL;DR - HL-Gauss PPO replaces the PPO critic's scalar MSE regression head with a categorical predictor over a discretized value support trained via cross-entropy on smoothed targets, improving LLM RL with verifiable rewards (RLVR). It matters because sparse binary rewards make critic calibration errors directly distort PPO advantages.
- The critic outputs a distribution over value bins and is decoded to a scalar expectation, so GAE and the actor update remain standard PPO — the method is not distributional on the actor side.
- Gains hold across mathematical reasoning, tool-augmented math, and Search-R1, on both Qwen2.5 and Qwen3 backbones, beating strong PPO and DAPO baselines.
- Ablations with one-hot, two-hot, and Bernoulli two-bin critics rule out "bigger output head" or "binary classification alone" as the explanation.
- On shared reasoning prefixes, HL-Gauss improves Brier score and calibration error and produces more symmetric, lower-variance advantages, supporting the claim that categorical value learning acts as an optimization surrogate.
Sources (1)
Start Classifying: Categorical Critics for LLM Reinforcement Learning
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - HL-Gauss PPO replaces the PPO critic's scalar MSE regression head with a categorical predictor over a discretized value support trained via cross-entropy on smoothed targets, improving LLM RL with verifiable rewards (RLVR). It matters because sparse binary rewards make critic calibration errors directly distort PPO advantages.
- The critic outputs a distribution over value bins and is decoded to a scalar expectation, so GAE and the actor update remain standard PPO — the method is not distributional on the actor side.
- Gains hold across mathematical reasoning, tool-augmented math, and Search-R1, on both Qwen2.5 and Qwen3 backbones, beating strong PPO and DAPO baselines.
- Ablations with one-hot, two-hot, and Bernoulli two-bin critics rule out "bigger output head" or "binary classification alone" as the explanation.
- On shared reasoning prefixes, HL-Gauss improves Brier score and calibration error and produces more symmetric, lower-variance advantages, supporting the claim that categorical value learning acts as an optimization surrogate.